I think the Hulk is the only character you can get away with having a different actor each time because as long as he turns into the big green guy, we're good.
I saw this in the theater with my mom and sister (any excuse for a family get-together) and they hadn't seen Thor or Captain America and I don't think they had too much of a problem with the set-up. They did all see (and love) Iron Man so I think that helps a great deal... and it also helped set up the realism of the universe, or rather the levels of realism we're prepared to accept, so that NOW we can accept the giant flying aircraft carrier with the spandexy outfits and so on.
And is it just me or does it seem like we don't see THAT much of the interior of the carrier, I guess we do but somehow that lab... the way that shot that lab and with that window, every time the camera was in there it felt like the only room on the ship. Just that and the big bridge and a corridor. It really made me think of the last Star Trek movie and how there was one hallway and a shiny bridge (which you could barely see) and the transporter room and that was it, when I remember the movies before giving us hallways, engine rooms, medical rooms, they made that lovely dining room and blew the everloving crud out of it, it feels like we're getting less spaceship per movie. With the possible exception of Lockout.
And can I say something about ScoJo? There's something about her as Black Widow I don't like. I think it's the hair, I think that hair is terrible on her, but to me the Black Widow was a Bond girl who decided she didn't need Bond anymore and tossed him off the side of a building, just drippingly dangerous. I was watching The Island not too long ago and the white body suit she wears in that immediately brought to mind the current white Fantastic Four outfits and it clicked, she could be a brill Sue Storm. Of course, I don't have an alternate in my head to cast Black Widow, and it's not like I can do anything about it, I just feel like I'm alone in this.
Oh, and the scene where they go to Germany, did it actually feel to ANYONE that they were in Germany? Didn't to me.
AND the one thing I would have LOVED, LOVED, gone nuts for and screamed in the theater if it happened, was that in the scene where those people are holed up in the bank, one of the hostages should have been Andrew Garfield, and he slowly lifts up one of his sleeves and almost reveals his web-shooter, but is interrupted by Captain America breaking in. Seriously, how epic would that have been?