Its interesting, because having worked at a startup for 3-4 years as my first job I'm looking to move into banking or a larger company where I can continue learning. A startup is great, the environment is brilliant and I love the people. I learned a lot in a very short period of time. I however have escalated to the top of the food chain in my area with no where to go. We have a limited budget and do not face scaling problems anymore. Moving to a larger company makes sense. Working with 500-1000 servers and the tools to manage them is experience I definitely need.
Both startups and banking have their advantages. Everyone should try gain experience in both.