"You see lots of very good ideas (like proper source control)" - you mean the group that still uses VSS 4.0 on a network share b/c they're terrified of moving their repo to Git (or even SVN)?
Unsurprisingly, companies that have millions of lines of code have to take extra precautions.
"You get to work with lots of clever people" - no, you get to work with people who work there for no reason other than they live close to the office and have spent the last 14 years building an impenetrable fortress around themselves.
This is a lazy dismissal.
In reality -- lots of smart people work at big companies and small companies. That being said, big companies will invariably have more people dedicated to research than startups. Microsoft Research alone has over a hundred people just doing brilliant things.
"They have lots of perks" - Yes, that one day a year I got to donate $5 to cancer research so I could wear bluejeans (collared shirt still, of course) makes it all worth it. I'll take my catered breakfasts, lunches, impromptu beer runs, work-from-anywhere policy and unlimited vacation days, thank you.
Again, this is lazy and untrue -- unless you want to say that Facebook and Google's perk packages are about wearing bluejeans -- but I think playing the perks game is silly anyway. Distilling things down to money (with some exceptions, like WFH) is always the smarter way to go.
---
Personally, I think the biggest difference working at a big company vs. a small one is that of depth vs. breadth. At a startup, its not beyond the realm of possibility to understand the majority of the code base: you'll be wearing lots of different hats, doing lots of things at once. At a big company, the organization is usually such that you'll spend your entire employment working on one little niche -- this has its advantages (you become an expert at that one thing) and disadvantages (you're only an expert at that one thing, after all).
Personally, I can't imagine why someone wouldn't want to just spend time at both types of companies and see which one they like more.