I've mostly worked at large (>10k people), though not necessarily hot, tech companies with established tech ladders.
Now I'm working at a small (<500 people) SaaS company in a non-tech sector. While there are definitely some perks (including, oddly, better pay) I have some concerns directly related to the size of the company and the lack of a ladder:
First, lack of opportunity for specialization. People who reach the top of tech ladders are often genuinely capable specialists in some sub-discipline. One has to specialize to develop mastery. When done well, a tech ladder provides a useful framework within which to cultivate mastery.
The question becomes, how do you develop (transferable) mastery in an environment that isn't organized to exploit it? Partly I'm hoping that as the company grows, I can grow my skill set towards and expertise that the company will be prepared to use.
Second, when the company is small enough, it's still possible for founders/execs to interfere with even relatively mundane technical decisions. Now, I think the people I work for are quite competent, and I actually have gotten to know them (which wouldn't happen at a big company). I'm just saying that for small companies it can be hard for those at the top to let go of control and switch to a more delegated mode of decision making.
EDIT: fixed formatting