Yes, that's my theory as well. As noted in other comments, some studies have shown that an office of 4 people all working on the same project is the most efficient, and other studies have shown that a single office is best for companies with < 10 people.

And Zuckerberg and Brin and Page probably switched from mostly programming to mostly management by the time their companies reached substantially more than 10 people. They probably have really fond memories from their time when their companies were <10 people.