It's more than just overhead - going public forces you to think in terms of quarterly earnings reports.
It's very hard for a tech company to thrive in that kind of environment - capital expenditures required to develop new products or enter new markets will often not be profitable for several years, and getting the public market to understand that is impossible.