As a rough guess, optimal business value in an enterprise software company probably results from engineers spending 20-40% of their time with users/product managers/other stakeholders and 60-80% of their time actually programming/designing/working with other developers. These numbers are highly imprecise and depend on circumstances, but they're probably in the right order of magnitude.
The issue with private offices is that in my experience many engineers will prefer to spend 95% of their time engineering and just 5% of their time understanding the problem. As a result they build more features but the features are significantly less valuable. 2-3 person offices seems to mitigate that pretty well, while open plan offices seem to go way too far in the other direction.
But I think we're probably in violent agreement here. I like team rooms. Small team including product people, engineers, qa, even support and sales sometimes, in one enclosed space. Much less random noise, and a more focused vibe. Either that or 100% remote with a single shared team chat room.
When engineering, I personally enjoy spending 11 AM - 4 PM interacting with others and 4 PM - 8 PM locked up in a room working on code. I wish that switching around like that were an option in more offices.
I worked at a company that had 3:00 tea time, like in academic departments. People drank coffee and played board games, but after half an hour, they were talking about work again and the cross-department communication that was needed would happen organically.
That's a lot better than subjecting engineers to a 24/7 open-plan arrangement that will, after 8 hours at a stretch, wear on a typical person's nerves and health.
I agree with you that 2- or 3-person offices will probably provide enough privacy for most people. Individual offices have their perks, but they do require attention to common spaces and communication to prevent the siloization you talked about.
The most individually effective tactic I've found is to eat lunch/play games/get coffee with people in order departments as much as possible. A lot of the best communication occurs in a relaxed setting-I'm convinced that a major key to success is to encourage/enable people who don't normally spend time together to hang out informally.
A manager who does this is incompetent or has hired incompetent people.
A technical leader who does this is controlling or -- again -- is dealing with incompetent people.