I've been writing software professionally since the late 1980s. So, I've observed this trend and the industry over a long period. I first worked for a startup in the early 1990s, and before that I worked mostly in small businesses.
30 years ago programmers were highly respected. We were mysterious to others and we were able to influence things like office layouts and the like.
At some point over that time period, things shifted. Programmers became seen as "geeks" who didn't really understand business and "business guys" took over. These people don't understand technology and they have disdain for it. I'm talking, in fact, about not just enterprise (Were for a long time the "IT Manager" who was not really technical ruled)... but startups where "business guys" end up being the CEOs.
They think that giving us big monitors and nice computers is valuable, and they do it, but they have no consideration for our workspace and will save even a tiny amount to have an "open plan". They have no concept that we might know what we're talking about because "office space is the realm of business.".
I saw this directly last year-- an office with 2 business people and 14 engineers. One of the "biz guys" was the "manager". They went looking for offices and didn't invite any of the engineers, of course, and came back talking about how great the office they chose was. How it was open plan and all that, and we'd really love it. This is after telling them before they even talked to the real estate person that open plan was the one thing we absolutely didn't want. They looked at several places that had been built out with lots of individual offices but they didn't like it because it was "too dark". OF course once they made up their mind about what THEY wanted, all of our comments were seen just as whining and it was "too late". (even though it wasn't as even to this date they haven't signed a lease due to other factors.)
In the 1950s a great many women went into the work force and there were huge numbers of them employed as secretaries. This was the era of the "Steno pool" and their job title was "typist". They were meant to sit and type at the typewritiers. Because they sit punching a keyboard all day (And because they were women) they didn't have much prestige. Business types don't see it as real work.
This is how programmers are seen today-- we're just typists "arranging the little ones and zeros" (direct quote from that boss last year)
There is no respect for us.
It's the fundamental difference between the "enlisted" and the "officers", the "managers" and the "grunts". We're grunts, and we're meant to be interchangeable cogs-- that's why they'd rather employ a dozen mediocre java developers than one brilliant erlangist. We're not capable of decision making and we have no understanding beyond our weird obsession with those stupid computers. -- that's how they see us. They'll lie and say otherwise, but deep down and a fundamental level, that's how non-technical people see us.
And I think we don't really deserve it until we stop joining places like this.
Whenever I see a startup with an open office plan I don't apply. But I think I'm making a mistake. I should apply, go thru the process until I see the office, and then tell them right then and there "sorry, your website said you value employees but by putting engineers in an open office it's obvious you don't".
Of course that won't accomplish anything.
The only people as low on the totem pole as engineers is HR, and HR feels the need to lord over others, so of course they couldn't care less about the needs of engineers (or any employees really.)
So, I call on YC and other venture investors. Start asking founders what kind of office they want and then don't invest in the ones who advocate for an open plan for developers.
I bet that alone boosts returns.
Further, I personally will no longer work for companies where the CEO is non-technical unless I'm the CTO, and even then I have concerns. (Eg: as CTO I need to have absolute authority over things like office arrangements for engineers.)
High tech companies where the CEO is not technically proficient are far less likely to be effective.
We need to demolish the idea that engineers can't lead and don't understand business.
As someone whose learned business, it's a lot easier to learn than adding another programming language to your repertoire.