Software Architecture Horror Story

The most important principle in software architecture is that no principles are absolute, except this one, otherwise it would be a paradox. Theoretical correctness must be balanced by practical necessity. I don't know what the ulterior motives were in this case, but I can think of many possible reasons. Unreliable, lack of support, lack of tooling, missing observability, operations heavy, deprecation plans, dependency on another team that is unable to cater, etc. there could be many reasons one might prefer to trade-off a small performance penalty against these. Trade-offs are unavoidable in any significant system, and this seems is one of the smaller ones you would see IMO. What's more important is to make sure the trade off is a conscious choice and to plan for the consequences.

The real issue I see raised here is one I experience myself:

You're hired as an expert. But when you use your expertise, people with the power to make decisions ignore you. You're told you're important to the company, blah blah yakkety schmakkety, but at the end of the day, you, like me, are just a cog in a machine. And there's a critical mass at which that machine becomes a grumbling corporate mech-tank, blundering across the terrain with nary a thought for the people inside or outside the tank - all eyes are on PROFIT.

I don't know how to fix this. I have just resigned myself to the understanding that this is how it is, and I'm paid for my time, so if they want to waste it, that's their prerogative. The sheer number of times I've heard someone raise in a point in a meeting, and devs that I work with all shout out "that's what he's been saying for 2 years", and then the surprise from the bean-counter muppets.

The only real solution is to found your own company (which runs the same risks of going down the technical toilet, and which I personally detested - I'm not a financial guy, and I didn't have the money to hire someone for books and chasing down people who aren't paying. I just wanted to build ): )

Or, do what I'm doing - resign yourself to the fact that the people at the top, whether or not they have technical expertise, over-estimate their technical abilities and use their social capital to push their confident wrongness. Sometimes you can step in the way. Other times, it's just a reason to keep you hired as you fix the mess.