The importance of good politics cannot be understated. Politics in this case is roughly defined as getting people onboard with your plans.
I say this because as a finance guy, I've been on lots of projects that look great on a spreadsheet and marketing, sales, or somebody else is really pushing for the investment. That's the point where a technical person takes over and builds what's asked, or a really great technical person may ask a few questions and raise a few points nobody else considered and the whole bubble bursts as we realize it's not feasible/or we can test it in a certain way before investing big time/money into it.
Probably not mastery of SE as OP asked. But mastery as a business partner within an organization.
Either the engagement with technology doesn't happen early enough, or when it does, the technology point of view isn't given the weight it deserves. The ideas have already caught and too many people are invested and have 'faith' it will all work out.
The crux of this is that many of those people who carry the weight in decision making don't actually carry any risk or responsibility for delivery. Given the scale of engineering and time to deliver, by the time it comes to fruition they've moved on, so they never experience accountability for ignoring the advice.
For instance, in my experience at a FAANG, the issue was often that a new technical solution was proposed, architected and built before any business/market people got involved with the result that loads of amazing, brilliantly architected and completely useless projects got built.
(My favourite time was when a (really smart, to be fair) software engineer rediscovered the normal distribution while looking for a way to reduce storage requirements for an analytics product).
Again as a finance guy, this is a major skill I have attempted to develop for myself/team. When projects spin up, I’m typically at forefront and involved in every project technical or not (controlling the purse strings). Many finance guys are know it all’s TBH. They build their ROI models, etc. and making dozens of assumptions it’s then treated as gospel. For some reason, it’s valid to ask “why is this project underperforming our expectations” but it’s often taboo to ask “did our expectations even make sense”. I try to change that where it exists too.
All to say, people like me find people at the onset of the project to gather inputs. If I know you as well versed with track record of valuable insights, I’m more likely to reach out early. If I have worked with you at this stage and you your input is just one of “we need specs/tell us what to build and we can build it” you’ve not added any value to the conversation and I’m likely to not include you next time because I know I just need to bring you specs once project is approved and that’s a different conversation for another day.
This is why I say “business partner”. Being able to understand your company, the politics within, how decisions are made, where you can inject value along the way. These are all traits of people with fast career growth in slower growing firms. You transform from Programmer #13 to people knowing and respecting your name. This is important in talent reviews, etc.
I find some in finance and IT get along fairly well and can build a bigger rapport fairly easy. We’re all excel nerds and many of us have taught ourselves how to program to an extent (VBA, etc).
Depending on the organization, we tend to be more focused on leadership support. A junior analyst will get much more face time with VP and up than down. So that’s part of the problem is those connections just don’t always exist. When they do, it might be with an IT director which has maybe become too far removed from codebase or implementation challenges. It happens with every department within the organization, not just IT. My experience is outside the software industry so may work a little different if sole function of the company is software.
Getting the users in the org to, you know, actually use that well-specced, well-built project, which was delivered within time and budget (or with only a small overrun). I mean, use it at all, or use it enough for it to deliver its benefits. Real life issue. Seen it on at least a few projects I've worked on, including two early in my career. Inertia, luddite fears, turf wars can be some of the causes.
Watch "The Effective Engineer | Edmond Lau | Talks at Google" on YouTube https://youtu.be/BnIz7H5ruy0 - at about 3:21 in.
In your analogy if I get really good at getting stuff passed through city council, what would I optimize for ?
A: Building better bridges ?
Or B: building better money making bridges (potentially of of inferior quality) ?
I'd bet there are more takers for B. And at that point we now have better city council optimizers than better bridge builders.
Lets say we now make a team of some folks from both groups. Now who's going to able to get more people onboard their plans? Its the city council optimizers again. And now some bridge builders figure it out and become better optimizers than builders.
We then take it a step further and add competing teams. Same thing repeats. The team better at city council optimizing beat the better builders all the time.
In this whole process its the quality of the bridges that stays the same ( in most cases it goes down) and the costs of it keep going up.
My sad, unfortunate observations in the field.
Love to see a way out.
Edit: in this way of thinking , the assumption is that there are higher chances of making more money from poorer quality bridge than a more advanced, higher quality bridge. One exception to that is the strategy where you first provide better bridges at lower cost and then bump up the prices for maintenance.
It's usually helpful to have an internal locus of control, and look for things in oneself (here, soft skills) that can be changed. However, sometimes reality just doesn't afford opportunities in a given situation. Accepting where that is that case means being able to extract oneself from a situation rather than wasting time on it.
An organisation that does this is broken.
Most are.
While it is important to realise that this brokenness is in fact widespread, I don't think it should be normalised.