The importance of good politics cannot be understated. Politics in this case is roughly defined as getting people onboard with your plans.
I say this because as a finance guy, I've been on lots of projects that look great on a spreadsheet and marketing, sales, or somebody else is really pushing for the investment. That's the point where a technical person takes over and builds what's asked, or a really great technical person may ask a few questions and raise a few points nobody else considered and the whole bubble bursts as we realize it's not feasible/or we can test it in a certain way before investing big time/money into it.
Probably not mastery of SE as OP asked. But mastery as a business partner within an organization.
Either the engagement with technology doesn't happen early enough, or when it does, the technology point of view isn't given the weight it deserves. The ideas have already caught and too many people are invested and have 'faith' it will all work out.
The crux of this is that many of those people who carry the weight in decision making don't actually carry any risk or responsibility for delivery. Given the scale of engineering and time to deliver, by the time it comes to fruition they've moved on, so they never experience accountability for ignoring the advice.
For instance, in my experience at a FAANG, the issue was often that a new technical solution was proposed, architected and built before any business/market people got involved with the result that loads of amazing, brilliantly architected and completely useless projects got built.
(My favourite time was when a (really smart, to be fair) software engineer rediscovered the normal distribution while looking for a way to reduce storage requirements for an analytics product).
Again as a finance guy, this is a major skill I have attempted to develop for myself/team. When projects spin up, I’m typically at forefront and involved in every project technical or not (controlling the purse strings). Many finance guys are know it all’s TBH. They build their ROI models, etc. and making dozens of assumptions it’s then treated as gospel. For some reason, it’s valid to ask “why is this project underperforming our expectations” but it’s often taboo to ask “did our expectations even make sense”. I try to change that where it exists too.
All to say, people like me find people at the onset of the project to gather inputs. If I know you as well versed with track record of valuable insights, I’m more likely to reach out early. If I have worked with you at this stage and you your input is just one of “we need specs/tell us what to build and we can build it” you’ve not added any value to the conversation and I’m likely to not include you next time because I know I just need to bring you specs once project is approved and that’s a different conversation for another day.
This is why I say “business partner”. Being able to understand your company, the politics within, how decisions are made, where you can inject value along the way. These are all traits of people with fast career growth in slower growing firms. You transform from Programmer #13 to people knowing and respecting your name. This is important in talent reviews, etc.
I find some in finance and IT get along fairly well and can build a bigger rapport fairly easy. We’re all excel nerds and many of us have taught ourselves how to program to an extent (VBA, etc).
Depending on the organization, we tend to be more focused on leadership support. A junior analyst will get much more face time with VP and up than down. So that’s part of the problem is those connections just don’t always exist. When they do, it might be with an IT director which has maybe become too far removed from codebase or implementation challenges. It happens with every department within the organization, not just IT. My experience is outside the software industry so may work a little different if sole function of the company is software.
Getting the users in the org to, you know, actually use that well-specced, well-built project, which was delivered within time and budget (or with only a small overrun). I mean, use it at all, or use it enough for it to deliver its benefits. Real life issue. Seen it on at least a few projects I've worked on, including two early in my career. Inertia, luddite fears, turf wars can be some of the causes.
Watch "The Effective Engineer | Edmond Lau | Talks at Google" on YouTube https://youtu.be/BnIz7H5ruy0 - at about 3:21 in.
It's usually helpful to have an internal locus of control, and look for things in oneself (here, soft skills) that can be changed. However, sometimes reality just doesn't afford opportunities in a given situation. Accepting where that is that case means being able to extract oneself from a situation rather than wasting time on it.
An organisation that does this is broken.
Most are.
While it is important to realise that this brokenness is in fact widespread, I don't think it should be normalised.
In your analogy if I get really good at getting stuff passed through city council, what would I optimize for ?
A: Building better bridges ?
Or B: building better money making bridges (potentially of of inferior quality) ?
I'd bet there are more takers for B. And at that point we now have better city council optimizers than better bridge builders.
Lets say we now make a team of some folks from both groups. Now who's going to able to get more people onboard their plans? Its the city council optimizers again. And now some bridge builders figure it out and become better optimizers than builders.
We then take it a step further and add competing teams. Same thing repeats. The team better at city council optimizing beat the better builders all the time.
In this whole process its the quality of the bridges that stays the same ( in most cases it goes down) and the costs of it keep going up.
My sad, unfortunate observations in the field.
Love to see a way out.
Edit: in this way of thinking , the assumption is that there are higher chances of making more money from poorer quality bridge than a more advanced, higher quality bridge. One exception to that is the strategy where you first provide better bridges at lower cost and then bump up the prices for maintenance.
Unfortunately in a large software engineering organization tact and political savvy is more important than raw engineering skill. Once you've been in the industry long enough, you'll understand it.
One of the problems that's common in organizations is that most of the work at the junior level is highly focused on individual contribution and that completely flips at some point in your career.
I tend to encourage folks who come to me about quitting to explore opportunities within the team/organization/company before leaving so they can leverage all the social capital they've built up.
The more important question is "how do you cultivate mastery?" How does your organization get to a place where people can develop mastery instead of the organization trying to create it instantly by hiring "the right people"? Regardless of what people may claim, hiring is not a solved problem. It's _really_ hard to identify candidates that are true masters at what they do (let alone getting them to quit their job and join your team).
By the same token, it's also _really_ hard for true masters to find a place that understands their value and, more importantly, will allow them to do their thing and provide career satisfaction. A lot of talented people are locked out of jobs because someone thought their career trajectory wasn't quite right/impressive.
Training and mentorship, however, are better understood and have lower stakes. It makes me think of Bell Labs. I know a couple folks who worked there who had very "unimpressive" backgrounds, they started as techs but rubbed elbows with stellar talent in a place that cultivated discovery and collaboration. Over time while there and in other places they were able to rise to principal design engineers, beyond what many who went to elite universities achieve. They credit their formative years in Bell labs with getting them started on their career path.
The thing is, Bell Labs didn't have an epically selective hiring process, nor was the compensation insanely high. They provided an environment that _attracted_ people with mastery and those who were passionate about what they do, they developed talent within their walls rather than trying to merely buy it or find it.
This was for electrical engineers, but the same ideas can apply to software engineers.
On the long run, if you stay in such a company your market value will by reduced as it is
(a) psychological very difficult to resist such culture to influence your mind, unlimitedly transforming you.
(b) will eventually turn your life into a saga of bitterness and resentment, lower self esteem.
(c) arguably, such a company will ultimately fail in their business.
so you're probably better off take a possible temporary pay reduction for a better happiness and higher pay on the long run.