I think the biggest thing holding engineering estimates back is that the people asking for them are actually not interested in accurate estimates; instead they are looking for inputs to be used in various games of corporate politics.
Most project managers I've worked with either have a desired estimate already in mind or they don't care about any of the extenuating circumstances.
On one hand, the desired estimate is often based on the knowledge that projects estimated to take more than a quarter aren't going to get a green light.
On the other hand, it's ridiculous how many projects blow through estimates when external dependencies are ignored, newly-hired engineers create a burden on the project, and de-scoped work turns out to be necessary.
Those project managers also pursue the same estimation agenda even after several projects turn out the same way.
Devs over-engineer, add way too much padding for refactoring and cleaning out tech-debt than is necessary, devs engineer solutions with resume padding, devs like playing with cool tech or trying new tech instead of just using "the boring old thing", they over-engineer (saying this one twice), they get it wrong, devs over-compensate because they got burned previously, they over-compensate because they got negotiated down and then it went bad, they want to impress their peers or whoever they report to, they get bullied by end users that somehow get access to them, etc etc. Yes a lot of those are avoidable, but we don't live in an ideal world.
This is an aspect of the https://en.wikipedia.org/wiki/Planning_fallacy.
I've worked for guys that shop around and give the work to the lowest estimator, even when they have a track record of low-balling and then running 5x over their estimates.
In other scenarios, to your point, optimistically low estimates are used as a political tool by product/management to wrestle some task/responsibility from some other team in the org.
Inevitably what I see again and again is everyone takes (and fights devs for) low-ball estimates, which assume the happy path of "nothing can go wrong". They are then happy to hear & communicate to clients the various excuses when each "downside surprise" is discovered through the development process. Of course the estimate high-baller has built in time for these as theres rarely positive surprises that make tasks faster, and few tasks are surprise free.