I know scrum and sprints get a lot of hate, but I've found them to be extremely effective tools for combating this.
There's no reason why anyone should need to be "certified" for a methodology to be applied. There isn't even a reason why SCUM methodology should be applied as a sort of universally applicable process.
The funniest thing about SCUM is that every diagram someone makes of it looks nearly identical to the straw man that is "Waterfall."
https://en.wikipedia.org/wiki/Scrum_(software_development)#/...
I mean, tell me that you couldn't change some of the jargon on that diagram and fool just about everyone into thinking it describes Waterfall.
The only thing potentially differentiating between SCUM and Waterfall is that the former has (in theory) a smaller iteration window and has liaisons between development and management. But the latter argument is nonsense because generally non-agile teams still self-organize into having liaisons and middle-managers.
And as you pointed out, the answer to why SCUM fails always comes down to not doing it "correctly". If no one can do it correctly, that's because it sucks. I've seen agile principles applied in different ways successfully, but I've never seen SCUM actually work any better than no formal methodology at all.
Scrum is only a capacity planning tool. Within a certain context, track velocity. When new work comes in, estimate it then compare to velocity. If you don't have the timelines you want, then you need to consider changing scope.
In reality, Scrum went from "empower engineers" to "allow managers to micromanage their employees even more, but we call it Agile so you can't push back against it because then you're challenging the orthodoxy."
* We track velocity as the key capacity planning metric. We don't use it for performance or anything else (e.g. we don't care if it creeps or if two different teams have two different velocities).
* We then break down all of our work and estimate it. This gives us a scope.
* From there, we have a conversation with our PM. Here's the timelines you want to hit. Given the data, we need X. Do you want to cut scope or extend timelines?
-----
Lastly, we focus a lot on making decisions that maximize velocity over a medium-term (typically 4 to 12 weeks). There is little value in pushing hard in one sprint if it's going to have an opposite reaction in the next sprint.