So some assortment of reasonable starting points, with their pros and cons described, would be valuable.
You can use Scrum, Kanban, ... as starting points. Just don't implement them mindlessly like most companies do.
These are 2 of the 4 tenets in the manifesto for agile development [0]. They go on by saying:
"That is, while there is value in the items on the right, we value the items on the left more."
Essentially, choose tools. Follow processes. Periodically and permanently reevaluate. If following the plan creates an impediment to interactions, or makes the people dissatisfied, respond to these changes. Rinse and repeat.
A previous commenter had it right imho. Its not that the approach isn't working, its that it has been co-opted by management and requires separation to actually succeed. Another note would be that agile itself says:
- non team members shouldn't be at stand ups etc.
- points shouldn't be used for evaluation of performance (good luck with that)
- if management joins anything other than the demo, they should not speak during the meeting (I've had some success here but also always been labeled a curiosity when I push management to be silent)
People first, tools second, and then process.
While scrum/agile does not = more developers, it can be leveraged to actually grow a developer's career and skills when done well.
In a world of continously created saas pursuing bits of walletshare and department budgets endlessly I would say there is a small % of tools that are specialized and uniquely required. There's also a TON of tools that are preference based selection.
Michelangelo sculpted with one set of tools and painted with another. Klimt may have had some overlap with one and none with the other set. Neither was being doggedly pursued by salespeople or having decisions about tools made by their bosses.
IMO: Its not an apt metaphor in the slightest.
And don't get pulled into success. Most anything can and will work with eager teams at the start. Life has a way of changing without you noticing.
That said.
I have never been anywhere where the process does not have room for improvement.
But often improvement seems impossible for structural reasons, for a team that is not empowered, where people are checked out, or not forthright due to whatever reasons. And like you say, people go through the motions for the sake of doing it, leading to wasted time and more frustration.
Eliminating the retrospective is the easy choice to avoid that.