I do agree with you. I about learned Agile first from my mentor (his formative years were in the 80s) who I did and do still deeply respect, but over time he and I disagreed over the nature of what Agile is and what it should result it as I gained more real-world experience.
I honestly have no answers as to specific suggestions that work for everyone. I've even asked teams where I implemented Agile practices what they though of my interpretation and the response I get is "Dunno...works fine to me."
The best advice I can offer has nothing to do specifically with Agile, but pertains to my experience customizing agile to an organization. I don't mean to plug my "book" (especially since I don't make money from it), but the best advice I can give anyone I put in here:
https://neilonsoftware.com/books/personality-patterns-of-pro...
Flawed and incomplete I know, but the best I could think to do.
We developers have done an abysmal job developing, socializing, and formalizing our standards and practices. Needing to keep agile things loose because teams and industries and people and projects differ is one thing. But, we don't even try to establish unambiguous definitions for general topics like test-driven development, big visible charts, or sprints.
Frankly, we need industry leadership and earned certifications for developer craftsmanship. Uncle Bob recently has shared some good food for thought in this area. Not only does he recommend the direction we professional devs need to head, but he also speculates as to what will happen if we don't (loss of freedom.)
Of course we all know that waterfall doesn't actually give you what it claims, but it says when it doesn't you failed to manage one of the early phases correctly which means it tells them how to fix things: work harder there.
https://neilonsoftware.com/2016/07/13/a-far-too-brief-rough-...
Yes, a specific date is absolutely what every manager at every level in the organization wants. As long a payroll is issued bi-weekly, and earnings reported quarterly, the desire for a fixed date will be a thing.
The issue is, fixes dates when it comes to new software development is a fantasy. If you really, really wanted to talk to experts about hitting dates, I'm thinking that a well-funded initiative done in collaboration with 3 of the 4 branches of the U.S. military and the most respected innovator in aviation technology would be where you would go to find them. I would like to offer up the F-35 Joint Strike/Fighter program as exhibit 'A'. Turns out, despite all that funding and discipline (and threats of what will happen if you miss a deadline), they still can't hit their dates. If they can't, what hope do we have? That's not a rhetorical excuse - that's a real question.
Is the answer Agile? Personally, I don't think it is. I think Hollywood has a far better model that anyone else does (pre-production -> production-> post-production). To get new ideas, I study how specific movies are made. One of my favorite case-studies is how they did "Max Max: Fury Road" (lots of storyboards - lots and lots and lots of storyboards). "But hollywood movies are always late/overbudget!!!", yes - which is my point. They know that and have specific adaptions that make this process work decade over decade - and generally a hell of a lot of cash in the process. At a minim, if Agile ain't workin' for everyone, we've got to try to get new ideas from somewhere. My personal choice is Hollywood.
Hollywood is broken. https://m.youtube.com/watch?v=H18RUB1cxfI (NSFW)
More consulting, of course, is required.
Process is built for people, not people for the process.
And it should be unique for the team. The software process is not very transferable because people are not transferable (in addition to many other things that are also not transferable). Each team is different and the agile process should reflect that difference.
Assuming there is not a lot of employee turnover, the team should become more consistent over time on a given project.
If your "process" works differently with different people, what you have is indistinguishable from not having a process.
My thought is as follows: McDonalds is designed around the core idea that people are fungible and that mediocrity is optimal result (by "mediocrity" I mean no better, but no worse than the average hamburger). I do not believe that software development resources are fungible due to the high degree of creativity and independent decision making required to strike the best balance of quality and time-to-market, and often mediocre results can hurt companies during critical stages of growth.
With that in mind, do we then say "If resources are not fungible and results cannot be mediocre then process cannot exist"? My answer is "no." A lack of resource fungibility does demand an individualized approach to training new resources on the process as it exists at a point in time, but provided resources do not change, the process itself should not change much outside of evidence-based optimization, i.e. "this ain't workin' so we better change somethin'"
To me, process is about expectation management. Every other aspect of a process is in support of maintaining or exceeding expectations. I believe that if a process is so catered to individual whims that no expectations can be reliably set, then there isn't a process. Having said that, I do believe you can achieve a relatively consistent level of meeting and/or exceeding expectations with a process customized to the individual needs of the team members.
Still, your point is very valid. Mine were only additional thoughts inspired by your astute observation.
I see agile as more a meta process. A process to build a process from. Agile attempts to accept the reality of the world that people are different, teams are different, and projects are different. It even describes the retrospective as a way to change the process so that it works better for a team. Over time different teams should end up with the agile process that works best for them.