Agile won the war but lost the peace
allankellyassociates.co.uk
allankellyassociates.co.uk
Everything around sprints to try to avoid the negatives results in huge wastes of time in meetings attempting to plan better rather than actually getting work done.
Sprint are the root cause. Just use Kanban and you eliminate most of your problems.
In 20 years, I've never heard anyone ask that in the software field. In software, I've always been expected to work as fast as possible at all times, and when we get near an (arbitrary) deadline, to stay late as needed.
Software is full of young people, and people who have never managed any kind of team before. I have no problem working for someone younger than myself, but I do think that people should have experience managing some kind of team before they're thrown into the deep end. Software is much more complex than almost anything else we make, and it's no wonder that just figuring it out as we go along has yielded less than stellar results.
https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/09...
There is no silver bullet for software development process. It takes hard work and flexibility to optimize your team's work flow.
If the project is not done in the 6 weeks you set aside for it, just ship something anyways and move on. It was only worth 6 weeks when you planned on getting it done in 6 weeks, so why spend more? Go ahead and plan another sprint if it turns out what you were able to build in the time allocated wasn't enough, and you really need more. (Put it up against the other opportunities you have, and give them all a fair chance at those finite development resources, something else might also turn out to be more important.)
Just don't spend forever and drag out the deadline, throwing off the timeline for whatever you had planned to do next, because you needed more time for the thing that underestimated how big it was, so it wasn't really possible to do that whole thing in only 6 weeks.
The release that is reduced in scope might be totally insufficient, or, it might be just what your users needed! The knife also cuts both ways – if you're willing to accept that project scope can change in either direction, you can occasionally deliver things early too.
I really like everything about this book.
Project A will bring in $X. You estimate 6 weeks. It takes 12. Project B will bring in $Y. You estimate 6 weeks. It takes 7. If X is similar to Y and you bet on the wrong horse, you've dented and/or sunk your cash flow for no reason.
Saying "Humans suck at estimating" is nonsense. It's a skill that can easily be improved - usually at the cost of some up front planning - and there are plenty of resources to help you with that.
The real problem with Agile is that it gives management an excuse to do no estimation at all. Some alpha-wannabe manager bangs their fist on a desk and says "I want this in three weeks".
Instead of saying "You're insane" teams are pressured into trying to make up for someone else's laziness and lack of contact with reality.
And yet it's the conclusion every major thinker (and practicer) in our field has come to (plus there's historical evidence from tons of mis-estimated projects, regardless of scope and budget).
So, whether does the idea that it's "a skill that can easily be improved" come from? Wishful thinking? I don't think the universe works that way.
Sure, you can learn to be less widely off. But you can't never learn to be regularly correct at estimating, or even close within a small margin.
At best, what happens is that quality get shitty and corners get cut, and you get the product at the correct deadline, but it's not the same that you wanted to ship before development started (even if the features are nominally there).
I've come up with two names for this kind of thinking, one a bit grumpier than the other.
The less-grumpy one is the "Is From Ought Fallacy": It's well-known in philsophical circles that going from "Is" to "Ought" is wrong; that is, saying "something is this way, therefore, that's the way it ought to be" is a terrible argument. Well, when you flip it, it gets even worse: "Things ought to be this way, therefore, that's the way they are" isn't just a bad argument, it isn't quite sane. Yet look around you, and see how often it's used. (Hint: Abstinence-only sex ed. Hint: Drug prohibition.)
My grumpier version is "Arguing With Reality": You can argue with people, whether you define "arguing" to be a screaming match or a logical defense of a thesis. That works. It's sane behavior. Going out and arguing with the rain that it needs to be sunny now... isn't. Rain's gonna rain, people are gonna be bad at time estimates, and making logical arguments about why the real world shouldn't be like that doesn't change the real world one iota.
Maybe they're just variants of the "Utopia Fallacy", or the perfect being the enemy of the good: "If the world were perfect, it would be like this, therefore, doing anything else, even if it gets good results, is imperfect and compromising perfection is wrong." Yeah, pull the other one.
I agree fully and I think the main issue is this: People estimating, wanting estimates, "fueling" the workplace and ways in which businesses operate where estimates and milestones are the norms we work around...
The big thing I see missing is the fact that mostly everyone involved in these "mental work simulations" (as I'll call them) is calculating all of the tasks for completion against time it takes to complete tasks.
There have been some people commenting on, "I am glad to give an estimate so long as I'm allowed to add ample padding time for unexpected issues, scope creep, and then I'm allowed to double the entire estimate or at least add 75%. If I can get that estimate signed off on I'm fine with estimating".
And that's probably where I'd say I sit and feel most comfortable: "I'm fine with giving estimates so long as the person singing off on them realizes I'm really just giving a best case scenario guess with a bunch of worst case scenario 'what-if's' built in as well as some 'inevitable scope creep / customers deciding to change specs on the fly' type stuff." But something is still missing. There is a lot of lingo and business-speak and cross-department buzzword ad-libbing taking place but not a whole lot of ground is being broken in terms of moving things forward. Why is that?
The big thing that's missing from ALL OF THIS (IMHO)? Everyone is basing all of these estimates as if things were taking place in a vacuum. Even in that vacuum, over time, estimators, project managers, and developers have come to realize that including padding time for inevitable issues, scope and feature creep, and unforeseen circumstances is an absolute must. Even then, that's inside a vacuum of sorts.
I truly believe it's impossible to plan without falling prey to the "Estimate inside a vacuum" issue. Even if you are a driveway paver and you've paved 10,000+ 100' driveways in your illustrious 20 year career there is going to be something unforeseen that comes up that will prevent your 10,001th job from being completed under time and under budget. The problem is that there is no way for us to truly determine any and all possible issues that could arise. The bigger issue is that bosses and clients don't tend to care when money is on the table, jobs are at stake, and careers are on the line. So we create estimates and we miss them and promise to get it right next time. Even though we know we never will, vacuum or not.
I'm also curious what kind of work you do that you find estimating effort to be reliable, and what kind of tracking you do to keep yourself honest (e.g. I often see people say something will take two weeks and the only reason it happens in two weeks is they end up working day and night for the last 5-7 days when they realize that's what it's going to take to make it happen).
The work splitting algorithm needs to monitor the overall depth of the queue, as well as the number of workers that are online and their propensity for finishing jobs.
It doesn't help to divide a task into 100 chunks instead of only 20, if there are only 8 workers, especially when each split has a marginal cost that will add up again and factor in to overall cost at the point of final integration.
But with a pool of 1000 workers, dividing the work into only 20 or even 100 chunks might still be leaving significant amounts of power on the table undesirably.
You could try to optimize for both, strike a balance between achieving an earliest possible completion time, and minimizing waste. But there are going to be maxima and minima of either on a continuum, may not be practical to find the sweet spot.
One place where this analogy breaks down, is that computers do not care one iota if their work is redundant, or if their work must be thrown away. Human developers care, even if it might save the employers some bit of money or reduce a risk. Humans usually will not produce identical outputs either, when they are assigned redundant workloads. Most humans would probably prefer not to work on a project that is redundant at all. Computers don't care, ML doesn't care. But it would be soul crushing for devs to have to work in an environment like that. I'm sure it happens more than I would have imagined.
Especially prevalent during the initial start of a new project
Not sure if we think about different things when saying "estimate", but I rarely was able to estimate more precise than "6 to 12 weeks". Unless I am doing something absolutely straightforward, I never really can give good estimate (and since I am forced to do - I just give some random number with a lot of padding which I "feel" I should be able to fit in)
In my practice so far to give good estimate means doing 80% of actual work and leaving these straightforward bits to finish it off. These last pieces are easy to estimate usually, but not before main work is done.
And yet, we do. Look at the number of failed government contracts in recent history:
https://warontherocks.com/2014/12/top-10-failed-defense-prog...
And while much of that is about the failures in program management for large contracts, the government aims to build large things at once (as opposed to an MVP and iterations).
When a critical dependency takes longer than expected, the cost snowballs down the chain.
Of course, it may seem to you like the majority of the work is done in the case of a prototype. This is not the case - a vast majority of the work that goes into developing a jet fighter consists of the documentation as to how fast it can fly, its flight ceiling, etc.
Maintnence manuals, and developing a supply chain to ensure we can have a resilient delivery despite war or changes in the economy.
Training programs, to ensure that pilots are able to safely learn to fly the thing
Testing, which of course requires that a prototype be built in the first place.
I can't break project down to 240 units of time on the spot. even 4 hours is still crazy tight, and produces poor estimates. that's barely a minute of thought for each hour. occasionally a manager will be comfortable with me staring off into space for a couple of days. but that's not super common.
I think anybody can come up with good estimates. Almost nobody is willing to say, Take two hours and figure out how long it will take to estimate Project A and Project B.
This is completely dependent on what you're doing. If you're implementing yet another CRUD app, then sure, you should be able to estimate that to within 10%. If you're developing something a little less predictable, like a game or something for R&D, then estimates go right out the window.
https://web.archive.org/web/20170920093850/https://www.const...
This is more like an issue tracker/project planner with statistical modeling capabilities built-in, I don't know if it's doing the same thing under the hood, this is part of their "special sauce"
> The real problem with Agile is that it gives management an excuse to do no estimation at all.
> Some alpha-wannabe manager bangs their fist on a desk and says "I want this in three weeks".
> Instead of saying "You're insane" teams are pressured into trying to make up for someone else's
> laziness and lack of contact with reality.
Don't blame bad managers and bad companies on Agile. Agile needs to empower the people doing the actual work, or it's not Agile. Managers need to facilitate, not bang their fists.
Throw a new type of project or a new client with special needs, and bets are off, Monte Carlo or not.
My general thoughts is that typical software engineering does not have drastically more degrees of project uniqueness than other engineering fields.
I really doubt that. In construction for example, tolerances, materials, requirements, etc are know well before even a single brick has been laid, and if they are to even consider changing them mid-construction, detailed new plans about everything that changed is in process.
That's almost never the case with software requirements.
The same tools, processes and technologies are also used for decades or in some cases centuries totally unchanged. In software a domain (e.g. web programming) can be completely different every 5-10 years.
lie to ourselves about our true capabilities
lie to our boss / team about how good we said we were
lie to the business about how well spec'd the job is, about how the current architecture matches their known requirements, lie about how their stated requirements and the real requirements match up.
Lies to avoid punishment, lies to gain a job that otherwise would go to competition, lies that simply hide the complexity of every job and estimate
without lies estimating would be easy
Being charitable, I'd prefer to say that mistakes were made, without pointing the finger. We're all not perfect, and pretending that we can all be perfect all the time sets the wrong expectation, so I wouldn't try to pin it on a lie.
And I'd prefer to not have to see my boss going under the bus when I have to say "the reason we're not done on time, is because the boss told you this deadline and list of features was reasonable on that day without perfect information. If she had come to us and asked for our opinion, and given us time to talk it over, we would have worked it up and told you, it couldn't be done without making some concessions!"
If only for the reason that, perfect information is not usually available in the time allowed for making a decision. That's not just being charitable to the boss either, that's also self-preservation. The boss can take a lot more of those kind of hits than I can. Even more if we stay together on issues like this. And extending the time needed to make the decision also can have its own opportunity costs associated.
I agree with your sentiment in general. But not every bad datum that costs in accuracy, is the result of a lie. I've seen it said frequently that you should work to defer every decision until it's absolutely required, so you can make it with the best possible information and avoid those costs.
I think that's honestly a lot easier said than done.
A good leader is able to create a safe space where most people will release their defensive positions and many of the most intractable lies will fall apart.
But we all do lie, most often to ourselves, and not always with evil intent
But I fear the attitude that says "we expect you not to lie (withhold information, fail to speak out) because to do so will hurt the organisation, and you should have the best interests of the organisation / team at heart
Humans don't work like that.
The core idea I took out of Agile is that once the high-level estimates are completed, the developers are meant to break them down into bite-sized tasks, put estimates on those, and prioritize them in a backlog with the project manager, customers, and any stakeholders. (In my case we're not building products to go to market with, I work in Higher Ed. We're building solutions for campus partners. So it's not really "Project B will bring in $Y," it's "Project will further our mission and better enable some common workflow." Not to go off-topic...)
What's happening instead is, we get the high-level tasks and then break the projects down into bite-sized chunks, then estimate the parts, and are told "great, now make that fit into 6 weeks, because that's all you have."
Developers don't estimate in hours or days, they use points or t-shirt sizes, because honestly, we do suck at estimating. It's not what you pay us to do. (I mean this honestly, it's really not. This is what they pay project managers to do, and they're literally a pay-grade above us. And they don't write code, though they may be good at math and judging t-shirt sizes. The better ones also know how to keep their poker face, when meeting with the customers. But if your project is slated to last only 6 weeks, chances are you won't even have access to a PM. You wouldn't want one if you could get one, since 6 weeks is hardly enough time to plan and later change course. What would they do?)
T-shirt sizes are better than estimates at a sprint level too, because all you're really doing is horse-trading. If a feature is XXL according to the devs, maybe it moves to the back of the queue in favor of something that costs less and is worth more. That's what you do at Sprint Planning.
Back to devs, we're experts at writing the features, and sometimes we can tell you how long it's going to take and get that right, ... and sometimes we can harmonize in concert with the project leadership, but that's not our core competency. It's just moving faster, and doing it better. Sometimes you find out that the specs are wrong after the project has already started, and it's going to take longer.
We know that some projects are going to take a year, and some are going to take two. And those projects get project managers, and they are frequently delivered on time, with all of the promised features in them! Six weeks isn't an estimate, it's a budget. (And it's not a real budget, hardly even registers on the balance sheet. It's more like an experimental timeline.)
The problem is, customers prefer that we blow the budget rather than delivering part of a product on-time, even if it might have turned out to be enough. They're not willing to put things in order of priority, take an honest look at what will take 6 weeks to deliver, and settle for that. They'd rather cut corners, and then we allow them to make those concessions. Only they don't really want concessions, so we wind up having to spend extra time papering over the gaps.
This is not a core idea of Agile. The high level estimates are not part of Agile- fixed scope, time, and cost is the opposite of agile. You are just doing extra fake work around what is a fixed design process.
"Developers don't estimate in hours or days, they use points or t-shirt sizes, because honestly, we do suck at estimating." No, they don't estimate in hours or days because those are commitments that are used against them- regardless of whether they are expected to attend other company meetings or other priorities are thrust upon them. Points or sizes are intended to be complexity or quantity of work estimates, which could take different amount of time.
In addition, estimates, whether of time or complexity, have uncertainty. Many enterprises are unable to deal with uncertainty.
If you fix cost and time as part of the budget, what you are left to be flexible on is scope. This is the most natural place for flexibility, as there is are often a wide variety of implementation choices that limit cost and deliver a low budget version of a feature versus one that would require more time.
In the first edit of this comment, before I submitted it, I said "we don't really do Agile, we do Agile-Lite" – I cut this line because I've heard this repeated by leadership and I've asked what does it mean, and never got an answer. I almost wanted to include it, so maybe I understand now!
Developer teams are Agile. Dev leadership is Agile-lite.
> The high level estimates are not part of Agile- fixed scope, time, and cost is the opposite of agile. You are just doing extra fake work around what is a fixed design process.
Leadership tries to fix scope, time, and cost on the long-term calendar because we want to deliver predictably, to not appear unpredictable or unreliable. As much as I tell upper management that you can't have fixed scope, time, and cost, they persist in trying to fix all of those.
And their budgets and goals seem to win over my objections, damn near every time!
I'm not doing extra fake work, I'm taking direction from senior leadership. Their work may be fake. Your opinion on their fake work, is duly noted, and I'm going to decline to state whether I agree for political reasons (in so many words, I like my boss and I rarely get time of my own to interact with the bosses boss.) I try to do my work and not rock the boat. Which is why I might get a demand to do something in 6 weeks that is clearly going to take 12, and just do the work. "We'll be given the extra time when we need it." That's healthy, even if it's somewhat dishonest.
> No, they don't estimate in hours or days because those are commitments that are used against them- regardless of whether they are expected to attend other company meetings or other priorities are thrust upon them.
That's a fact. But at some level, if you have this toxicity in your culture, someone is going to have to translate from t-shirt sizes into hours and weeks at some point, and when these estimates turn out to be wrong, someone's head will have to hit the block. (Hopefully theirs and not ours.)
I'd like to think that doesn't actually happen here, and also that's not why we use t-shirt sizes. Maybe just why your team uses them. Or maybe I'm just naive about it.
On my old CMS team and with my current development team, we use kanban because it aligns with how we function. We get ad hoc tickets in our queue, we pass them through a triage lane, then they're picked up by data, dev and qa as they're completed and within the items allowed with our WIP.
Waving a broad brush to say x is what everyone uses, will fuck everyone up. Neither agile, waterfall nor kanban are the ideal solutions for everyone.
Anything that doesn’t have a Why next to it is essentially carrot dangling. You can have targets/forecasts when you expect to have something ready while making it clear that these are guesses, but if it’s not real then it creates real problems if you pretend that it is.
How would you force your manager into converting that "why" into one you would accept?
All that said, client work is the one place where sprints serve a real purpose. The scope doesn’t expand because you finish early so there is a built in reward, agreed upon price for the scoped work, and a built in penalty.
Only sometimes, we could really have been done with them, without necessarily finishing all of that pet feature which has thrown off the whole timeline, by being more complicated to implement than anyone was able to imagine upfront.
Scrum by the books usually implies that releases happen after the po or the customer has accepted the increment at the sprint review.
I find a good sprint rhythm stops the constant task shuffle. When business knows the dev team is mid-sprint it forces them to think about priority.
We do CI/CD, and only big items (in reality UI/UX changes) wait until a sprint review to go out.
But honestly, I’m not sure what the problem is. I’ve found they they’re perfectly suitable at providing:
- an opportunity to check in with a wider team and demonstrate features.
- the chance for the development team to periodically review progress and identify any recurring issues.
- the chance to plan upcoming work and identify any external tasks that need to be taken care of.
Is the problem maybe that some organisations are over-using the “commitment” aspect as a metric? To my teams, “commitment” has generally meant “what we have agreed among ourselves we will be able to deliver in a unit period”. The metrics are purely internal, and useful for allowing us to see if we frequently over- or under-estimate how much work we can do; the sprint is really just the period of time across which we measure and refine.
Except the people being forced into the sprint aren't the ones with power to opt out. It has always been my experience that there is a "product owner" or agile coach or some other title who makes the decision, and isn't one that has to live inside the system they create.
So I agree with you in theory. Some people thrive in the sprint, others suffer and struggle. If everyone could opt in or out, it would be great. But in practice I've never seen the ability to opt out.
Nobody sprints non-stop, that’s a marathon.
If they were called something like “timeboxes” you get all your bullets without the demoralizaing notion of inability to maintain an unsustainable pace.
bwhahaha, good one!
I'd be even stronger on that point – not only should metrics probably not be available outside of the team, but they are even completely without meaning if they're used outside the context of the team. Maybe one of the biggest problems people experience with Scrum or Agile generally is the inappropriate use of metrics.
Sprint objectives are by no means arbitrary or meaningless - or any more so than other deadlines - they have value to the extent they are intelligently formulated.
Sprints have a very important operational dynamic, in that they set a fairly clear, achievable, short term goal.
Software is always a pandora's box, and left to their own devices, a lot of devs could work on the smallest thing ... forever. We are all guilty of this.
Long term milestones (i.e major product iteration) are hard to internalize, they are vague to each individual even if they have meaning at scale.
Sprints provide the means for large things to be broken down into achievable pieces, which much more strongly enable small teams and individuals to focus on that specific outcomes.
Maybe they shouldn't be called 'sprints' maybe just 'mini milestones' or something ... but they are useful. Or 'waypoints'.
Getting lost in a sea of software is very easy ... so setting 'waypoints' I feel is very, very useful.
Lately, the trend with those using Jira in large organizations is to just "carry-over" what's not done in the current sprint into the next. And so instead of talking about why something didn't get completed, and how to resolve it quickly (move it to someone else, change the way it's done, postpone it until something else makes it easier), the work snowballs, with even more work getting piled onto the sprint. That's because while the dev team is agile, management deadlines often won't budge.
The other problem is dependency upon another process/service internal to the organization that is not on your sprint cycle. Typically they have other, different priorities, which don't match yours. So they may get around to it eventually, which might mean a week to a month typically.
I think Kanban is in someways vastly superior in these cases. You can still do "waypoints" at say every two weeks to see how things are moving along, but at some point if your company isn't "agile" then it really makes no difference if your team is.
When they are treated as deadlines, the problems come. And there is no reason you can simply forgo sprints entirely...and still check in, see how things are going, pivot, evaluate new information.
To this point in my career, I’ve never seen sprints present when they didn’t go down the destructive path for teams.
Because 'checking in' on something doesn't create incentive to completion, or actual material outcomes.
You can have 'the software developers doing stuff and meeting objectives on their terms' - or have deadline driven wherein business/software objectives are set and people move towards them.
The problem with the objection to 'deadlines' is the fact that inherently software is not created for it's own aesthetic sake, or for it's cleanliness, or architectural integrity, or some arbitrary measure of quality.
'Time' is the key ingredient often missing from the architectural/development equation, because 'time is money' both in cost, and opportunity cost.
'Time' (which implies cost, ROI etc.) will likely be driven as a business impetus and has to be put into the equation in very serious terms.
Many devs might complain about 'technical debt' or 'messy software' or 'some things that don't work' - but in reality, those situations may actually represent optimal business scenarios.
It may not make sense at all to make certain systems perfect, or at least, in the short run or on first iteration.
In other terms, the 'QoS' expected is I think always way higher on the dev side than real business terms may mandate. What devs view as 'a mess' may be a perfectly good condition.
'Building the wrong thing' is the #1 reason that companies fail, not 'technical debt'.
If customer outcomes are generally good, revenues are increasing, then technical debt is normal, and it's a function of pragmatism at that point more than anything.
Taking too long to build something, or taking a long time to build something that is 'the wrong thing', is almost guaranteed death.
Finally ... I think milestones with deadlines are important also for the psychological and organizational factors.
In NATO, for examples, all 'orders formats' are the same. Any NATO General, from Polish to American, gives a set of orders that are crudely similar to those given by section commanders to their privates. That's to enable operational efficiency. The 'most important' part of the orders are the mission objective with a timing. Literally it has to be said twice. I'm not kidding - while giving orders, the mission statement is verbally repeated. Always. This is because in the fog of war 'everything goes wrong', nothing goes according to plan, things break, people die, obstacles abound, opportunities arise (just like at a startup) - but if everyone involved knows what they need to do, then they can adapt and make it happen by hook or by crook. So 'the gap at grid reference 0000 1111 needs to be bridged by 0600h tomorrow so 6th calvary can cross with heavy tanks' is a mission. Everything else about the plan is secondary, everything could go wrong, but when individual sappers/soldiers, NCO's, junior officers all have that 'this spot has to be gapped by 0600 for heavy vehicles' ... then they can rally around it whatever happens.
Obviously there are tradeoffs and obviously devs have to participate in the establishment of their own deadlines and objectives ... but in my view it has to be done, and it's healthy to break it down.
If you have a dev team with no incentive for completion, you have a dev team problem.
If you have a hold up to deliver perfection instead of what the business needs, you have a scope creep problem.
Neither of those are solved by sprints but they are often used to justify them.
Devs naturally lack incentive - it's an existential nature of software, i.e. it can go on forever. It's not like building a house.
"If you have a hold up to deliver perfection instead of what the business needs, you have a scope creep problem"
A 'Quality' issue problem, not so much scope, but again - there's a natural dissonance between Engineering instinct, and outcomes. Engineers naturally want to build something robust, clean, and performant. It always makes sense to them, pushing against that is always hard.
Ergo - the need for clear milestones - with timings.
But we're talking about adult humans who have agreed to a contract to fulfill business needs in exchange for money, I tend to presume that its on the developer to guarantee their end of the contract, otherwise business won't fulfull there's and I further presume business gets to decide what the criteria are. If things are otherwise, which this seems to be suggesting, then that would be an egregious business error, keeping on developers that aren't delivering anything, and if your tech lead isn't giving you honest assessments you let them go too. This looks the business problem is being coupled into the warrant behind sprints being key cause devs "naturally lack incentive".
I also don't know how to agree with developers "naturally" lacking incentive, given they could lose their job, but I've definitely met developers with little motivation. That level of motivation tends to be tied to wider organizational issues as well...
Technically, that's true, but in reality not.
Devs go to work, build stuff, and get paid every few weeks.
Their direction really is oriented only towards the goals they are given.
Business needs are often counterintuitive to the natural position of a developer.
The reason I believe this, is because I'm a developer who moved into the business world. I see this utterly and fundamentally different now. Even how I approach writing code is different.
Business needs are very difficult to understand for anyone who has not sat in a true business role: customer facing, selling i.e. direct sales etc.. A few people have a 'natural born' business instinct, but it's rare.
Even people in regular marketing jobs (i.e. not marketing ops, like branding, creative) I find don't have an instinct for it.
I like specific goals, that are attached to timings. They're not always going to be made, but both of those things are important.
Developers moving in the wrong direction is more common in software than not.
There is a big difference between infinite knob-tuning and solving the problem in a manner that meets needs.
It’s very much a scope creep problem.
It's easy to think something is a step forward, work on it, then realize it's a bad idea later.
If you know what the individual, bite-sized tasks really are, you are nearly done with the project.
It's a mostly a business problem. Not mostly an engineering problem.
Cost, product issues, customer issues, support issues, IP issues, margins, positioning, partnerships, finance, M&A, staffing - these are all part of the equation. Eng is only one piece.
If there are not enough tasks you pull from backlog, if it's too much then they get finished in next sprint. Work never ends, so why the need for extra overhead to group it into 2 weeks?
I always see it as a "marathon", constant sprinting is no good long term.
However, Kanban works as long the team knows priorities and can choose between good and over kill solution, which is easier said than done. Working in a good pace without a deadline is challenging itself, since you think you have all the time in the world.
A sprint goal is meant to provide focus. You've got something to work towards, but if you don't make it, you don't make it. Presumably there was a good reason you didn't make it. It's good to examine that reason to see if that reason can be taken away, or otherwise it can be taken into account in planning. But it's not the end of the world if you don't complete a sprint on time, and you should certainly not work late to complete the sprint.
I've worked in very successful teams that never finished a sprint on time. I think that's totally fine. It's not about the deadline, it's about the focus.
Of course it never is. As mistakes are revealed, unexpected challenges appear, and requirements are updated or clarified, the plan unravels instead of adapting.
The only real benefit of Agile is that it should - in theory - allow the plan to be modified, so projects can be steered towards a target instead of being treated as fire-and-forget. (And miss.)
Task granularity should not be a specific benefit - it should fall naturally out of competent goal-based planning in whatever paradigm is being used.
What happens as a result, is that more major tasks never get sufficient priority to actually happen. This can easily lead to an incremental "refuctoring" of the code base, where you have a lot of individually-nice bite-sized tasks that, in aggregate, turn the source tree into a total mess.
Agile as I've seen it in the wild is easy to twist into "plan more verbosely" instead of "plan better," though.
The most recent way I've seen this manifest is coming into a team that planned out each ticket in great detail when it came to execution - gotta maximize that committed/completed ratio - but invested very little in making sure they were planning the right thing. Someone would ask for something, it would become a ticket. But whether or not that was the right thing short-/medium-/long-term? That's a conversation that didn't have anywhere to happen.
Another is confusing "producing more estimation artifacts" (subtasks, hour allocation, etc) with "investing the right amount of time in planning." Going from a 30 second story point estimate to a 2 minute subtask breakdown in sprint planning isn't what it takes to make an accurate estimate for a project.
We use them to structure our work, but decoupled them from releases. The Dev team owns the sprints, but the business owns releases. DevOps uses Kanban however, because it works better for them. That makes us more flexible. Agile.
It's been a years-long process but I'm really happy with our "Agility" these days. Our sponsor is the VP of Tech and he gets it (used to be our QA lead). If we have rollover we talk briefly about estimating better but it's not the end of the world, because we actually have gotten better at estimating. Some sprints we nail it. Other times we miss, but management isn't using those metrics as a bludgeon so it actually helps us to work smarter.
To give some context: We have multiple pieces of software we build and maintain for multiple clients. When a client requests some features, they buy some sprints and we do the work and then it's done. During the life cycle of the software there is a maintenance contract for service and support and resolving bugs. This does not have much room for software craftsmanship either.
This is just where I work,but I can't imagine we are the only one working like this. This is an implementation problem, not an methodical problem.
One of the ultimate causes is that our org is project based, so no project, no work for that client. And the software quality suffers IMHO.
I disagree. there's nothing inherently wrong with short iterations or a predictable release window.
trying to do agile with fixed budget, fixed requirements and a fixed timeline is the core issue.
most of the problems with agile, including "devs having to do this much tasks or die" within each iteration, comes from trying agile on an inflexible plan.
You move a lot without going no where, with a lot of broken things.
Companies are basically dictatorships that want to control their employees ("resources") and everything else so it's pretty natural that they will distort every new movement into a tool to extend that control.
The printing press being the only other well documented time of this explosion of knowledge/technology allows us to see that no matter how power expands and then coalesces, it takes the good ideas, fragments away, and then repeats the process as a sect of the greater power.
There is just no way that some salesperson is going to be able to sell a product that’s going to be developed in collaboration with a customer in 2 week increments (unless the customer is already specifically dealing with a software consultancy).
They sell a package of ‘this is everything the product is going to do’ and then somehow try to squash that waterfall list of features into ‘stories’, and call it agile.
Except that we don't sell "a product", we sell time. We'll do everything we can to help you getting a product out of this time, but if you can't, you were doomed from the beginning.
It's just a different thing to sell for a salesperson, they have to adapt. But it works.
Traditional sales knows how to sell either a product or a service, hardly the mix of them, which makes software special (but not unique).
Any particular incarnation of it, say Scrum, is going to have issues when applied to real world problems. Guidelines like "get quick feedback" are useful, but decrees like "one week sprints" can get tyrannical.
Luckily a lot of people seem to understand that and I've never met anyone who insists on doing exactly according to some specific scripture.
I 100% agree with the other comment around "checkin" points so that work doesn't linger forever, but the forced short delivery boundaries very likely slow teams down overall especially when all the process overhead of points, backlog grooming, standups, sprint planning, demo session, etc are all factored in and mandatory every 2 weeks.
Commit to some scope, on some cadence/date, loosely check in on progress, and adjust as needed as a team. Deliver as frequently as possible and demo your work when you can/should.
I’ve seen Agile done well, which I really liked. Mostly what I see is water scrum.
It’s important to pack every single thing you can into a sprint based on your estimate of points, otherwise you’re wasting precious resources. Of course that means you never get everything done so huge chunk of the sprint always rules over.
Everything is dictated from the top, there is no autonomy on the part of the team (or almost none).
As bad as all this is, a quickly falls apart whenever something doesn’t go exactly the plan. In other words, constantly. So you just get further behind and more things roll over into the next sprint.
They’re “agile”. They hold demos of the tiny bit that got done and congratulate themselves. Because they don’t do waterfall anymore. They GET IT.
But none of the benefits. Because that would require actual change.
This will, naturally, cause hairs on necks to rise waiting to scream back "What are you talking about? Of course Agile is the problem!" But really, the problem is that any business trend/fad is inevitably doomed to become a hollow shell of what it was intended to be.
Or more pointedly: Businesses don't really want to overhaul their development process, and any new process integrated will inevitably become just an extension of whatever they were doing before. This is basically what the article's saying without directly saying it.
I mean, the very first phrase in the Agile manifesto is:
> Individuals and Interactions over processes and tools
And yet see how many anecdotes in this thread are about how the most annoying part of Agile is the rigid adherence to certain processes. If 1 week sprints are too short, change them to 4 week sprints. Figure out the cadence that makes sense for your team. That's what Agile is supposed to be about.
Regardless of whatever Large Co wants to call their development process, let's stop calling it Agile and start calling it what it really is: Waterfall with a Kanban board and shorter deadlines.
Sure, any true Scotsman can see that.
What reason do we have to believe that Agile is so great, if (as you point out) most business aren't actually doing Agile?
I call BS. Yes, the "any true Scotsman" fallacy can happen in a discussion.
But it's also true that some things are presented as X but are not X, and it's not a fallacy to point this out.
Here the parent specifically called attributes of Agile that are there in the original manifesto, but are not adhered.
That's not shifting the definition (which is what a no true Scotsman is about), it's about pointing that a thing doesn't adhere to the original definition it was supposed to hold.
>What reason do we have to believe that Agile is so great, if (as you point out) most business aren't actually doing Agile?
That's an orthogonal question. Agile might be great or bad.
But we can't deduce from businesses doing something non-agile and calling it Agile whether agile is good or bad.
At best we can deduce that Agile has a tendency to be wrongly applied.
Read the anecdotes on the other end. People who have reached the shining city the article describes. It's a nice place to be. There are a few in this thread alone. I've personally experienced it as well. It's great.
> Sure, any true Scotsman can see that.
To continue the analogy, this is roughly the equivalent of declaring myself Scottish because I'm wearing a kilt and playing bagpipes. I have no connection to the country of Scotland presently or historically, but I'm doing the things that Scottish people are stereotypically associated with, so I must be Scottish, right?
At that point, I think it's fair to call my bluff.
Indeed, it's my experience this is often ignored, along with "ditch what doesn't work for you". I've seen countless articles (and job descriptions!) claiming, for example, that "pair programming" is the sine qua non of Agile; if you don't do it, you're doing something wrong. Or "sure, use what works for you, except if you don't follow these guidelines you are on your own".
I, for example, consider pair programming extremely exhausting. It works for me in very short spans and very limited applications; pushed too far it makes me consider switching jobs.
It's fairly obvious that engineers are not fungible, and that an entire process centered around that idea, is not only degrading and disrespectful, but an outright lie that needs to be propped up by illogical thinking, and faith-based arguments.
I'm not sure how you determine what is true, but using empirical evidence and sound reasoning is how I prefer to be presented with justifications for a process change. So far, I haven't seen any research showing that agile is an effective management system.
No, “people and interactions over processes and tools” is based on exactly the opposite.
OTOH, most concrete methodologies sold as “Agile” do seem grounded in the interchangeability of workers (not just engineers, though those are the primary workers involved), but the idea of concrete development methodology (instead of metamethodology that specified how a group adapts dev. methodology to group needs) usable “as is” by multiple organizations is directly contrary to the Agile Manifesto.
The core premise seems like it's pushing a human-centric idea really hard only because it actually removes the humanity from it.
Getting hounded every day to try to justify your existence in a company does not sound human-centric to me.
No, because the Agile Manifesto isn't a slogan for a concrete methodology, it is the explanation of a set of values.
The concrete methodologies come from different sources than (and post-date) the Manifesto; the Manifesto wasn't written to sell them.
It's true that there is a mismatch between the values articulated in the Agile Manifesto and the concrete methodologies that bill themselves as “Agile”, but that's not due to the Manifesto being a dishonest sales pitch for the methodologies, it's because the methodologies are dishonestly appropriating the label associated with the Manifesto.
In any case, the people pushing the methodologies overlap very substantially with the people who originally signed the manifesto.
Most “Agile” methods are fine starting points for adapting a methodology to your team needs, which is the only thing a published methodology should be if one takes the values of the Agile Manifesto seriously. But the even where they aren't actively pushed as rigid, one-size fits all, certification and standardization driven monoliths like Scrum, they tend to fail to actually address Agile application...
Agree that the consulting/certification industry makes things worse. Again, that overlaps with the signatories for the original manifesto.
It was only ever going to be actually implemented in developer-run organisations, because only then are developer problems more important than everyone else's problems.
Maybe that's the answer - more developer-run organisations.
I don't agree. Once you put a developer in a project management role, he starts to experience problems associated with managing projects and using solutions to those problems.
When that happens, the developer ceases to be a developer and becomes quite plainly a project manager.
If you take the business out of the equation you usually get development models that just-for-fun open source projects have.
Waterfall is cool with customers and sales, at least it was if it wasn't risky. So agile is a risk minimation strategy.
In life, we will always find “shining cities” to chase. The perfect house, the perfect job, the perfect romantic partner, the perfect software team.
But, like a mirage, when we reach these goals it seems that the thing we thought was perfect is actually riddled with flaws.
“The shining city” is purely a mental construct, one that gives us structure and purpose, staving off “existential nausea” - the anxiety that comes from living a directionless existence.
The way to everlasting peace is to be able to acknowledge: “we live in the shining city right now. Everything is operating according to the laws of physics, humans follow human nature and their genetics. Everything is exactly how it should be.”
But this isn’t “giving up”. It’s ok for us to have “shining city goals”. It’s human nature. It’s fun and allows us to experience a journey.
But at the end of the day when you see the mirage fade, smile about the journey and the experiences you had along the way.
This is not wisdom, it’s nonsense.
If we followed this advice 2000 years ago we would still be practicing open defecation and dying at the age of 30.
But I did not mean this as an excuse for “non-progress”. Humans will “progress”, it’s their natural instinct and drive. It makes more sense to make our lives easier than to make it harder.
But the shining city doesn’t exist, and there’s nothing that will significantly increase our internal contentment but winning the internal battle of acceptance.
You will need both, change and the ability to see what is good and works in the current reality.
"Task estimation" for sprints never seems to involve any actual data on which to base the estimates.
Usually the conversation with management about implementing agile ends at the point where they realise the upfront costs.
If the person who does it just remembers it themselves and doesn’t do anything to help the other developers then they haven’t completed the spike.
Having that ownership stimulates a much greater level of craftsmanship and skill growth. Documentation is great, but the idea that a spike author and implementor should be different people is awful.
How to do something would be sorted out during grooming or perhaps when adding the story to the sprint.
Do you have other people on the team review this? Or do you just trust that it's complete?
Sorry for all the questions, I just feel like these are things that are never answered adequately.
I’m not sure what I’m used to at my job is correct, so I’m trying to stick to my understanding of how I think it’s supposed to work.
mandatory Dilbert Comic: https://www.flickr.com/photos/lucaminudel/6059269914/
Shame too, as this used to be a really great workplace.
You need to always be "sprinting". There is no time to take a step back and think deeply about the product you are trying to build. No time to think through what a new feature means for your customer and for your existing system.
How often do you work on tickets that are completely ironed out? All the actors of the feature are described, all the interactions with their outcomes, all the things to test for, all the new metrics to add to the system?
And don't hear me wrong. It's probably the right MO for small companies/products. But once you've reached a certain size for the company, the system and the user base, it's not gonna work. People often laugh at how long it takes for Twitter to introduce a minuscule change in their product, but seriously, I wish a lot of "agile" companies would apply this extreme instead of the completely opposite side of the spectrum: "just start coding this shit, and we'll figure it out later".
Someone needs to convert those Jira/Trello/Wall/... items into nice Excel sheets because upper management won't look at anything else.
While you are at it including some nice Powerpoints for the weekly/monthly updates.
And on the remaining time do those development tasks, while coaching junior devs at the same time.
I think most of the developers that are hating Agile are probably more disgruntled with their organizations and their culture (usually hostile to high quality engineering in most places fundamentally) than Agile itself. In a way, I think Agile (similar to a lot of automation efforts in infrastructure / IT processes I’ve worked on) simply brings out the true nature and culture of who is driving, who pretends to drive, and where things are falling apart in team dynamics. The software is usually a casualty of bad dynamics is the implication, not bad team members, but we all know this is not true.
I think the #1 thing that is missed is that the implementing team should have more control over what they work on. Someone writing a bunch of tickets and pointing them out alone reduces team ownership / responsibility and is pretty much how traditional Taylorist work models are built. Agile is fundamentally much more in line with Deming’s ideas giving much more control to the implementors and that is antithetical to at least 80%+ of businesses in the West and probably Asia too.
When viewed through the lens of Cynefin, Agile seems to make sense when you've narrowed down a problem space you want to tackle, but you still require rapid iteration and experimentation to finish defining all the finer points of the goal (the liminal state when moving from Complex to Complicated).
My impression and personal experience of Agile being a bad fit seems to be associated with a misunderstanding of the decision domain, and the general Agile "religion" that has arisen where it's applied uniformly to every situation.
[1] http://cognitive-edge.com/videos/cynefin-framework-introduct...
It helps prevent the sicknesses that come from attempting to do all this up front, where an eager team and an innovative customer try to specify everything up front and nail down a price tag right away.
It should be sold to management as a form of risk management, not as a way to extract more value from a fixed amount of dollars. If you have $4 million to spend, and you spend that up front, you may waste $4 million. If you spend it bit by bit, and have a functional system after every few weeks, that you can migrate to if you like, then you have not wasted any money, even if you end up throwing it away partway through.
I agree with anyone who says the problem with Agile is that it is watered down. It was a least-common-denominator of all the various similar techniques, and I think a lot was lost by encouraging people to adopt Scrum instead of eXtreme Programming.
Nevertheless, I've consulted on a lot of projects and there are environments in which I would not want to attempt to implement any new methodology. The people already hate each other and the system is totally sick. They need something more like Deming's management techniques where they get the top to commit to empowering the bottom, retrain and move out people who are in the wrong job, and get a remaining nucleus of players who trust each other to implement a new method (or their own).
What ends up happening is programmers come into sick environments, run by "Agile" managers, and say that Agile is the problem and if they could just go back to their office and ignore everyone they could get it done. They're probably half right, in that they also need the full support of the customer. Since programmers are a noisy bunch, they complain far and wide about Agile, saying that it's the cause of all the diseases in their world. But sick systems often have names that are misleading. Deming once saw a poster on a client's break-room wall (or something) touting "Total Quality Management" and he said something to the effect of, "I hope that you aren't predisposed to think I'm wrong because of this", meaning that someone had already set up a sick system and claimed it was inspired by Deming.
Also, war can be sustainable, especially if it isn't total war. Look at Europe a few centuries back.
Personal anecdote from working at the largest ISV, which went gung-ho on scrum such that careers called scrum evangelists were created, engineers and engineering managers became scrum masters and the program managers, jobs that should have become obsolete to an extent as scrum mastersv take in lots of those responsibilities, started doing who knows what. But of course anyone saying anything against the scrum was a dinosaur.
They are rediscovering the PMI ad-hoc.
Not because the Agile ideas weren't better but because human and hence organisational dynamics still are and will be the same.
No surprise then, that as Agilists are gaining traction and power they are making their Agility more rigid and dreaming of Chief Agile Officers and roles like that.
Organisations have power structures and position there determines progress and survival.
They have to de-agilize agility to survive. If not they will have nowhere to go, nothing to latch onto when things go wrong
To sum it up: Problem: Missed deadlines/cost overruns and lack of success in fulfilling the actual problem. Solution in theory: The theoretical, engineering/developer, solution to the problem: Less commitment and more flexibility. Basically set the right expectations that we'll drive in the wrong direction and then need to correct it rather than continue because that's what's originally agreed. The "solution" in reality: Less commitment and less responsibility and ownership of the business situation (which hasn't gone away and never will). What happens: Less focus on the original problem. Developers thinking that deadlines and not fulfilling the purpose is even more something that is the problem of managers, and not developers. When shit still goes south because developers are still often not any good at tackling the original problem (although the promise was made by them it'll be better if they run the show), they (as obvious from the comments here) still put the blame on management.
In other words. Developers thought they were better at solving a problem than a person in a role dedicated to solving those types of problems (compared to developers where it's not in focus at all), still blames management when their proposed solution failed. Ironically (but very logically) the least capable are the ones complaining the most and owning the problem the least.
What I believe has caused the situation: An immature industry (due to young age), a complex problem and a lot of demand for developers making companies/managers not having enough leverage to be able to pressure devs into focusing on the right thing/development (the rockstar mentality). Plus of course inexperienced managers that doesn't dare to take control over the situation.
The worst part of it all is that it takes away focus from the actual problem, and hides a lot of incompetence (both on the manager and the developer side) which I find is often a root cause (and my understanding is that science agree with me - which and how correctly you use a specific methodology/process is not strongly correlated with overall success.
PS. I still believe Agile is good in many cases and there a lot of smart developers doing Agile with great results, but there are equally a lot of people who believe they at better at solving problems they are perhaps the least capable of. What's really sad is that many of those people are actually quite good developers.
There used to be a joke, “this job would be great if it weren’t for all the users”. It was supposed to be a joke.