There is no silver bullet for software development process. It takes hard work and flexibility to optimize your team's work flow.
If the project is not done in the 6 weeks you set aside for it, just ship something anyways and move on. It was only worth 6 weeks when you planned on getting it done in 6 weeks, so why spend more? Go ahead and plan another sprint if it turns out what you were able to build in the time allocated wasn't enough, and you really need more. (Put it up against the other opportunities you have, and give them all a fair chance at those finite development resources, something else might also turn out to be more important.)
Just don't spend forever and drag out the deadline, throwing off the timeline for whatever you had planned to do next, because you needed more time for the thing that underestimated how big it was, so it wasn't really possible to do that whole thing in only 6 weeks.
The release that is reduced in scope might be totally insufficient, or, it might be just what your users needed! The knife also cuts both ways – if you're willing to accept that project scope can change in either direction, you can occasionally deliver things early too.
I really like everything about this book.
Project A will bring in $X. You estimate 6 weeks. It takes 12. Project B will bring in $Y. You estimate 6 weeks. It takes 7. If X is similar to Y and you bet on the wrong horse, you've dented and/or sunk your cash flow for no reason.
Saying "Humans suck at estimating" is nonsense. It's a skill that can easily be improved - usually at the cost of some up front planning - and there are plenty of resources to help you with that.
The real problem with Agile is that it gives management an excuse to do no estimation at all. Some alpha-wannabe manager bangs their fist on a desk and says "I want this in three weeks".
Instead of saying "You're insane" teams are pressured into trying to make up for someone else's laziness and lack of contact with reality.
Throw a new type of project or a new client with special needs, and bets are off, Monte Carlo or not.
My general thoughts is that typical software engineering does not have drastically more degrees of project uniqueness than other engineering fields.
I really doubt that. In construction for example, tolerances, materials, requirements, etc are know well before even a single brick has been laid, and if they are to even consider changing them mid-construction, detailed new plans about everything that changed is in process.
That's almost never the case with software requirements.
The same tools, processes and technologies are also used for decades or in some cases centuries totally unchanged. In software a domain (e.g. web programming) can be completely different every 5-10 years.
The core idea I took out of Agile is that once the high-level estimates are completed, the developers are meant to break them down into bite-sized tasks, put estimates on those, and prioritize them in a backlog with the project manager, customers, and any stakeholders. (In my case we're not building products to go to market with, I work in Higher Ed. We're building solutions for campus partners. So it's not really "Project B will bring in $Y," it's "Project will further our mission and better enable some common workflow." Not to go off-topic...)
What's happening instead is, we get the high-level tasks and then break the projects down into bite-sized chunks, then estimate the parts, and are told "great, now make that fit into 6 weeks, because that's all you have."
Developers don't estimate in hours or days, they use points or t-shirt sizes, because honestly, we do suck at estimating. It's not what you pay us to do. (I mean this honestly, it's really not. This is what they pay project managers to do, and they're literally a pay-grade above us. And they don't write code, though they may be good at math and judging t-shirt sizes. The better ones also know how to keep their poker face, when meeting with the customers. But if your project is slated to last only 6 weeks, chances are you won't even have access to a PM. You wouldn't want one if you could get one, since 6 weeks is hardly enough time to plan and later change course. What would they do?)
T-shirt sizes are better than estimates at a sprint level too, because all you're really doing is horse-trading. If a feature is XXL according to the devs, maybe it moves to the back of the queue in favor of something that costs less and is worth more. That's what you do at Sprint Planning.
Back to devs, we're experts at writing the features, and sometimes we can tell you how long it's going to take and get that right, ... and sometimes we can harmonize in concert with the project leadership, but that's not our core competency. It's just moving faster, and doing it better. Sometimes you find out that the specs are wrong after the project has already started, and it's going to take longer.
We know that some projects are going to take a year, and some are going to take two. And those projects get project managers, and they are frequently delivered on time, with all of the promised features in them! Six weeks isn't an estimate, it's a budget. (And it's not a real budget, hardly even registers on the balance sheet. It's more like an experimental timeline.)
The problem is, customers prefer that we blow the budget rather than delivering part of a product on-time, even if it might have turned out to be enough. They're not willing to put things in order of priority, take an honest look at what will take 6 weeks to deliver, and settle for that. They'd rather cut corners, and then we allow them to make those concessions. Only they don't really want concessions, so we wind up having to spend extra time papering over the gaps.
This is not a core idea of Agile. The high level estimates are not part of Agile- fixed scope, time, and cost is the opposite of agile. You are just doing extra fake work around what is a fixed design process.
"Developers don't estimate in hours or days, they use points or t-shirt sizes, because honestly, we do suck at estimating." No, they don't estimate in hours or days because those are commitments that are used against them- regardless of whether they are expected to attend other company meetings or other priorities are thrust upon them. Points or sizes are intended to be complexity or quantity of work estimates, which could take different amount of time.
In addition, estimates, whether of time or complexity, have uncertainty. Many enterprises are unable to deal with uncertainty.
If you fix cost and time as part of the budget, what you are left to be flexible on is scope. This is the most natural place for flexibility, as there is are often a wide variety of implementation choices that limit cost and deliver a low budget version of a feature versus one that would require more time.
In the first edit of this comment, before I submitted it, I said "we don't really do Agile, we do Agile-Lite" – I cut this line because I've heard this repeated by leadership and I've asked what does it mean, and never got an answer. I almost wanted to include it, so maybe I understand now!
Developer teams are Agile. Dev leadership is Agile-lite.
> The high level estimates are not part of Agile- fixed scope, time, and cost is the opposite of agile. You are just doing extra fake work around what is a fixed design process.
Leadership tries to fix scope, time, and cost on the long-term calendar because we want to deliver predictably, to not appear unpredictable or unreliable. As much as I tell upper management that you can't have fixed scope, time, and cost, they persist in trying to fix all of those.
And their budgets and goals seem to win over my objections, damn near every time!
I'm not doing extra fake work, I'm taking direction from senior leadership. Their work may be fake. Your opinion on their fake work, is duly noted, and I'm going to decline to state whether I agree for political reasons (in so many words, I like my boss and I rarely get time of my own to interact with the bosses boss.) I try to do my work and not rock the boat. Which is why I might get a demand to do something in 6 weeks that is clearly going to take 12, and just do the work. "We'll be given the extra time when we need it." That's healthy, even if it's somewhat dishonest.
> No, they don't estimate in hours or days because those are commitments that are used against them- regardless of whether they are expected to attend other company meetings or other priorities are thrust upon them.
That's a fact. But at some level, if you have this toxicity in your culture, someone is going to have to translate from t-shirt sizes into hours and weeks at some point, and when these estimates turn out to be wrong, someone's head will have to hit the block. (Hopefully theirs and not ours.)
I'd like to think that doesn't actually happen here, and also that's not why we use t-shirt sizes. Maybe just why your team uses them. Or maybe I'm just naive about it.
I'm also curious what kind of work you do that you find estimating effort to be reliable, and what kind of tracking you do to keep yourself honest (e.g. I often see people say something will take two weeks and the only reason it happens in two weeks is they end up working day and night for the last 5-7 days when they realize that's what it's going to take to make it happen).
The work splitting algorithm needs to monitor the overall depth of the queue, as well as the number of workers that are online and their propensity for finishing jobs.
It doesn't help to divide a task into 100 chunks instead of only 20, if there are only 8 workers, especially when each split has a marginal cost that will add up again and factor in to overall cost at the point of final integration.
But with a pool of 1000 workers, dividing the work into only 20 or even 100 chunks might still be leaving significant amounts of power on the table undesirably.
You could try to optimize for both, strike a balance between achieving an earliest possible completion time, and minimizing waste. But there are going to be maxima and minima of either on a continuum, may not be practical to find the sweet spot.
One place where this analogy breaks down, is that computers do not care one iota if their work is redundant, or if their work must be thrown away. Human developers care, even if it might save the employers some bit of money or reduce a risk. Humans usually will not produce identical outputs either, when they are assigned redundant workloads. Most humans would probably prefer not to work on a project that is redundant at all. Computers don't care, ML doesn't care. But it would be soul crushing for devs to have to work in an environment like that. I'm sure it happens more than I would have imagined.
Especially prevalent during the initial start of a new project
And yet it's the conclusion every major thinker (and practicer) in our field has come to (plus there's historical evidence from tons of mis-estimated projects, regardless of scope and budget).
So, whether does the idea that it's "a skill that can easily be improved" come from? Wishful thinking? I don't think the universe works that way.
Sure, you can learn to be less widely off. But you can't never learn to be regularly correct at estimating, or even close within a small margin.
At best, what happens is that quality get shitty and corners get cut, and you get the product at the correct deadline, but it's not the same that you wanted to ship before development started (even if the features are nominally there).
I've come up with two names for this kind of thinking, one a bit grumpier than the other.
The less-grumpy one is the "Is From Ought Fallacy": It's well-known in philsophical circles that going from "Is" to "Ought" is wrong; that is, saying "something is this way, therefore, that's the way it ought to be" is a terrible argument. Well, when you flip it, it gets even worse: "Things ought to be this way, therefore, that's the way they are" isn't just a bad argument, it isn't quite sane. Yet look around you, and see how often it's used. (Hint: Abstinence-only sex ed. Hint: Drug prohibition.)
My grumpier version is "Arguing With Reality": You can argue with people, whether you define "arguing" to be a screaming match or a logical defense of a thesis. That works. It's sane behavior. Going out and arguing with the rain that it needs to be sunny now... isn't. Rain's gonna rain, people are gonna be bad at time estimates, and making logical arguments about why the real world shouldn't be like that doesn't change the real world one iota.
Maybe they're just variants of the "Utopia Fallacy", or the perfect being the enemy of the good: "If the world were perfect, it would be like this, therefore, doing anything else, even if it gets good results, is imperfect and compromising perfection is wrong." Yeah, pull the other one.
I agree fully and I think the main issue is this: People estimating, wanting estimates, "fueling" the workplace and ways in which businesses operate where estimates and milestones are the norms we work around...
The big thing I see missing is the fact that mostly everyone involved in these "mental work simulations" (as I'll call them) is calculating all of the tasks for completion against time it takes to complete tasks.
There have been some people commenting on, "I am glad to give an estimate so long as I'm allowed to add ample padding time for unexpected issues, scope creep, and then I'm allowed to double the entire estimate or at least add 75%. If I can get that estimate signed off on I'm fine with estimating".
And that's probably where I'd say I sit and feel most comfortable: "I'm fine with giving estimates so long as the person singing off on them realizes I'm really just giving a best case scenario guess with a bunch of worst case scenario 'what-if's' built in as well as some 'inevitable scope creep / customers deciding to change specs on the fly' type stuff." But something is still missing. There is a lot of lingo and business-speak and cross-department buzzword ad-libbing taking place but not a whole lot of ground is being broken in terms of moving things forward. Why is that?
The big thing that's missing from ALL OF THIS (IMHO)? Everyone is basing all of these estimates as if things were taking place in a vacuum. Even in that vacuum, over time, estimators, project managers, and developers have come to realize that including padding time for inevitable issues, scope and feature creep, and unforeseen circumstances is an absolute must. Even then, that's inside a vacuum of sorts.
I truly believe it's impossible to plan without falling prey to the "Estimate inside a vacuum" issue. Even if you are a driveway paver and you've paved 10,000+ 100' driveways in your illustrious 20 year career there is going to be something unforeseen that comes up that will prevent your 10,001th job from being completed under time and under budget. The problem is that there is no way for us to truly determine any and all possible issues that could arise. The bigger issue is that bosses and clients don't tend to care when money is on the table, jobs are at stake, and careers are on the line. So we create estimates and we miss them and promise to get it right next time. Even though we know we never will, vacuum or not.
Not sure if we think about different things when saying "estimate", but I rarely was able to estimate more precise than "6 to 12 weeks". Unless I am doing something absolutely straightforward, I never really can give good estimate (and since I am forced to do - I just give some random number with a lot of padding which I "feel" I should be able to fit in)
In my practice so far to give good estimate means doing 80% of actual work and leaving these straightforward bits to finish it off. These last pieces are easy to estimate usually, but not before main work is done.
lie to ourselves about our true capabilities
lie to our boss / team about how good we said we were
lie to the business about how well spec'd the job is, about how the current architecture matches their known requirements, lie about how their stated requirements and the real requirements match up.
Lies to avoid punishment, lies to gain a job that otherwise would go to competition, lies that simply hide the complexity of every job and estimate
without lies estimating would be easy
Being charitable, I'd prefer to say that mistakes were made, without pointing the finger. We're all not perfect, and pretending that we can all be perfect all the time sets the wrong expectation, so I wouldn't try to pin it on a lie.
And I'd prefer to not have to see my boss going under the bus when I have to say "the reason we're not done on time, is because the boss told you this deadline and list of features was reasonable on that day without perfect information. If she had come to us and asked for our opinion, and given us time to talk it over, we would have worked it up and told you, it couldn't be done without making some concessions!"
If only for the reason that, perfect information is not usually available in the time allowed for making a decision. That's not just being charitable to the boss either, that's also self-preservation. The boss can take a lot more of those kind of hits than I can. Even more if we stay together on issues like this. And extending the time needed to make the decision also can have its own opportunity costs associated.
I agree with your sentiment in general. But not every bad datum that costs in accuracy, is the result of a lie. I've seen it said frequently that you should work to defer every decision until it's absolutely required, so you can make it with the best possible information and avoid those costs.
I think that's honestly a lot easier said than done.
A good leader is able to create a safe space where most people will release their defensive positions and many of the most intractable lies will fall apart.
But we all do lie, most often to ourselves, and not always with evil intent
But I fear the attitude that says "we expect you not to lie (withhold information, fail to speak out) because to do so will hurt the organisation, and you should have the best interests of the organisation / team at heart
Humans don't work like that.
I can't break project down to 240 units of time on the spot. even 4 hours is still crazy tight, and produces poor estimates. that's barely a minute of thought for each hour. occasionally a manager will be comfortable with me staring off into space for a couple of days. but that's not super common.
I think anybody can come up with good estimates. Almost nobody is willing to say, Take two hours and figure out how long it will take to estimate Project A and Project B.
And yet, we do. Look at the number of failed government contracts in recent history:
https://warontherocks.com/2014/12/top-10-failed-defense-prog...
And while much of that is about the failures in program management for large contracts, the government aims to build large things at once (as opposed to an MVP and iterations).
When a critical dependency takes longer than expected, the cost snowballs down the chain.
Of course, it may seem to you like the majority of the work is done in the case of a prototype. This is not the case - a vast majority of the work that goes into developing a jet fighter consists of the documentation as to how fast it can fly, its flight ceiling, etc.
Maintnence manuals, and developing a supply chain to ensure we can have a resilient delivery despite war or changes in the economy.
Training programs, to ensure that pilots are able to safely learn to fly the thing
Testing, which of course requires that a prototype be built in the first place.
> The real problem with Agile is that it gives management an excuse to do no estimation at all.
> Some alpha-wannabe manager bangs their fist on a desk and says "I want this in three weeks".
> Instead of saying "You're insane" teams are pressured into trying to make up for someone else's
> laziness and lack of contact with reality.
Don't blame bad managers and bad companies on Agile. Agile needs to empower the people doing the actual work, or it's not Agile. Managers need to facilitate, not bang their fists.
This is completely dependent on what you're doing. If you're implementing yet another CRUD app, then sure, you should be able to estimate that to within 10%. If you're developing something a little less predictable, like a game or something for R&D, then estimates go right out the window.
https://web.archive.org/web/20170920093850/https://www.const...
This is more like an issue tracker/project planner with statistical modeling capabilities built-in, I don't know if it's doing the same thing under the hood, this is part of their "special sauce"
On my old CMS team and with my current development team, we use kanban because it aligns with how we function. We get ad hoc tickets in our queue, we pass them through a triage lane, then they're picked up by data, dev and qa as they're completed and within the items allowed with our WIP.
Waving a broad brush to say x is what everyone uses, will fuck everyone up. Neither agile, waterfall nor kanban are the ideal solutions for everyone.
Anything that doesn’t have a Why next to it is essentially carrot dangling. You can have targets/forecasts when you expect to have something ready while making it clear that these are guesses, but if it’s not real then it creates real problems if you pretend that it is.
How would you force your manager into converting that "why" into one you would accept?
All that said, client work is the one place where sprints serve a real purpose. The scope doesn’t expand because you finish early so there is a built in reward, agreed upon price for the scoped work, and a built in penalty.
Only sometimes, we could really have been done with them, without necessarily finishing all of that pet feature which has thrown off the whole timeline, by being more complicated to implement than anyone was able to imagine upfront.
Scrum by the books usually implies that releases happen after the po or the customer has accepted the increment at the sprint review.
I find a good sprint rhythm stops the constant task shuffle. When business knows the dev team is mid-sprint it forces them to think about priority.
We do CI/CD, and only big items (in reality UI/UX changes) wait until a sprint review to go out.