Hacker News Reader

Comment by commandlinefan | Hacker News Reader

> The buyer has to make commitments

Not surprised this discussion devolved into this bizarro world point so quickly. The developer says “there are too many unknowns to provide any sort of estimate”. The manager retorts “but we need estimates!” As if they expect the developer to say “oh, you need them? You didn’t say that. In that case, the unknowns are now known and it will take four days, three if you remove this inconsequential detail that nobody cares about”.

Unfortunately this bit of illogic is so common that I know it’s a waste of time to argue, so I’ve learned how to figure out what they want the estimates to be, give them that, miss them (just like everybody else does), and then get to work on actually providing the feature.

View on HN

This is why it’s a good idea to have ranged estimates. If a simple known thing with zero unknowns has maybe a 200% margin (1-3 days) then a more unknown task can have 10x as wide margin like 1 to 20 days.

As a developer I never get “that’s a useless estimate we can’t prioritize based on that/tell the customer that”. It’s a perfectly good estimate. It tells everyone there is a lot of uncertainty. The low lower bound indicates there is a chance it might turn out to be cheap (so maybe worth taking a gamble and trying it and giving up). The big variance indicates it could/should be given some time to prep/investigate in order to narrow the estimate.

When there are unknowns, make a wide estimate. Convey the uncertainty. If the unknowns are ridiculous, make a ridiculous estimate. Even “5-200 days” is useful because something else with less uncertainty can be chosen instead.

I agree with this sentiment.

I'm running through a similar situation now, where I'm somewhat adjacent to a client and a dev team. The client has a list of tasks he wants done, and he'd like banded estimates for the effort the dev team thinks it will take to complete each.

The developers are obliging, but one of them complained to me privately "why doesn't he just give us 2 months of budget and we'll get through as much as we can?" - which is an interesting statement. From the developer's perspective, this is ideal. They are very good and work hard and have integrity, so I'm sure something like that would work, from their perspective. They can just 'go' and get stuff done.

From the client's perspective though, it just raises a bunch of uncertainty. He wants to know what he could get out of his 2 months of development budget. If one task took all 2 months, and the others (15 in this case) didn't get done, is that OK? What if some important task wasn't that much effort? You'd potentially want to priortise that over a bigger item.

Just saying 'give us two months of budget' doesn't take the other party into account, and what they're trying to balance or plan for. I've sat on both sides of the table, and honestly just asking for a budget with no actual commitment to completion feels selfish in my opinion. Discounting their opinion because they're 'business' people and not developers, or treating them like an enemy to fight against won't win you any friends or long term relationships.

> Just saying 'give us two months of budget' doesn't take the other party into account, and what they're trying to balance or plan for. I've sat on both sides of the table, and honestly just asking for a budget with no actual commitment to completion feels selfish in my opinion. Discounting their opinion because they're 'business' people and not developers, or treating them like an enemy to fight against won't win you any friends or long term relationships.

The irony is that at least some of the time, a "safe" estimate will actually make the project take longer. Work naturally expands to fill a deadline, and not having a deadline but instead frequent progress checkins with an expectation of foward progress can result in more getting done, because it changes the conversation from "how at risk is the deadline" to "how much happened in the last week and what are the next steps", forcing progress to happen continuously rather than one big bang at the end.

I also often hesitate to give estimates / establish deadlines because it tends to create an unhealthy confrontational atmoshepere, where developers are incentivized to not incorporate feedback or make any changes even if they are the right thing to do, because it would put the deadline at risk (but if they incorprate allowances for these things during estimation it's considered sandbagging).

> a "safe" estimate will actually make the project take longer

That’s probably the most important thing I’ve taken away from software project management techniques - everything they do is hurting them, not me. If somebody insists on shooting themselves in the foot, and they’ll shoot you if you try to stop them, you just sit back and let them shoot themselves in the foot and shrug your shoulders when it happens.

It’s easy for this honest mentality to turn into malicious compliance. If you’re actively fulfilling that prophecy, this won’t just impact a manager or a PM, it will absolutely impact you as well. I’ve seen it happen many times, obviously this goes beyond just being wrong or missing deadlines but eventually it will come back around to bite you.

At the end of the day, these estimates are helpful but maybe not in ways that are intuitive from a developer’s perspective. They’re useful in weighing priorities, judging the uncertainty, and identifying where to fill the gaps in understanding, time-boxing research and scope (e.g. do we really need to spend 6 weeks reading release notes?), and help communicate to outside stakeholders. We see the successes of this all the time with big game releases, product launches, movie releases, etc. It’s a high level measurement of progression and devs need to be a part of that planning process. It gets easier for senior engineers to do this efficiently with experience. It’s not necessarily an exercise that a more junior engineer will a) get right consistently or b) think is valuable at all.

In my experience there's two main reasons that orgs outside engineering / product want estimates:

1. Coordination of external dependencies. Stuff like product marketing, sales training, documentation, etc. as well as sometimes things external to the company like customer commitments or sales commitments. These are for the most part legitimate, although in many cases the deadline doesn't need to be known at the outset of development, and too many of these is a very unhealthy sign.

2. Accountability / improved productivity - This usually manifests as an exec feeling that engineering is running too slowly / inefficiently, and by asking for deadlines (and often compressing them to "push" the team) they can improve productivity. These are almost always in my mind completely counterproductive, and there's much better ways to accomplish this.

> From the client's perspective though, it just raises a bunch of uncertainty.

It may feel good to have some of the times quantified. But... If your estimates are literally fiction. It won't stop being fiction in front of the client. You should really spend that energy that went into the estimate for something useful instead.

Not to say you can't derive some useful information from a short planning session. Just don't expect anything other than thumb to the wind guesses.

> When there are unknowns, make a wide estimate. Convey the uncertainty. If the unknowns are ridiculous, make a ridiculous estimate. Even “5-200 days” is useful because something else with less uncertainty can be chosen instead.

As a PM, I couldn't possibly agree more with this. When you give me 5-200 days instead of "we can't estimate this", you've started the beginning of a productive conversation. I can ask you questions about why the range is so big, and you can explain to me what steps we can take (research, narrowing scope, etc.), to reduce the range.

I’ve tried this. Inevitably the response is: “We need to book UAT testers for a specific date.”

Ordinary LoB staff apparently are superstars that have their schedules locked in months into the future, but consultant SREs costing thousands a day can have their time wasted on a daily basis trying to determine how long it will take to complete a task that hasn’t even started yet.

This has literally been my last two months. The dev team lead is asking me — an outsider — how long it’ll take his team to finish something. Not because I know better, but because he doesn’t know either and prefers to make it my fault because I’m expendable.

Well, analysing the situation that seems like quite an easy one to negotiate through:

1. Talk to the person who'll get angry at you when the UAT testers are brought in at the wrong time.

2. Make it clear to them, with a paper trail, that this is an impossible task and you can either book in a best-estimate with the likely outcome that the UAT testers will arrive and there will be nothing to test OR that you can book conservatively and there is a risk the project will be delayed waiting for UAT. They can pick which risk they want to bear.

3. Put in formal feedback, as loudly as possible, that booking in UAT for a specific time introduces obvious risks and someone needs to optimise the process to be more flexible otherwise it will delay projects & waste the businesses' money.

Deadlines are a reality in all businesses (Real deadlines or made up ones). But if you have a thing to create that takes 5-200 days what will you do?

You can pull a different estimate of 20-40 out of your behind when pressured to narrow it, and make the deadline 40 days. That might be a 50/50 hit/miss the deadline, with a worse case of running over to the full 200 day worse case (or even worse - even the 200 was a guesstimate). In some cases that's fine too. The decisionmaker might just want a date with 50% chance of hitting because it's better to land a sale and apologize for the delay, than to not land the sale and go bankrupt.

The correct course of action though is to not set the deadline in that meeting. The 5-200 day estimate is a way of explaining "Either don't take this on at all, or at least don't try to schedule anything urgent based on it".

In that case as a developer I'd suggest "Let me look at this for one afternoon, and I can return with an estimate that's 1/10th as large. Then let's meet again ad prioritize some work, schedule UAT and so on". PM's often look really surprised when they realize that all it takes might be to check if there is a functionality available already or if it needs to be implemented - which represented the difference between 5days and 200. If I go off and find it's already there, the estimate is 5-10, if I go off and find there is nothing and it has to be built from scratch, it's 150-200. But without that knowledge: 5-200.

Or, if a 20-200 day estimate is actually due to real implementation uncertainty, I'd suggest "This is potentially large and uncertain. Let's try to break this apart if necessary into something that has a 10-20 day estimate and some parts that have a longer horizon. Schedule the delivery/UAT of the first functionality in 20 days, and we'll by then have a good estimate of when the remaining work will be done".

If someone in my organization would try to convince me to pull a stupid estimate out of thin air, then pressure me to haircut that estimate, then schedule something based on it (such as a delivery), then pressure me to work overtime to meet it, then one of us would be leaving the org very soon.

An interesting swap would be to get them to give an estimate for when they'd be ready for UAT, and then you take on booking the UAT.

If you're the user or representing the user, shouldn't you be doing the UAT?

Aren't we "agile" now? No UAT testers should be stuck for longer than a sprint's worth so just work it out closer to the date?

> what steps we can take (research, narrowing scope, etc.), to reduce the range

I quite literally quit my last job in February because of this shit[0]. Stop. No. These are not useful things for developers, much less senior developers, to spend their time on. 95% of my job ended up being coming up with estimates. My top deliverable to my management chain was estimates. They would only care about the actual development work insofar as it validated or invalidated the estimates deliverables.

Hard facts: If time is such an unimportant resource that you're willing to waste most of it doing "research" and worrying about scope, then you don't actually need an estimate. For reporting purposes, just BS one yourself, that's literally your job as a PM.

[0] It was well-paying, I was basically fucking around every week doing whatever I wanted, yet this issue alone outweighed all of that and was enough to make me quit.

The SO question addresses this, though: after giving an estimate with a range, the developer is pressured to narrow that range, mainly by pulling in the farther-out date in the range, and then the developer is blamed when the project inevitably comes in late because of that.

Maybe the solution here is "stick to your guns", but that can have negative career implications.

That’s perfectly expected. But the key is: narrowing the estimate isn’t free.

You can’t pull a different estimate out of thin air. And zero people will expect it.

The developer needs to say “I can say 10-100 days right now in this meeting, but given just 1 day of investigating, I’ll narrow the estimate range by an order of magnitude”.

I'm fairly certain that you aren't accounting for uncertainty properly, especially with long-tail events. Let's say you have a ticket to enable a config flag. That's low risk and relatively small. There's still a risk that there's the flag hits a silicon bug. Does your estimate reflect that risk and is it useful if 95% of tasks don't hit those issues? Consider also that estimates delivered by contractors or even other teammates will not reflect these risks and thus you look correspondingly pessimistic or even lazy.

For context, my team is in exactly this situation with several silicon bugs we found last month.

Assuming you’re pretty close to the hardware? As a standard backend web service dev os/library/device bugs are incredibly unlikely and usually defective application software is to blame. Not saying it doesn’t happen, but I can’t recall in recent memory this sort of thing happening as high up in the stack I work.

That's just the sort of stuff I see, but long tail issues that take weeks or months to diagnose can happen anywhere in the stack. Sometimes it's a hardware issue, sometimes it's a compiler bug, sometimes a VM bug, a missing browser feature, etc. Any of that can crop up even with "trivial" changes. I'd argue that it's not useful to include that in your estimates, but they're nevertheless possible and need to be incorporated in scheduling.

May I ask how you identified them as silicon bugs? And by silicon bugs do you mean random bit flips, or something else?

I probably should have said hardware bugs, as only one is in an IC (specifically an IP block). It manifests as small (but out of spec) timing errors across a network. The rest are things like improper layout, faulty components, wrong value passives, etc.

Yes - it's better to be accurate than precise. If you said "1 to 20 days" and it took 15, you were accurate. Being too precise gives you no space for the unknowns to show themselves.

> The developer says “there are too many unknowns to provide any sort of estimate”.

You are misunderstanding your job. You are a subject expert and paid accordingly. Answer with what the unknowns are, an optimistic and pessimistic take on them, what you could do and how long it would take to lift them.

If you don’t, your boss is going to ask you for exactly that in a roundabout way and micromanage you because you are forcing them to do your job.

It still baffle me than approximately half of developers don’t understand things which are obvious to any seasoned blue collar worker.

Thank God somebody said this. When I was working as an IC, I would be very communicative, highlight the unknowns, and get buy-in from my boss. If a surprise came up, I would make sure everyone knew the game had changed.

As a manager now, I get scoffed at if I even hint at a deadline. Excuse me what? We have a limited budget, a limited runway, and we need to prioritize which product bets we are going to make. This seems entirely logical, but a lot of devs I work with now find it impossible. They just enjoy tinkering away with whatever thing interests them, and delivering value only means what they personally find valuable.

Personally, I find it a bit pathetic and childish. I say this as a long time software engineer who knows perfectly well how this business works.

Will Wright had a quote on this … if you don’t have any constraints you are just playing.

That said, however, The problem I run into during practice is dependencies— people want estimates to integrate into systems you have never heard of or used before. Not only that, there is no in place architecture or design before estimation. Imagine a construction company estimating a house without knowing how big it is or without having blueprints.

Often times software development is designing and architecting the system — not just implementing it. And that is the expensive part.

Then you communicate!

Say that the estimation can depend on the complexity of the system. I would think worst case is when the system has these A, B problems, then it would take N time. Otherwise best case it could take M time. If we want more accurate idea, we can spend O time on it to figure out what it could be.

I would say this is being negligent if it is generalized. Often times architecting a system involves meetings with multiple parties — stake holders , legal counsels, and months or years of planning ahead to give a proper estimate for a large system. Time and materials is the right way to do this. However I can see this working in a very small scenario.

> Personally, I find it a bit pathetic and childish. I say this as a long time software engineer who knows perfectly well how this business works.

How does this business work?

Why is software special? This happens the same in construction and other industries. The estimates usually make 0 sense and there are often huge delays. Same as store openings, film production, etc.

There's a problem with tinkering and devs going off in the wrong direction but on the flip side when people say that something is done in an hour or a day often never happens. Managers and agile practices that want tasks broken done to the smallest atom just messes things up. To make things work product often never has accurate requirements to that detail either so that complicates things even more. The requirements change on the fly.

Blue collar vocational training is more mature than software development vocational training. It's typically taught by people who have professional experience, and there may even be an apprenticeship period where people are explicitly learning from an expert in their profession while on the job.

In software, on the other hand, we often have most training being taught by professional academics who do not understand and are not qualified to teach the parts of the job that don't have a straightforward academic analogue. Combine that with a business environment where things like budgets and cost control are often only theoretical, and all but the most gross liability for defects due to poor workmanship has been waved away by EULAs, let it marinate for several decades, and... yeah. Not only do people not know these things, but I'm not entirely clear on how or why they should be expected to learn them, either.

> You are misunderstanding your job. You are a subject expert and paid accordingly. Answer with what the unknowns are, an optimistic and pessimistic take on them, what you could do and how long it would take to lift them.

Add a small feature to an existing system. This estimate can be anywhere from several months to a day. Most of the error bars are org and system specific rather than anything to do with the feature. The exact same feature built by the same person in different orgs and different systems can be opposite ends of the range.

Seldom do people have the contextual understanding to provide an accurate narrow estimate, particularly since both org and system are in flux. They change constantly. Unseen landmines potentially await in both.

> If you don’t, your boss is going to ask you for exactly that in a roundabout way and micromanage you because you are forcing them to do your job.

And such orgs take forever to get anything done.

> It still baffle me than approximately half of developers don’t understand things which are obvious to any seasoned blue collar worker.

Software is probably the hardest kind of engineering you can do. All the system constraints are virtual. Just consider comparing a dirt ground for consistency and behavior vs libc... 304 stainless vs react.

You can go deep in both, but a shallow understanding of one takes you much much further than a shallow understanding of the other.

> Add a small feature to an existing system. This estimate can be anywhere from several months to a day.

That’s extremely untrue in my experience. Sure, hard to interface with systems are a thing and organisational roadblocks happen but an onboarded senior team member can tell you what’s going to take days and what’s going to take months or they have no business being senior in the first place.

I feel like my answers in this discussion have been particularly harsh but I would really like people calling themselves software engineers to start acting like some. I think my view on this subject has been strongly shaped by spending most of my career doing software development for industrial companies: people who claim software is particularly complicated just don’t understand and respect how complex the rest of an industrial project is.

an onboarded senior team member can tell you

You're making an unwarranted assumption here. There's many pieces of software that have no development team at all. There's no one to ask; the last person to touch this software left the company three years ago. All they can tell you about it is how they use it now.

It's likely there's at least someone that can explain their business process to you, but identifying them and receiving enough of their time to piece the picture together is another story. And maybe, if you're lucky, that person can tell you what the business process was at the time the software was written, and what has changed since then.

But this process can take weeks, and most of the time you're expected to provide an estimate before even talking to anyone from operations.

Sure, for internal software with an active maintenance/development team, informed estimates can be given with a relatively high level of confidence. But as an outside consultant working with non-software companies, all estimates are completely made-up based on your reading of the buyer's budget: there are no other data points to base your estimate on.

Working in the field, I’m fairly certain you already know that this kind of job always starts by a paid diagnostic and scoping phase which is there to give you the elements you need to do an actual estimation. It’s either that or you are lucky to still be solvent.

There are lots of places where you get 0 of that before providing an estimate.

And yes they are still solvent.

As others mention there are lots of unknowns in an organization.

I've been in places where the other team involved is across the globe in a different timezone and isn't even actively willing to respond and yet we continue...

There are also cases where lots of parts are outsourced and you're lucky to get a reasonable response i.e. changing part x requires vendor a, b, c & d, which have an SLA 1-8 weeks and so you're already starting 4-32 weeks without even working anything else out. If you're really lucky it could still all be done in 1 day.

Not sure if you're harsh as you say or just lucky. Not everyone and every team is well organized.

> Answer with what the unknowns are, an optimistic and pessimistic take on them, what you could do and how long it would take to lift them.

Yeah, most of the time my boss just needs to know if this job is a few hours, a few days or a few weeks. I never answer with exact numbers, but a range. If there are some major unknowns I let him know. If I made some important assumptions then I let him know.

Sometimes I'm wrong. Most of the time my estimates are sufficiently accurate.

Sometimes the customer really needs this done before some deadline. Then it often turns into "how can we make this happen before then". After a bit of back and forth with some added assumptions and cutting functionality, we usually end up with a solution that can be realized in time.

I mostly agree, but if it's a technology I've never worked with and I'm asked to do something I've never done before, I'm not an expert. That doesn't stop them asking for estimates, which turn into deadlines.

estimates, which turn into deadlines

Well, that's the problem right there. An estimate is not a promise. It should not be treated as such.

I just say that to my boss: "this would involve using a library/tool I've not yet used, so I cannot give a good estimate until I've spent some time digging into it. Right now I don't know if it will be easy or very difficult."

If he really needs an estimate he'll tell me to start digging into it right away, and I'll let him know after a bit of time how it's going. If there are still large uncertainties and my boss still wants this completed by some specific date, we might try to explore alternative solutions.

Yes it boils down to, if the person asking for estimates understands the problems with them, it's safe to try. With bad managers it's better not to give them as they will be abused.

In the above case, I didn't get that time I would need, they wanted the estimate to include that time I would need to dig into it. But that's above my estimation powers. I didn't even know if what they asked was possible at all.

They went and asked someone else who did guess some number of weeks, but the work never got priority so we still don't know.

Did you say what you are saying here?

Did you say that

1) Without knowing further best case is it takes N time, worst case M time, and if there are certain obstacles it could be completely unfeasible. 2) To get more accurate time and to understand whether it's feasible I would need O time to figure that out first, and I can give you the more accurate estimate then.

This seems unexpectedly optimistic, to me. I'd think: best case = it's already done; worst case = it's impossible, but we don't know that until we've spent too much time and money on it; the "certain obstacles" are almost always uncertain or completely unknown; and the "O time to figure out [how much time it will take] is also unknowable until the "figuring" is already complete. Now, if this were manufacturing with well-known processes, then I feel I could use your formula.

Yes, of course. To the bad managers it sounds like just yet another excuse, and they ask other devs until they find one who gives a number.

> they wanted the estimate to include that time I would need to dig into it

Yeah that ain't gonna fly with me.

As a manager that's all I ask. Give me an estimate of the knowns and an estimate on how's long you need to figure out what is realistic from the unknowns. Just let me know what your next move is.

The problem is when we assume everything is a known. I try not to do that and I try to set that expectation with my manager. If everything had an easy answer we wouldn't need highly paid problem solvers.

That's a special case then.

You mean we're all a special case :)

My optimistic/pessimistic takes at my last job would look something like "roughly a week to roughly 2 years"

Depends on how deep the yak shaving goes, and you wouldn't know until you got halfway through fixing the problem.

Often I could take an educated guess as to which side of the scale it came down on, but I've definitely started pulling on a string inside of the codebase and it just kept going and going and going. All of those baked-in assumptions that other developers unconsciously made over the course of a decade can pile up into a mess that you just can't implement the feature/bugfix without doing a pile of potentially breaking changes that need to be shipped in major versions.

Generally if I worked the problem for a week then I could give you a much better estimate of when it would be done, and often that estimate was "it's done now" after a week. But if I hadn't looked at that particular subsystem for a few years, I couldn't trust my memory of it up front and had to do exploratory surgery first.

(Really exploratory surgery is a much better metaphor for what you're going to be doing, and surgeons can routinely find surprises -- "oh look that barely perceptible shadow on the x-ray turns out to be a stage 4 tumor...")

Yes, the codebase was horrible, yes it needed to be cleaned up, no we didn't have the manpower to do that and it would have taken me 10 years to rewrite it all.

Quitting of course was an excellent solution.

> Generally if I worked the problem for a week then I could give you a much better estimate of when it would be done, and often that estimate was "it's done now" after a week.

You're lucky if your management chain allows you to work on problems for as long as a week without giving them an estimate first.

Can you do it? Provide an estimate that's accurate and stick to it with the routinized scheduling of a blue collar job?

It's not an apples to oranges comparison. You give me an architect's blueprint and guys who build to the legal building code, and I could probably get a house built. Expensive and painfully, but it can be done.

I don't even know what the coding analog to an architect's blueprint is. They are fundamentally different jobs.

> Can you do it? Provide an estimate that's accurate and stick to it with the routinized scheduling of a blue collar job?

Yes, I work as a delivery lead nowadays. Producing and respecting estimates for software projects is literally my job.

I routinely fire developers who pretend they can’t do estimate and strangely the ones I keep are actually not only able to do it but good at it.

You just made two contradictory claims: (a) that your job is to produce estimates; (b) that the developers who you work with are the ones who need to "do" the estimates.

I've met your sort before. You're a net waste of resources in an org: instead of developers getting things done, they're dealing with you asking them to do your job for you, i.e. produce estimates for the things they'll never actually get around to doing (because of producing estimates).

Your username would be accurate if s/My/Everyone's/

I produce estimates and coordinate eight concurrent projects involving ~70 people in cross functional teams. At some point, the actual domain experts have to give me their inputs so I can consolidate and prioritise things. Nothing contradictory here.

If it helps you sleep at night, sure, assume I’m a net waste of ressource. Good luck fighting with the other seven teams for some time with the shared senior UX expert.

Are you sure you’re not mistaking guess=deadline mode for actual estimating? And selecting among those who can meet deadlines.

If you rarely eat your time buffers, great. If you do that often, maybe you just coinflip-fix deadlines this way and maneuver between them under the guise of successful estimation.

It's more likely than not that everyone is giving the wrong estimates i.e. just claim things take 3x as long, skip out on anything too hard and produce lower quality.

You always get what you ask for. Like sure I can estimate. If you ask for a web server you'll just have that. It won't be secure. It won't be fast. It may work. It will certainly pass the tests you think of.

Is that actually good? No. The real estimates get rejected. Everyone cuts corners. When something does happen e.g. a hack occurs the engineers not these managers get fired :(

Fair enough. Guess I should admit you squarely got me.

See in this case the manager has done a bad job, but the developer has also done a bad job, both in the exact same way.

The correct solution here isn't just to insist estimation is impossible and be offended that someone would ask for it - that just misunderstands how businesses work.

The correct thing is for either of those parties to say, "Okay, I understand it's not estimable at the moment. What are the steps that we need to take in order to get a reasonable estimate?"

A good PM will do this, but where there is a bad PM, a developer can step in and say the same thing. In your scenario, "The developer says “there are too many unknowns to provide any sort of estimate”." What the developer could say is, "There are too many unknowns to provide any sort of estimate - let me give you a few examples. <examples>. What I can do is come back to you after <amount of time> with a list of the unknowns and an explanation of the work required to evaluate them, so that we have enough information to estimate. Once you have that list, then you can decide if we want to proceed with the work to understand those unknowns."

I totally understand why developers get frustrated when they're asked to give estimates for things that can't reasonably be estimated with the current information available. But the flip side is it's also very reasonable for PMs to get frustrated and ignore developers who insist that estimation is an impossibility, because that's virtually never true - maybe it's impossible to do now, but there is a path to make it possible. If you work proactively to expose that path to the PM, it'll generally result in everyone being happier.

> There are too many unknowns to provide any sort of estimate - let me give you a few examples. What I can do is come back to you after amount of time with a list of the unknowns and an explanation of the work required to evaluate them, so that we have enough information to estimate

Sometimes we don't know what we don't know. If we don't know what we don't know now and if we don't know what we'll find we don't know later, then we don't know how long we will need until we'll know.

Uff. Sorry. If I had more time, I would have written a better comment.

Let's put it this way, then - either you're totally incapable of building whatever we're talking about under any circumstances, or, even if it has unknowns, there are steps you can take in the right direction to get started and move things along. You will learn things that will help you understand the problem better along the way.

If it's the latter case, then a good developer can describe the immediate steps they'll take and what questions they expect to answer. Whatever the case, there is always some concrete next step you can take to which you can assign a reasonable estimate of time. That might just be that you'll spend one day perusing some documentation, and the only outcome you expect from that is that you'll develop a reasonable enough understanding of how large/complex the documentation is, such that you can come back with an estimate of how long it will take to find and carefully read all the relevant sections of the documentation.

That's totally fine - if it's the sort of wildly unknown problem that you're describing, then I recognize we'll have to take steps to understand the problem. There's still never a case where it's impossible to create an immediate plan of action and understand roughly how long that will take (unless you are simply unqualified to take on the problem).

I had a boss loved the phrase, "a date for a date." He just meant that if you can't give me a date for when the thing will be done, then you need to give me a date for when you can give me a date for when the thing will be done. Can't do that? Date for a date for a date. However many levels we have to go, you're not getting out of whatever meeting we're in until you can give me a date by which you'll be able to come back and give me another date.

The problem is that in programming the discovery part is often 90% of the task. There can be situations where there is no 10% discovery phase and the rest of the time we’re banging out predictable code.

Right now I’m replatforming a legacy app. It’s undocumented, developed by dozens of staff — all of whom are gone, and full of crazy bugs and gotchas.

Out of the box it wouldn’t compile, the source code was missing files, etc…

The task is to find every such hidden issue, fix it, which then allows us to find the next hidden issue, and so on.

When we’re done finding issues, we’re basically done.

Can you tell me how many issues we’ll encounter? I can’t. I’m being perpetually surprised by the code base, finding random binary blobs that actually are from some other project we didn’t even know exists until that moment.

Fundamentally, it’ll take as long as it takes. The application is mandated by law, and it must be made to work. It’ll cost what it costs, and it’ll be finished when it’s done.

Instead of begging for estimates, a good manager would be bending over backwards to accelerate the process. Get in the original devs for a week under contract. Procure tooling that helps find bugs faster. Instrument the source servers to get better comparative info. Etc…

I’ve never seen that occur. It’s always just the same question repeated over and over as if that can make the unknown known.

> The problem is that in programming the discovery part is often 90% of the task.

That's exactly it. Scoping out the solution is 90% as hard as just doing it. By the time you've figured out enough of the details that you're sure there are no more surprises waiting... you're just about done.

I suspect that a lot of counter-examples people have witnessed are examples of poorly factored codebases. I've seen teams/projects where they estimated projects accurately because the "new" thing being created was a slight variation on a kind of thing they'd created many times.

For example, I once worked for a company that generated e-commerce sites for clients. They'd make a new one basically by copy/pasting an existing site from another client and changing some text/images here and there. Super predictable after doing this a bunch of times... but also a maintenance nightmare having dozens of slightly different codebases. And how do you iterate? They couldn't evolve.

Shopify ate their lunch. I bet Shopfiy has a really hard time estimating the time to ship a new feature. But they add it once for everybody.

When the discovery process is iterative, there is never a point at which you have an estimate for the whole task. Each new discovery step may be the last one or it might lead to more steps, and you won't know until you complete some step, look for the next one, and realize the project is done.

A knowledgeable developer can predict how this cycle goes and explain that an estimate of the whole task is impossible, and they will be quite literally and completely correct in that case.

> I had a boss loved the phrase, "a date for a date."

I love deadlines. I love the whooshing noise they make as they go by.

- Douglas Adams

> The correct thing is for either of those parties to say, "Okay, I understand it's not estimable at the moment. What are the steps that we need to take in order to get a reasonable estimate?"

Ah, that would be nice. In my experience it's more like, "Okay, I understand it's not estimable at the moment. Just make your best guess and we can revise it later. (Psych! I will hold you to this estimate like you were under oath in court!)"

This leads to things like the Scotty rule. Since management will always misuse, misunderstand, or misrepresent your estimate anyway, might as well look out for number one and inflate the estimate, just in case.

Let me genuinely encourage you to try to find a job at a very early stage startup. I hear these horror stories about PMs/managers/etc., but I've yet to really experience them. I suspect that's because I usually join companies of <50 people. The engineers care about getting things right, the PMs care about getting things right, and management is well aligned with everybody, because there's little to no hierarchy.

The only time I've really had management "misuse, misunderstand, or misrepresent your estimate" is once they got past the point of being several hundred people with the bureaucracy that entails. Then I leave and go to a startup that's <50 people.

I don’t see how it’s unreasonable that someone would want to make plans based on the development velocity of software. Just because it’s actually hard to estimate some things accurately doesn’t mean it’s not valuable from a business perspective. The original post was someone saying that they should never be asked for estimates because every single thing they make is a unique snowflake of unknowable complexity, which is absurd. There are definitely features which an engineer can easily estimate delivery timelines on, and even knowing ballparks like “this is hard (as in weeks)” or “this is easy (as in hours/days)” is super valuable to communicate to actual business stakeholders.

In a healthy culture it should be possible to have a back and forth about how accurate an estimate is likely to be, but when engineers insist that all estimation is useless you can imagine why other stakeholders may lost trust.

> Unfortunately this bit of illogic is so common that I know it’s a waste of time to argue, so I’ve learned how to figure out what they want the estimates to be, give them that, miss them (just like everybody else does), and then get to work on actually providing the feature.

I do that and I've become exceptionally good at blaming the business persons for the outcome, making them admit their guilt in public and all that in what seems a friendly, positive and constructive manner and using business motivation and explanation. They never understood what hit them. Every programmer can do that if he or she learns a bit of psychology.

Can you elaborate?

I just confront business people with the results of their choices in a civil and polite manner.

If some manager made a promise to a client out of his/her ass without asking us, and if the PO pressures us towards meeting an unrealistic time-frame, when that explodes I just explain in a public meeting where upper management and all stake holders take part why it was wrong to promise that, using arguments and business considerents because bussineses people don't care about tech considerents. It usually results in that person admitting in public their mistake and even saying they are sorry.

I don't do that to humiliate people but to teach them to ask the team implementing the feature to estimate or to say if that feature is even reasonable or possible given current state and resources.

The estimates are useful even if they're wrong. Not all managers realize that, so they ask the question badly. And they never teach it to programmers.

The estimates aggregate to a number more accurate than each individual. The manager cares more about the sum total than each individual one. Many can even slip without affecting the end date.

This isn't so hard, if management and developers didn't treat each other as enemies. Each can get wrapped around the axle concerning each deadline, rather than the real goal of allocating resources across the entire scope.

For these I like to say "by the time I can give you an accurate estimate, I will be done with the work."

Seriously though. It's always a time tradeoff. How much time do you want an expensive developer investigating how long this problem will take. Will they learn something they can use during development.

I always follow up my above quote with the unknowns and how hard I think they really are; what are the possible risks; _and_ I include the experience of the developers working on the problem; our ability to bring in other devs who understand this better etc etc. I like to make sure whoever is quoting this problem out understand everything as well as we can in the given amount of time.

Is this not what estimating and refinement means to developers? I’m always pulling a number out of my ass, but that number is generally informed by current and past work, unknowns, and a “gut feel”.

If that number is too high, then that’s a sign to management that the work needs to be changed. The only way to determine that is to estimate it then make that call.

That's relevant for order of magnitude only, and only for "body sbop" work that is the same as previous work.

And only works if management wants an estimate. More commonly, management wants a lie they can tell the client.

> and only for "body sbop" work that is the same as previous work.

That’s absolutely preposterous. As a senior engineer, I have seen enough that I have at least a high-level idea of what most of the dev tasks and stories I’m estimating are. I don’t always have the specific details that may change the estimate, but I also have a whole team of other people who have seen even more things and can inform that opinion.

The point of refinement sessions is to share this information with everyone else so an estimate can be created and if needed, tasks can be better defined, split up or merged.

> And only works if management wants an estimate. More commonly, management wants a lie they can tell the client.

Wat? Estimates is used for plenty of reasons that have nothing to do with external clients.

How do you think your manager, their manager, etc knows what is going on and approximately how long it’s going to take? They’re going to be far more disconnected from the work than the people doing it day to day, so they would generally have no clue how long something is estimated to take without someone telling them.

App idea for tracking projects: Each individual developer can anonymously indicate how far along they believe a project is. For example, Developer A thinks the project is 60% complete, but Developer B thinks the project is 30% complete. Use algorithms to predict the completion date.

However, I don't think this idea would be very popular because it removes the manager's ability to control the presentation and play politics.

The original question was "Why are developers expected to estimate tasks at all?" @benjaminwootton gave perfectly sane answer to that question.

HOW developers should make their estimation is a totally different question, but it is not some arcane knowledge, and it should be an everyday craft for every mid-level engineer. First of all, we are doing those tasks by hundreds through our careers, so an easy way is to remember how long similar tasks took us last 10 times an produce a bracket, like in "from 5 to 8 working days, let's say 10 to be sure". Every business person will be happy with that answer.

OP says that every engineering task is unique as a piece of art. This is BS, IMO, except for the very junior engineer. Most of the time business wants some well-known and hundred times done things they see their competitors do. All the uniqness stems from particular codebase. If your codebase is a mess - you are in trouble, but it is you problem, not the general problem of software engineering.

In case I have some really unique task, what I should do is to say: "I cannot estimate this right now. Give me two days and I will get you an estimation". In those days I decompose the task into smaller, and divide those smaller tasks into "well known" and "unknown". Then I imagine (or even draw) a trivial Gantt chart and will have my estimation.

It is also totally normal to change your estimation during the work. You just should do that the moment you see a problem, not at the day of the deadline.

The response to "there are too many unknowns to provide any sort of estimate" shouldn't be exactly "well, we need estimates." But "okay, just do whatever" is also not a working outcome. Software is weird, yes. But other disciplines are able to provide useful planning information. Why are we unable to do that?

Most other disciplines have this exact same problem. Construction contractors for instance, are notorious for it.

The difference is, the law says if a contractor signs a contract to do something at x price, they HAVE to do so or suffer consequences. So contractors go broke/bankrupt, lose their licenses, or figure out how to bid correctly enough they can either weasel out of it if it goes sideways or they have enough wiggle room to deal with the inevitable problems.

Software Engineering Consultants have the same situation. But directly employed software engineers are in a different position of course.

One thing that typically makes this worse, is the implicit assumption that planning/quoting/estimating is ‘free’ work, and since it CAN make things easier to deliver, therefore we should do more and more of it as, since it’s free, it will solve the problem no?

Other disciplines have problems with slippage, but none of my friends who are in physical engineering fields insist that they shouldn't even have to provide estimates in the first place.

Of course not, their protected place comes with conditions like giving estimates (and being held to them).

And their disciplines involve massive capital outlays which can’t be refactored in a cost effective way, or at all really.

Not defending the ‘no estimate ever’ approach, it doesn’t work. They are different though.

Really, they can, can they? Even the bespoke, one-off projects with unknown unknowns?

Tell me, how accurate were the estimates for the launch date of JWST?

I’ll wait here while you look up how many times and how many years that “slipped”.

I bet some dumbass NASA manager asked the team to schedule the launch date back in the 1990s.

It’s monumentally frustrating to be forced to give estimates too soon in the planning process. The compromise, I hope, is to end up in a place where most of the problem is known and some is unknown, and you get there by breaking down the original problem. Leave it up to the “buyer” to digest your best estimate into their planning process.

Estimates are theatre. It’s not even theoretically possible to provide accurate estimates for anything but the most trivial project, yet here we are.

As an industry we just seem incapable of coming up with any other way of negotiating and running projects.

It gets even worse when you are in a line of work where the estimates are part of some kind of sales pipeline. Then you aren't just estimating based on effort required, but also against a likely number that the customer will ever agree to.

A general framework is that on each decision, including architecture, cost, time one at least adds the following:

1. Assumptions (including soft such as who will actually do it, resources are not equally relevant or productive) 2. Constraints (as agreed by stakeholders) 3. Risks 4. Clear scope 3. Alternatives and why they were not preferred

Business should be aware and own the uncertainties, or else the system gets biased towards sales people offering vaporware.

You can say this, but when a plumber or an electrician comes home, they are always expected to provide quotes and duration of work

The fair thing to say is that there's something like 80-90% chance that this will take X time, 99% chance that it will take less than Y time.

This would allow for planning, prioritising and is the most accurate communication we can achieve.

The best thing to do is to define tradeoff ranges and set the deadline, so that developers could try an optimistic path, learn what’s wrong with it halfway, and then choose a tradeoff that both seems not that bad enough and, when it fails, leaves time to rethink and use another tradeoff which is even good now, in a sense.

Because that’s what we do either way. Round numbers from ass make no science to plan and prioritize upon.

No, hard no.

In my experience good communication solves this every time.

MGMT: We need an estimate ENG: I have several unknowns MGMT: We really need an estimate ENG: This is my rough estimate, but be aware that the unknowns could change this drastically MGMT: Ok, what do you need to get a firmer estimate.

And so on.

As a project progresses, the communication between ENG and MGMT is crucial to ensure that MGMT are aware how those unknowns are affecting the project, MGMT will decide at every juncture if the project is worthwhile to continue to invest resources into it (assuming they avoid the sunk cost fallacy)

Reply on news.ycombinator.com