Hacker News Reader

Comment by Aaargh20318 | Hacker News Reader

Aaargh20318

The reason that planning software development is bullshit is that you simply cannot know all the little details because in software development you're always doing something you've never done before (because if you did you could just copy-paste your previous work).

To use the 'going to the supermarket for milk' example. I could make a fairly accurate estimate for that because I've gone to the supermarket hundreds of times. I know all the different things that can go wrong and account for them, because I've encountered them before. The elderly person who wants to pay cash and has a bag full of coins. The guy who finds a 10 cent discrepancy on his €92,30 receipt and has to argue for 5 minutes with the cashier (while the row behind him keeps growing). etc. etc.

Now imagine that today is your first time going to a supermarket. In fact, before today you had never heard of the concept of supermarkets, or milk for that matter. How good will your estimate be ?

The only time you can make a decent estimate is after you've finished. Or to put it differently: making an accurate estimate is possible, if you're going to accept that making the estimate is going to take a long time, but I can't tell you how long.

Development time estimation, and every methodology that attempts it (I'm looking at you, Scrum), are little more than desperate attempts by managers to feel in control and relevant.

View on HN

This claim of inescapable novelty has a kernel of truth, but it is being exaggerated beyond any sense of proportion. If everything you have done in your day-to-day work has been beyond-estimating original, then either you could out-Knuth Knuth (in which case, kudos to you, and I hope you will have time to write the books someday, but your experience is not generally applicable), or you have been goofing off some of the time.

That which is passed off as novelty is often ignorance (not necessarily on the part of the developer.) To continue the retail analogy, software projects are often like the guy who is building a deck on his house, but keeps going back to the hardware store because he made no attempt to think things through, or measure before cutting (note that, in this analogy, the would-be builder is not necessarily playing the role of the developer. In some cases, that might be the hardware store owner's role.)

But it's exactly this 'ignorance' that is the major part of every software project. There are unknown unknowns, and there is no way from a standing start to grasp either their number or their nature. I'm happy if people are saying that this exploratory work should be done (in part or in total) before estimating, but in my experience that's just not palatable to most organisations.

I envy _hugely_ anyone who builds software that is even 50% the same as what they were working on six months ago.

Well, then let's call it what it is, because there are things that can be done to mitigate ignorance to some extent, if there is the will to do so from all parties.

> there are things that can be done to mitigate ignorance to some extent

Sure, but all you're doing is moving the uncertainty from the development to the planning phase, so instead of a development phase of unknown length and cost, you end up with a planning phase of unknown length and cost.

Firstly you you are still assuming the same gross exaggeration of novelty, and secondly, you can do analysis and planning at a much higher level of abstraction than development.

Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction, before doing anything else.

> Firstly you you are still assuming the same gross exaggeration of novelty

I disagree. Even when you're doing mundane everyday stuff you have done a thousand times before, one trivial variation may throw of your estimate completely. Maybe an edge-case in this specific project triggers a bug in a 3rd party's library, and you're losing weeks waiting for the 3rd party to roll out the fix (been there, done that). You just don't know in advance

> you can do analysis and planning at a much higher level of abstraction than development

Of course, but then you'll get an inaccurate estimate.

> Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction

That's only necessary if you want an accurate estimate. I think accurate estimates are pipe dreams.

There is a false dichotomy here between knowing everything and knowing nothing, between perfection and utter chaos. You may only have experienced the latter, but there are places where people know pretty much what they are doing, most of the time.

> ...there are places where people know pretty much what they are doing, most of the time.

It's called spec programming, where you have programmers slinging code against precise specifications. Common in ERP and regulatory or business compliance-driven code.

On the other hand, you have development. That is where you're attacking the less defined problems. Sure, we know the tools in our toolset, how to apply them, and time it takes to apply each tool. Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time, time that could have been spent towards a MVP.

> Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time.

In the use of the adjectives 'precise' and 'enormous', you are just repeating the same false dichotomy in a slightly different form, by ignoring (or not understanding) my point about the use of abstraction in analytical thinking.

The people who are most successful in what you distinguish as development understand a good deal more about what they are working on than spec programmers need to, and they are not, as you imply, groping around in the dark for a solution - that's what spec programmers do when faced with a development problem.

Just to give an example of two projects I worked on at the same company within a few months of each other:

The first involved taking all sorts of situational data from various sources in an airport, e.g. radar systems with their own binary formats, and combining them with the work of a postgrad researcher who had come up with an algorithm to optimise when pilots should turn on an aircraft's engine in order to save fuel and cut emissions.

The second involved managing 3G dongles on fleets of laptops, tracking and sending alerts on their usage and billing info in realtime, back in the days when business users would accidentally run up £10,000 bills abroad.

If at any point in the first half of either of these projects I'd been asked to come up with an estimate for the total amount of work, it would have been wildly inaccurate. Each project had both low and high level surprises at every turn that would have just compounded to blow those estimates up. There is basically no significant way in which one informed the other, even though parts of the systems used similar event sourcing architectures on the exact same server-side stack, and were done for the exact same startup which broadly offered real-time systems to utility companies of various sorts.

It's entirely possible that there exists a breed of programmer who could have estimated these projects in their entirety, much earlier and more accurately than we did, but I've never met them, and I'd rather work in places that didn't compel people to.

I've known and worked at other places where basically any software methodology would (and did) work, but I certainly don't think it's rare (as you seem to suggest) to find yourself at the other end of the spectrum.

Aaargh20318 et al are making the claim that it is always inescapably like this and there's nothing you can do about it, because every day of programming is always fundamentally unlike anything you have ever done before. All I am suggesting is that this is an exaggeration.

Doing something about it takes some commitment, however. Nobody estimates well without having practiced it, and been honest with themselves about where it went wrong.

Understood, nothing to disagree with here.

It's like building a deck except even the screw hasn't been invented yet. And you think: I'll just create a device that attaches the wood like a big paperclip but then realize after you make it that it's not strong enough and won't scale so you create a nail. But then you realize you may need to reuse the wood so you finally create a screw.that can be removed.

That analogy might have worked in the 1950s.

You aren’t wrong - for some things. I think it’s a spectrum.

For the things I work on, and frankly always have, I spend most of my time in the “R” part of R&D. Now granted, I’ve never worked on classical “business” or CRUD type problems.

It’s natural to think that once you have lots of experience, that estimating will be easier. In my experience, I found myself simply tackling harder or unique problems. Instead of getting comfortable, you find yourself mostly operating at the edge of your competence. This is great fun [1] but makes estimating pretty hard.

[1] I personally enjoy being on the “black diamond” part of the learning curve. (“Black diamond” = most difficult ski trail)

Agree 100%. IMHO, a lot of it has to do with the programmer mentality of wanting to be "challenged" to solve problems one step away from their competency. If you're fresh out of school, implementing your own allocator might be a challenge you take upon to solve a memory constraint, when you can just take what someone else (with more experience) has built. And then senior people insist on designing yet another over engineered platform technology because everything has to be 'future prof' (when in reality it never is and only ends up being performance-proof). How many JS frameworks have we had so far that do the same thing and just repeat mistakes of previous ones? Its like one every year.

I agree.

There is also a lot you can do to decrease impact of novelty on the overall schedule and most people I work with do exactly nothing to reduce variability in the process and improve their estimation ability.

It is easy to claim every time you write code you are doing something new but, hey, every time they build a building they do something new. Yet buildings (except for extremely ambitious projects) typically can be delivered reliably by reputable developers.

A lot of people have been praising this "estimates based on data and rigorous planning" for decades. Steve McConnell wrote a whole book about it (Software Estimation: Demystifying the Black Art).

It works only very special circumstances: when the task can be very well defined because it has been done a lot of times before and the people doing the task are very experienced on it.

But the caveat is that if you are doing such a task you're not really doing any innovation at all. Actually, if it is so well defined and trivial it should probably already have been automated.

True innovation must have a lot of unknowns, it isn't innovation without it.

For what it's worth, the examples in that book are usually contract development shop where you're doing the same thing for multiple clients over and over.

If you actually read Steve McConnell's book the focus is really on accurate estimates more than data and rigorous planning. He makes it clear that a high level of precision is impossible in some cases. (Many people fail to appreciate the difference between accuracy and precision.)

I haven't read the book and was going to ask about accurate vs precise but found this helpful article discussing it as it relates to software here if anyone else is curious about that distinction: http://itsadeliverything.com/accuracy-vs-precision-in-estima...

Yet if I ask you how long it takes to buy icecream in Japan, based on your experience grabbing milk in the US, your guess will be "less than half a day". Which is good enough for me.

And if I ask you how long it takes to build a Wordpress blog for a music band, based on your experience building blogs for realtors. You guess will likely be good enough for me. :-)

But then you say go for it and get upset about all the time I spend getting a passport, booking flights, and figuring out how to transport the ice cream back.

I’m kidding, of course. It really depends on how well business and development understand each other.

> I’m kidding, of course. It really depends on how well business and development understand each other.

Yup. This is the crux of the whole issue, and it's being completely blown out of proportion. There's always going to be details the estimator (dev) won't know, but the more clear a picture you have in your head, the more confidence you can have in an estimation.

This is directly related to the size of the project too, of course. If it's a 12 month project, a single developer is massively unlikely to have a reasonable understanding of all the potential roadblocks that the project will take. However a 1 day task, with tools you know/etc, you can have a pretty good idea how long it'll take.

There's always some unknowns, but that's literally what an estimate is for - attempting to plan for the unknowns. I can write an add function and I'd bet a lot of money that it won't take more than a day.

Unreasonable management seems to be giving developers PTSD here. As I'm having trouble understanding why the idea of an estimate, by definition exactly what we're doing, somehow doesn't apply to software development.

If you're being pedantic you could say the same about a lot of things. How long to dig a 1foot hole in open ground? Well who knows, there might be a massive rock - but that doesn't mean you can't give an estimate. It's literally what an estimate is for, an approximation based on assumptions and expectations.

> Unreasonable management seems to be giving developers PTSD here.

It's inexperience all around. Also, most places have no clear long term track, so people bounce after 2 years rather than growing to understand each other.

Managers are generally free to cut estimates in half, and move to a new role or move to a new organization when things fall apart. They're not trying to be villains, they have a lot of pressure coming from a lot of directions and think that estimates are bogus anyway, thanks to star trek and the occasional miracle that happens. (sometimes things really are much easier than they seem)

Developers are optimistic, wildly underestimate, and feel bad when they miss them. The overreact and start massively overestimating. When they progress to the next stage they take the time to give a good thorough estimate to have it slashed by a manager, who knows they overestimate.

Nobody sticks around long enough to really understand the state the other is in, so you get this vaguely Kafkaesque world that makes everyone a little crazy.

Estimates in software are effectively experienced guesses. This is in contrast to other industries, such as construction, where planning is done for months to ensure the estimate of time and materials is accurate. If you attempt to treat one as the other, you run into huge problems.

Software isn't like building a bridge; it's like planning to build the bridge.

I'm pretty good at software estimating; I can roughly tell when something will take an hour, 2 hours, 4 hours, a day, a week, or month, 3 months, or 6 months -- in those intervals. It's more difficult and time consuming to get more specific than that.

I've also had a 1 day task blow out to entire week more than a few times. At that point, you have to start questioning the predictive value of estimates.

Heh. But even in construction projects, serious cost/time overruns are very common. I can't find a good source through quick googling, but what I do find is horrifying.

Industrial Megaprojects by Edward W. Merrow essentially shows that we're comparing ourselves to a myth of our own invention.

Thanks, wow, this looks like a really interesting read!

It is actually quite fascinating to see how our cousins get on. He's quite blunt about the dynamics that cause failure and that relatively few megaprojects ever live up to their business case.

And when he returns with the ice cream I’ll tell you I wanted Hokkaido ice cream.

And "quick question: what's the level of effort for making it tempura ice cream?"

I estimate it takes 20 mins to buy an ice cream in Japan. If it takes "half a day" that's (charitably) 12x (8-hour days), or (uncharitably) 24x (16-hour days).

If I estimate a month to build your feature and it takes me a year or two, you're not going to be very happy...

> you simply cannot know all the little details

This is the reason big design up front hasn't been the gold standard in planning. Most modern methodologies acknowledge that there'll be a lot of changing requirements and learning-while-building.

The point is, given this information, you could a) run around like crazy doing all manner of reckless and crazy cowboy programming and putting out fires b) carefully plan a relatively short period of time on a small, achievable, well defined goal (not "build the project"), then making a note of what worked and what didn't, and readjust before you plan the next short period of time

This is often taken to an unhealthy extreme.

You need to have some fairly specific goal in mind to start doing anything. AKA are we building a PS4 game or a self driving car?

Methodologies like Scrum are all about transitioning a long term objective into short term progress. But, you really do need some fairly well thought out goal.

Further course corrections quickly get very expensive. Sort of like sailing from Virginia to Europe then deciding 1/4 of the way in on France then changing your mind to Spain at the midway point is probably fine. However, changing your mind to Australia is a much bigger deal.

Agreed! Nothing I disagree with here. But last minute itinerary changes from France to Australia back to France again point to a larger, more systematic problems IMHO. Perhaps it's a lack of understanding of the market and customers, or just bad management and lack of company vision - but no software process is going to fix that.

And then a poor maintenance programmer is faced will lack of architecture because the thing was built without any in mind... Just as a perpetual prototype.

I completely agree. I'm lucky to work in an environment where management aren't too fussy about sticking to deadlines, as long as progress is pretty decent. It's a nice way of working, I use system 1 for estimates which generally works pretty well and saves time needlessly trying to pin down an exact eta. On the whole our team delivers a lot of features on time and generally doesn't go over 2 weeks late for anything, even though it often wouldn't matter if we did go over.

I appreciate that sometimes hard deadlines are necessary but if I can I'll always avoid them.

> Development time estimation, and every methodology that attempts it (I'm looking at you, Scrum), are little more than desperate attempts by managers to feel in control and relevant.

More charitably, they're an opportunity to introduce system 2 slow thinking on the team.

There are certainly novel engineering problems that simply won't yield to estimation, but there are also lots of well-solved problems that will. Managing the latter isn't desperate, its just sound practice toward reproducible results (ideally deployed only where reproducible results create value).

> but there are also lots of well-solved problems that will.

Certainly when you're writing a double-linked list a third time you can estimate how long will it take you. The problem is that in real life you never solve sufficiently the same well-solved problem. Feature details, matters of integration with other pieces of the system intervene - and in practice those errors introduced by new circumstances are significant comparing to predicted time. So the practice may be sound, but it doesn't give the desired results, like predictability of required time.

So you integrate with a few systems, you learn where the gotchas tend to be, you call those out as things you'd like to know more about when making your initial estimates, etc. Today's software dev has probably integrated with more systems than they have written linked lists. We just have a habit of forgetting all the little details.

If you're looking for a formula that lets you estimate once and then move on to the "real work" now that that tedious BS is out of the way, there isn't one. But you can still do something. There's ground between "I can't estimate that in 30 minutes" and "I can't estimate it at all" and it's up to you and your stakeholders to come to terms on how to balance the risks in your estimates (in broad terms: too inaccurate estimates vs too much time wasted estimating requirements that have since changed).

(There's a parallel theme in some other comments: "if it's a well-understood problem why am I not using the existing solution?" Similarly, there's a difference between "well-understood" and "perfectly abstractable and automatable", otherwise almost none of us would be employed these days. :) )

> there are also lots of well-solved problems that will

But if it's a solved problem, why not use the existing solution ?

Even if the solution exists you still have to integrate and test it. Those tasks can require a lot of engineering labor, but fortunately they're fairly amenable to estimation.

This is being a little extremist, no? I'm consistently asked to give estimates, and I mostly give estimates that are accurate within sane expectations.

I think what's missing here is that I can predict how long something is going to take when I talk with the managers/feature-planners about what the feature is. If I feel I have a solid understanding of the requirements, which pieces of code it'll touch, etc - then I can give a reasonable estimate.

Now, the bigger the project the more difficult this becomes. Details I've missed is always a possibility, and that just stacks up the bigger the project is.

So I think being specific in this type of discussion is really important. A feature that's going to take 6 months to implement is insanely tough to estimate, and very likely to have a wrong estimate. A feature that takes a week can pretty reliably be accurate within a day or two, in my experience.

I'm not defending management in these sorts of things.. I think it's our job to not only give estimates, but include understanding about the expected validity of that estimate.

All estimates are worth while though, your post makes it sound like they're not. You tell me you need a feature that's going to take 12+ months? It's so hard to predict that it's laughably wrong.. but that's still insanely useful. Why? Because now the management knows it's highly unlikely to be a 1 week, 1 month, w/e feature. It's big.

> I think what's missing here is that I can predict how long something is going to take when I talk with the managers/feature-planners about what the feature is. If I feel I have a solid understanding of the requirements, which pieces of code it'll touch, etc - then I can give a reasonable estimate.

I'd argue you can't.

Right now I'm working on a feature that I estimated as 'a couple of hours'. I've been working on it all day by now. Not because the feature is that difficult, or because I didn't get a set of good requirements. The requirements are crystal clear and should be easy to implement.

The reason it's taking so long is that I'm fighting with a UI component that doesn't behave as expected (or documented). What should have been a trivial part of this task, a matter of seconds and not worth estimating at all, turns out to take all day because I didn't know this specific component is misbehaving and I couldn't have known until I started implementing it.

You cannot predict things like that without doing it.

Oh. I feel your pain.

But to clarify: You can't predict things well that you haven't done before.

The point being, when trying to predict an unknown it's best to give a range and not a fixed estimate. If possible.

And then management only hears the low number of the range.

"Will management abuse the estimation process?" is a different question from "is estimation possible and improvable?"

I like to use the range: “2 weeks to 2 years - roughly.”

> Right now I'm working on a feature that I estimated as 'a couple of hours'. I've been working on it all day by now. Not because the feature is that difficult, or because I didn't get a set of good requirements. The requirements are crystal clear and should be easy to implement.

Just because you're wrong sometimes doesn't mean you can't be right more often than not.

Another way to look at it is, as a company how can you budget anything if you can't make even the slightest of estimations?

What you're suggesting results in throwing away any ability to budget, roadmap, coordinate, etcetc. Absolute prediction is of course impossible, but if what you're saying is true than I should buy a lottery ticket because my estimations are insanely lucky.

And sure, I get estimations that run long, we all do - I'm not saying otherwise.

This is for me the dissonance between agile methodologies and reality. Agile methodologies assume having a flexible scope and time due to the project unknowns (technical, business, social, etc.). But usually budgeting in the real world is required to follow: I promise to deliver X scope that’s going to cost Y dollars in Z time.

I'm trying to get an app to the app store. Mysteriously, the archive button on XCode isn't working. It says it compiles successfully and yet there is no archive to submit when I'm done. I've googled all over and everyone else has a different solution to this problem. This little issue has delayed release a day so far.

I've done this 50 times before (10 times in this project!) and these kinds of issues still occasionally occur. How does anyone predict for this kind of thing?

To be fair, Scrum doesn't try to capture development time estimates, only the effort needed to provide value. Work items inherently vary in size, or estimated effort. However, enough work should be planned during planning for the dev team to forecast what it believes it can do in the upcoming Sprint. It's all about transparency, not adherence.

That said, requirements emerge daily. Transparency is essential for honest dialogue.

I agree and disagree with the notion that an estimate can only be provided after the work is done. If the work is broken down properly, you can use empiricism to get a good understanding of what is needed based on previous, similar work completed. But to think an estimate can only be provided after the work itself is done is kind of ridiculous. If you section off variables with manageably-sized work you should have a good understanding of what's needed to pull it off.

And TFA's suggestion (use our magical time tracking software) isn't going to solve that problem, either.

Massive software companies deliver high-quality software, on-time all the time for one reason: resources. They got the money and the time and the developers and all the enterprise accounts up the wazoo.

Typically no one wants to pay more for less, but that's the best strategy to deliver high-quality software on time.

You reduce the scope drastically but do not adjust the timelines. You pay for more development resources(tooling, developers, servers, time, etc.).

If you can't deliver at that point, then there's a serious fundamental issue with either the leadership or the team or both.

Startups don't have this luxury. So they must execute on the least complex scope on a very tight timeline. High-complexity scope increases the fatality risk to the company.

So as the scope increases in complexity the risk increases and the likely the company itself will fail increases. Again, because startups do not have the cash to pay for the proper amount of resources.

”Massive software companies deliver high-quality software, on-time all the time for one reason: resources. They got the money and the time and the developers and all the enterprise accounts up the wazoo.”

I think that is a myth right up there with “If I was rich, I’d be happy.” Massively funded software projects fail all the time.

https://news.ycombinator.com/item?id=17235832

If you're really working on new problems all the time, consider yourself extremely lucky.

To keep with the 'going to the shops to buy milk' theme you can break it down into bits you can estimate. Assuming you've never bought milk before:

- how long will research take? Where do you buy milk? Where is the nearest shop? 3 mins.

- how long to get to nearest shop and back (eg 10mins x 2) = 20 mins.

- find milk in shop (30 secs to 5 minutes), so say 4 mins.

- Buy milk. (1 to 5) say 3 mins.

Don't forget that these are estimates and always add a bit of headroom. The above comes to 31 mins, so say 35 and you know you're covered. If the manager isn't happy with it, give him the low down figures.

Recent discussion:

https://news.ycombinator.com/item?id=17154355

Very well put. It reminds me of the “Halting Problem”. Although for a subset of programs, it can be proven that they will finish in a predictable amount of time. Maybe this also holds for projects.

Programmers aren't special snowflakes and there's nothing special about programming. The exact same can be said about anything be it marketing, sales or design.

Also consider that the management or project is looking for just some vague white liquid and at the store you only see cans (powdered milk? Is that white?)

If you can plan and execute putting a person on the moon, you can plan and execute software development.

I think the problem is the overwhelming majority of software development is so low stakes, easy to replace, assumed that bugs are par for the course, etc... that most orgs have stopped putting real effort into planning AND executing on that planning, instead just cargo culting the planning phase.

> If you can plan and execute putting a person on the moon, you can plan and execute software development.

But the budget for planning is limited. To realistically plan out most software projects would entail almost as much work as simply executing on them. That level of resources for planning vs. execution is acceptable for putting people on the moon. Probably not for building a line of business app.

You've missed the point. Nobody can (or did) accurately estimate how long the planning and development would take to get someone on the moon. Half the materials weren't even invented yet in 1962. Nobody knew that Velcro was flammable in pure oxygen.

Sure once all that expensive and unknowable planning and development is done, you can double-click that executable and put a man on the moon.

> If you can plan and execute putting a person on the moon, you can plan and execute software development

Okay, but I think you've missed the point. Planning is hard, and you can't possibly account for all the variables.

Everything after that statement I can agree with

> If you can plan and execute putting a person on the moon

I think you vastly overestimate the quality of the software and systems that took men to the moon. Try reading https://www.google.com/amp/s/amp.space.com/26593-apollo-11-m... for example - you might conclude "they got lucky"

> If you can plan and execute putting a person on the moon, you can plan and execute software development.

Do we have the planning and estimates of projects that are similar to "put people on the moon"? I'd like to see how the plan/estimates of these projects look like and how they change.

Getting to the moon took many many man years of planning, experiments, material development and cost a few lives. It also involved several of the worlds leading scientists, and probably the best computers money could get at the time.

If you can accept those kinds of preparations and also agree to not change the specs much (i.e. no, we aren't going to Mars instead) after we've started developing it I'll say we can get you a pretty nice estimate.

Reply on news.ycombinator.com