Software houses want to project themselves as intelligent collection of beings, but why can't they see the truth that there?
Said differently: good programmers are not afraid of making estimates. This classic article says it all better than I could: https://www.kitchensoap.com/2012/10/25/on-being-a-senior-eng...
From The Unwritten Laws of Engineering, quoted in the article:
Promises, schedules, and estimates are necessary and important instruments in a well-ordered business. Many engineers fail to realize this, or habitually try to dodge the irksome responsibility for making commitments. You must make promises based upon your own estimates for the part of the job for which you are responsible, together with estimates obtained from contributing departments for their parts. No one should be allowed to avoid the issue by the old formula, “I can’t give a promise because it depends upon so many uncertain factors."
Also from the article:
Avoiding responsibility for estimates is another way of saying, “I’m not ready to be relied upon for building critical pieces of infrastructure.”
The reverse is also just as true, of course.
It doesn't mean you know what feature is going to be delivered in 32 weeks in the future though.
Software is different from most other branches of engineering in that it is far less limited by physics and that the requirements are far harder to define.
For example the requirements for a bridge or a building are relatively straight forward (size, load capacity, climatic resistance etc etc). That's not to say that designing the bridge or the building is necessarilly simple (though it probably is for small bridges and houses because we have centuries of experience with them) but even for complex construction projects the requirements are relatively simple even if the implementation may be very complex (think channel tunnel or mileau viaduc).
In software there are much flexibility that most customers don't know what they want until they see it. They generally aren't capable of saying up front what they really want (and when they try to they are often wrong and end up saying "yes, that's what I said I wanted but now I can see it it's not good".
It is about mindset. Of course I do not mean that a programmer should suddenly be responsible for "doing everything."
What I mean is, the programmer should have as their personal mission the writing of the highest possible quality code that is ready to go to production as quickly as possible with a minimum of rework. And crucially, you have to be able to provide a realistic and reliable estimate of when your work can be expected to be completed. This ability is gained through experience.
Working well with project/product/design/etc. is part of being a good programmer. It is not enough to simply be a "good dev" - you must also be able to explain what your code does, why it does it that way, and any potential tradeoffs or flaws in design/implementation. If you cannot critique your own code in this way you are not a senior engineer. When you work with other people part of being senior is being willing to be held accountable.
Hopefully this makes what I mean more clear. It is about mindset.
When you order food online, do you expect to know when it will arrive?
When you go to your mechanic for a repair, do you want to know when you may get your car back?
When you go to a lawyer to verify a contract, do you need to know when they will be done?
Why should building a software solution be any different?
I want! However, for my old 2000 car, repairs exceeded the original estimate 80% of the time, forcing me to change my day-to-day plan. This could be a hint: good/new things breaks less, giving you false confidence when estimating for old/bad stuff.
They also all suffer from estimation issues in practice. Luckily these constraints set a limit on how much a competent professional can underestimate the project.
If you're asking someone to install or update some software they've installed before on a similar system, they should be able to give you a solid estimate. This has parallels with many of the services you mentioned.
If you're asking them to research, develop and test entirely new components, which is the basis of software engineering (an engineer does more than install and update packages), you should be able to understand why this is different than delivering food.
> They also all suffer from estimation issues in practice. Luckily these constraints set a limit on how much a competent professional can underestimate the project.
Where exactly did I claim these weren't complex services? All I said was that there are more constraints on these services than pure software dev. I went further and broke software dev down into different buckets of complexity to further pare down the argument and concede that some software dev should be constrained enough for accurate estimates.
Dispensing with the straw man, what exactly is incorrect about my comment?
Keep comparing apples to oranges and we will turn Software Engineer like any other menial job. We all know it there are a lot of unknowns, like health issues, that need to be evaluated before formulating any opinion about estimation.
Stop with non sense about estimation.
While it may be true, saying "it'll be done when it's done" isn't useful. All tasks involve some aspect of uncertainty. Coming up with an at least half-way decent estimate requires a team to break down the problems they're facing into smaller chunks that can be estimated. This is a useful exercise when it comes to prioritizing projects, budgeting, and go/no-go decisions.
There are many ways in which estimation as an exercise can be corrupted. Some people, particularly those who haven't written software themselves, treat all estimates as accurate. In their mind, that "I guess 3 months" estimate a developer gave quickly turns into "I can 100% deliver this to you fully tested, with all of the additions you added along the way in 3 months". This over-indexing often leads to the situation where a developer is disciplined for not meeting expectations, even though they tried their hardest to do so. Some people don't like the estimates they are presented with, so they seek out a lower estimate. They are then shocked when the project takes longer.
This is my biggest trigger, when a vague, hand-wavey estimate is turned into a signed guarantee, then the developer gets disciplined or treated as a problem creator when more detailed information comes in.
I found this was the problem with working at agencies - No matter how warm touchy feely they are, the fact is the agency is economically incentivised to burn out their developers (in the short term, which sadly, is the most frequently perceived time horizon).
This is why I only work at venture funded startups now with a preference for default-alive ones, yes it's a bit chaotic because resources are not infinite everywhere, but it's much better than working at an agency who are just middle-man reselling your time and trying to keep their margins up
I don't think starting from I think X feature will take Y days ever works. By contrast, we need to launch by Z date is technically a valid statement, it doesn't need to be estimated.
I think for this to work the business actually needs to be willing to admit that either more people need to be hired, features need to get cut or dates need to be pushed back.
If we're talking about effort, timelines, etc, then it does work. You are right that it generally doesn't work on the surface. However, even if the estimates are wrong, the benefit is that the artificial dates and such provide a mental push. In some cases dates are important, to meet a business deadline, beat a competitor, etc. Breaking that up and pushing it down to smaller stories means more feedback which provides more of that 11th hour push more often.
And then of course things like costs and staffing are crucial to any business.
As a rough ballpark figure (1 day, 1 week, 1 month, 3 months, 6 months, 1 year, 2 years, 3 years, 5 years) estimates are doable and reasonable.
It's also reasonbable and doable to let scope be the factor rather than time. "Tell us the most important thing and we'll do that and deliver the results every couple of weeks, tell us when what you have is good enough".
You can also do fairly accurate estimates when you're basically just doing the same thing as you've already done many times or when the customer doesn't have to give much input. Most of the construction industry is like this because it's older, goverend by physics and building codes and we have a few standard models for most things. Most of the software industry isn't like that.
However for something new, where the customer doesn't really know what he wants in much detail accurate estimates aren't really possible. In this case anyone who purports to give you an accurate estimate is either lying or using extensive padding.
Customers, and non techinical people in general, have a hard time understanding what things are difficult / important from a complexity perspecitve. Often seemingly "simple" (to them) changes are actually very complex, and sometimes the other way.
This turns out to be much easier to reason about, discuss with colleagues, calibrate one's intuition etc; If one person thinks it will take 100 lines at the end, and another thinks it will take 1200 lines at the end - it provides a more concrete discussion when trying to resolve the difference than "10 days vs. 3 months".
And the number of bug-free structured-programming-style lines a programmer produces per day is surprisingly low variance, when measured at the end of a project. (And ... the mean is often in the 1-10 range for good projects and good programmers for places that require high quality like medical devices or aviation). If that sounds too low to you - consider a project 6 years in the making (~2000 days) worked on by 50 programmers. That's 100,000 programmer days. If it has 1M non-comment lines, it's only 10 per days on average.
Somehow, I've never been able to apply this to Java code or classful Python code, and have never met anyone who was - I suspect it's because these projects are dominated by boilerplate and huge bureaucracies, which weren't prevalent in the old days when this trick was devised.
(Which is not to say that object oriented programming and classful Python and Java are bad; Just that they seem to be resistant to this form of estimation)
For peers, far apart estimates might indicate differences in understanding of the problem/solution.
For a team, if they are wildly diverging or low-certainty it can indicate the right next step is experimentation or exploration, rather than committing on delivering something to production.
For an organization, they can provide a barometer for investment decisions, such as what the appetite to solve a given problem space is, or how to evaluate sunken costs.
I particularly like a progressive framework on estimates, i.e. 10% certainty when a project is just taking shape, 20% when scope gets tightened up, 50%+ after spikes / rfc drafting / design etc.
Often they are used for rote accountability, which can be detrimental and/or naive - they should be a stake in the ground to have discussions around risks, assumptions, and trade-offs!
1 day
1 week
1 month
3 months
6 months
1 year
3 years
if you had a random project, do you think you wouldn't be able to pick the most likely estimate and be within one level above or below? so "i have no idea, it'll take as long as it takes" is not really true.
A situation like this can turn a "X days" task into an "X months" task. I know it is not super realistic but it represents a type of situation.
Also, the process of estimation very often reveals difficulties or unknowns that nobody was even thinking about previously. That's the whole purpose of estimation happening via unanimous planning poker -- the discussion as to whether a feature is a 3 or a 10 is incredibly valuable in getting on the same page as to what the feature even is.
If I ask an average web developer if they can create a basic landing page in 3 months, they will always say 100% yes. Now shrink it to 1 month: 100%. 3 weeks: 90%. 1 week: 50%.
Now as the person paying for the work, you can make a choice. Do you plan to start advertising on the odds of a coin flip or do you wait until one of the 100% points?
Two drastically different estimations can highlight that there may be a lack of understanding or a high difference in skill / experience on the team. Knowing that allows empowers the team to break a task down smaller, make the task more clear, or spend some time educating team members.
I don't think those estimates should be shared outside the team.
I once had a huge discussion with the Architect because I said something would take an year to finish and the Architect was claiming it would take 2 months. He would not understand that the remaining 10% of the task would take 90% of the time (yes, it was partially complete and I had been following its development from the start, although never involved in it directly). In the end they went with his estimation but it took 2 years to finish the thing.
Still, I think estimation is important, once you actually listen to the estimations and understand it really depends on who is doing it.
1. Business pressures to 'accelerate' timelines without understanding the tradeoffs
2. Frequent developer turnover / ramp-up time
3. Developer + engineering manager level of competence, mostly due to people going where the money is and it being difficult to assess skill level
4. 4-year degrees teaching very few applicable real-world skills and bootcamps teaching even less
Steve Mconnell dispelled many of the misconceptions around project planning and estimation around twenty years ago. Worth a read.
I understand your point and estimating is never easy, but software engineering alone won't keep the company a float
Although not all future projects are of a similar scope with similar people as a previous project but when it applies it seems to work well.