Hacker News Reader

Comment by commandlinefan | Hacker News Reader

You don’t hear the name Joel Spolsky much any more, but he was pretty influential in software process thinking in the 90’s - not really for being particularly insightful or original, but more because he was one of the first people who thought of writing a blog about software design. One of his early “observations” about software project management was that “you wouldn’t buy a pair of jeans without knowing how much they were going to cost, why would you expect your manager to sign off on a software project without knowing how much it was going to cost?” The utter blithering stupidity of this perspective highlights how managers look at software design: the purpose of software methodology is to put a fixed cost on every software project. As long as that’s what they’re looking for, software methodology is going to continue to be a lost cause.

View on HN

Your argument here is against "business". In a dream world, where we could build for building's sake, and create "more perfect" things with no considerations for cost, I'd agree with you, but businesses (in the abstract) are a math function: Does it cost us less to make this than we charge for it? If so, we are a profitable business, if not, we are an unprofitable business.

Managers are there to manage costs, to protect the "business".

Casinos are quite profitable, yet they have no idea if the next card being drawn will be profitable or cost them thousands of dollars.

I think every owner of a casino knows there is luck involved in their profits, and that the odds are stacked slightly in their favor.

I don't think any owner of a business thinks that there is luck involved in creation of their software systems. They absolutely look at the $10 million per year they are spending on "IT" and look at the $26 million per year in value that spending is unlocking in absolute dollar terms.

> thinks that there is luck

You mean "realizes there is luck". And hence the problem.

I'm sure they have very good idea. Statistics, game processes design, everything is engineered towards stable and predictable profits.

But they know that the sum of all cards drawn will be profitable, because their practices guarantee it. Normal businesses cannot guarantee that the sum of all their projects will be profitable.

That's one of the big reasons that I've become a proponent of Donald G. Reinertsen's approach to product development.

He emphasizes a variation of the cost benefit formula + urgency, called Cost of Delay for prioritizing that basically boils down to "Value / Time Remaining" because your cost is the time remaining. It naturally prioritizes work based on shortest time to value because, unless the value proposition is enormous, dividing by estimated time is almost always going to favor the shorter tasks.

This will of course vary by team, but in general it puts a focus on responsiveness and getting the little stuff out of the way first (including tech debt). When you have a formula that doesn't let the little stuff build up, it gets easier to focus on the big stuff too. It's also make it easy to pivot because it totally disregards prior time investments.

It meshes better with what agile is supposed to be better than any other methodology that I've encountered to this point.

> including tech debt

Tech Debt is not little, or we wouldn't carry it around, rather than devote time (and risk of revealing new unknowns) to fixing it. My agile experience has always favored conquering little stuff first, since ppl recognize stories more than points (which vary in value, over time). This, unfortunately, sometimes leads to the hilariously subversive act of splitting a story up into as many tiny pieces as possible so that some people may inflate that "What have you done for me lately" list while more tangible coding is done by others.

Cost of Delay metrics do a good job of explaining how the cost of that debt can grow over time the longer it goes unfixed. Some are smaller than others, but uniformly it tends to get harder to fix the longer you wait to fix it.

He also doesn't place any type of value in points or other, bluntly, useless metrics for measurement. The entire focus is around delivered value. You can break it into however many stories you want, the value is the same. The measurement of how many things you delivered or how much time they took is not important to the company.

What's important is how much value was been delivered to the company. It's the only metric that matters.

splitting a story up into as many tiny pieces as possible

Honestly I think that's the intended outcome from the business's perspective. The smaller the piece of work, the easier it is to assign a cost and a benefit to it, and the easier it is to hand off to another developer.

>... and the easier it is to hand off to another developer.

I have never seen this work out in practice. And it's like a 1%-of-the-time thing anyway.

On the flipside, I've seen quite a few hours wasted adding fake subdivisions to tasks, probably close to 30% of the time. Outside of small, known bug fixes or tech debt cleanup, it's rare for a thing to wholly fit within a sprint.

The subtler point is that if the developer won't make an estimate for how much time a feature will take, than the manager will. And the manager's estimate will tend to be even worse than the developer's, so we should all prefer that the developer makes the estimate even though it will be a bad estimate in an absolute scale.

Note that the manager's estimate may be implicit or explicit, conscious or unconscious, but they will make one in the absence of the developer's. An implicit estimate would look like the manager doing a "gut call" when deciding whether or not to give the go-ahead to a feature, with no numbers attached to time and cost.

The real problem is treating the estimate as a commitment, but Spolsky does rail against that.

The problem is that if the developer gives an estimate, the manager might hold him accountable to it. If the developer can't even meet his own estimate, that proves to the manager that he's a bad developer. (He's not; he's just bad at estimating development time, as is everybody else.)

And that's what Scrum and story points are for: to help you make long term estimates once part of the work has been done. You use the feedback on how abstract story points turn out in practice, to get a rough long term estimate of how long the work is actually going to take. Without being so explicit that anyone can be held accountable to them.

What's stupid in wanting to get an idea of how much a project costs in order to decide if it's a good idea to pursue, let alonr allocate resources?

Nothing, inherently, except perhaps the fallacy of comparing a pair of jeans to the rather chaotic and unpredictable world of bespoke systems development. One is inherently known (a pair of jeans you've presumably already manufactured), the other is one big unknown, basically.

Aircraft, hospitals, hotels — a million complex bespoke things are built every day. Cost and time projections are inevitable.

> Cost and time projections are inevitable.

As inevitable as blowing through the projections.

Right. You can imagine a conversation something like:

"Ok, how long is this project going to take?"

"Well, it's still being defined..."

"What do you mean you refuse to tell me how long this project is going to take?"

The analogy is so ridiculous that the person making it clearly doesn't even comprehend the domain he's discussing.

He compared to jeans because they're an extremely trivial purchase that someone would give little thought to, but still demand to know what they cost. The point is that in all cases you need that information, with a major software project you simply need it far more.

> you need

And that's the core of the problem (which, ironically in the context of the article, agile methodologies were supposed to address): wanting or needing something to be true/exist doesn't make it so.

I think maybe the point they're trying to make is that comparing a software development project that may include a number of unknowns to buying a pair of jeans where there are probably very few if any unknowns is fallacious and a waste of time.

That said, I don't think it's black and white. Some projects, making a simple web site that follows whatever Squarespace template is popular and filling it with some relevant content can be a pretty simple project, even to the point where the buying a pair of jeans comparison makes more sense.

The problem, I suppose, is when you believe strongly that every project can be accurately priced up front. Not every project is a moonshot, but some truly are and it's not always obvious up front which ones are.

And every engineer likes to pretend that their project is a moonshot.

If you are inventing something incredible,why are you someone's employee instead of getting capital investment or self-financing from your last achievements?

It is when stakeholders throw curveballs.

"We want you to do this."

"Sweet, easy enough, I'll just use X, Y and Z, no problem."

"Oh but sorry you have to use A, B, and C instead. Oh and it'll need to connect to D for reporting, and we'd like it to sync with E which we've never done but they have an API so it can't be too hard, right? Oh and it needs to fit this design we've already approved that's about as far from native to the platform you're developing for as it could be and has a ton of UX issues that require fundamental changes to fix, which you'll uncover as you work. That's about as easy, right?"

"...."

[EDIT] oh and also one "minor" feature we're going to toss in later is actually big enough to build an entire company around so you'll have to waste a bunch of time talking us down from that while we quietly lower our opinion of you, and we've accidentally described a few things which are basically impossible by egregiously violating e.g. CAP theorem or being what's effectively a highly-available distributed filesystem that needs to fit without our timeframe and budget, which is 1/20 what it would take to maybe build that and that alone.

Writing software is design, not manufacturing [1]. Do companies know accurately in advance how much it will cost to design a new nuclear power plant, aircraft carrier, or jet engine?

[1] http://www.bleading-edge.com/Publications/C++Journal/Cpjour2...

There are times when something truly original is being built and it's a valid argument for estimate uncertainty. But a significant portion of the industry is engaged in building CRUD app #237 or Ho-Hum SaaS #17, and a significant percentage of estimates end up wrong because some developer who was bored with his job decided to use a blingy new Javascript framework he had no experience with because he wanted a challenge. Money gets lit on fire again and again this way and it's frankly selfish and irresponsible.

If the software is truly that unoriginal, use something off the shelf. The issue is that in literally every non-toy product I've every been involved in the product owners will always turn CRUD app #237 or Ho-Hum SaaS #17 into something significantly more custom and much more complex.

>because some developer who was bored with his job decided to use a blingy new Javascript framework he had no experience with because he wanted a challenge.

That definitely happens, but I think it's a different problem.

> The issue is that in literally every non-toy product I've every been involved in the product owners will always turn CRUD app #237 or Ho-Hum SaaS #17 into something significantly more custom and much more complex.

YES! The cost of custom-everything design and bespoke gee-whiz is vastly under-appreciated, IMO. CRUDy Android and iOS apps using built-in widgets and simple color theming? Relatively easy to estimate, and pretty damn fast. Custom-everything, we want both to look pretty similar (so, probably Material-ish given current trends, which you'd think would come for free at least on Android but... doesn't), custom animations everywhere, oh I hate that default date picker on this particular Samsung phone (and sure, it's god-awful) can we customize it, and the designer came up with this layout for this form that requires all kinds of twisty-bendy manipulation to reproduce versus something more straightforward but that's what the "stakeholders" approved so let's do that.

Et c. et c. and pretty soon the app's 10x as expensive (no exaggeration!) because you couldn't live with the easiest-for-the-user-anyway default styles with some custom coloring.

And the Web's at least as bad. Often you're also making your page way less usable and breaking it on certain browser/device combos due to the extra effort you put in to make it "pretty", at great cost.

[EDIT] in fact I've in the past suggested providing design points for "pretty gee-whiz versus not-actually-bad straightforward implementation" to make product owners choose between getting e.g. three normal (and OK-looking but not "pretty") features or one pretty feature in a sprint, to make this cost more visible and give owners on a budget more flexibility in shipping features, but it never caught on—in particular designers hate, hate, hate the idea, I think because it might expose that their work is sometimes (often...) more expensive than it's worth (and vastly more expensive than they're being paid, since it also eats huge amounts of developer time) given you've got some developers with even a hint of aesthetic and UX sense on a project.

I wonder if you looked at time estimates from developers of common process applications (like CRUD) in say, the 1980s, who used COBOL or FORTRAN, vs say those using Pascal or C - if those developers of code using extremely well known processes and libraries had better estimates than those using the "newer bling"?

Design or repair.

These are the same people who get furious at their mechanic or plumber when they double the time estimate.

"How long will X take?"

"I don't know, I've never done X before."

"Yeah, yeah. It doesn't have to be perfect analysis, just a rough idea for scheduling. Not written in stone, ha ha."

"Uh, a week?"

"Okay, great, so if I say 8 days you should have it done by then?"

"Yeah, I hope so."

"Great, thanks."

Day 3: actually doing X requires unforeseen Y and Z which will each take a month.

"We need to add Y and Z to the schedule, which will each take a month."

"There's no room for that in the schedule."

"Then I can't do X"

"We've already costed X and committed to it. We've got to do X. We need to make X happen. When can you get back to me with a path to green for X? And also, let's schedule a post-mortem to figure out why we missed on X."

At the post-mortem, PM doesn't show up: "Uh, turns out we missed on X because I estimated it too quickly and the PM committed to those estimates. I didn't know about Y and Z when estimating, and these are things I could have done to detect them as requirements if I had known to do so."

Email from PM: "Hey, could you give me an estimate for X'? Doesn't have to be perfect, ha ha."

Yes. This happens at my shop all the time and it's absolutely terrible. Each sprint starts out positive and ends with panic attacks.

I've solved this by not panicking about software someone else owns. Worked out OK so far. They can fix their processes and expectations (I'll help!) to get more value out of me, or choose not to. Not my problem. Environment gets too toxic, well, see ya, and good luck (you'll need it).

"A week" is a terrible estimate for complex things that have components that you can't foresee. If you lose "a day" on some problem outside of your control (compiling starts to fail and it turns out someone checked in some code into your branch by mistake), you've basically wasted 20% of your estimate.

Never say a week when there's the chance something will require a month because you didn't understand it.

You need to learn this lesson. A week is a blink of an eye in software, and only things you understand take a week.

"Really? X will take 'a month at least'? Why?"

"I don't know why, but I don't feel comfortable guessing less because I don't fully understand X."

"Okay, that's not a good way to estimate. Why don't you spend some time to come up with a plan for X and we can estimate it after that. When can I have the plan?"

You can have the plan in a week.

> "How long will X take?" > > "I don't know, I've never done X before."

I fail to see how your example makes any case on how estimating how many resources a project needs is stupid.

At most your example argues that asking inexperienced and clueless devs to estimate stuff they know nothing about produces highly unreliable information.

The main difference between hacking away at a code base and software engineering is identical to the difference between construction workers and civil engineering. Sure, estimating is hard. Yet of you want reliable estimates you need to check with experienced professionals.

I've worked about 10 years at FAANG companies and I'm yet to meet the experienced and expert devs who are good at estimating things. I think your characterization of "inexperienced and clueless" devs is off.

The reason I think my example illustrates a problem is it's moving away from agile. I shouldn't estimate X without understanding it? Okay, I'll take the time to figure out, look around corners, etc. I realize I need Y and Z. I've got to estimate those, to estimate X, so I start working with other teams to figure these things out, and drawing up plans and schedules and who's doing what when to get all this stuff delivered. Now we're basically at the waterfall model where we're planning everything out before doing it.

The other problem is that all the process is genuinely a waste. If we need X then let me work on it rather than do all the planning and scheduling nonsense. The schedule will be wrong anyway. If I have problems, need help, etc with X then I can raise those concerns and management can react in an agile way.

> If we need X then let me work on it

If you look through the managers who are weighing in on this thread, you’ll notice a common perspective: they don’t trust you to actually work on something. That’s why I have so little hope for any software methodology - even if it starts from something positive (like XP, which was the precursor to “agile”, did), it will be turned into “I know all of my programmers are stealing from me, how can I stop them?”

In part because we still measure effort in hours instead of in wear and tear. That guy answering comments on Hacker News is probably trying to recharge, not steal from you.

I learned recently that the crane operators that unload cargo ships at some major ports can’t work more than four hours at a time. The movements are too precise and take a great deal of attention. I think you can work two shifts per day but there’s a mandated gap of some number of hours between them.

> At most your example argues that asking inexperienced and clueless devs to estimate stuff they know nothing about produces highly unreliable information.

Wow, so much hubris in such as small post.

Unless you have been doing the exact same thing forever(in which case you are at a risk of being replaced), new work _always_ comes with a degree of uncertainty. Be it new technologies, new business requirements or what have you. I have never worked in two identical projects in my life.

The correct answer would be a 'spike' or proof of concept. Either that, or hire those magical developers you seem to have who know everything there is to know.

This would've worked if you'd taken your initial estimate (1 week) - bumped it to the next unit (1 month) - then doubled it (2 months). Maybe you would've been given 9 or 10 weeks, so you'd have the week for X, and 1 month each for Y and Z.

I understand this is a contrived example, but it's strange that, when you take a real project that "overran" based on the initial estimate(s), and you apply the formula - just how often the formula is actually really close to the actual amount of time.

Not always - but more often than not.

As a consumer, I can buy an already manufactured pair of jeans or an already programmed Adobe Photoshop for a fixed price.

As a producter, the cost of building a jeans factory does not seem to be inherently more predictable than the budget of a software development project.

I think, to take this analogy a step further, you need to make sure your manager realizes the factory is not built yet.

You think when Tim Cook signs a contract with Foxconn, he says "charge me whatever you want after it's done"?

The recent Foxconn plant in Wisconsin that may or may not be happening is a great example. They committed earlier, and claim now that their initial estimates were incorrect and that a plant in that area will not be profitable for them. They have now stopped the plan entirely and reworked what it is they are going to build completely from scratch. This despite having already committed to building the factory.

Even at scale people can pull out of complex agreements if their estimates turn out to be faulty.

You should be able to put it in a range otherwise you probably don't know enough to build it. You don't want to spend 100 millions on something that will save 1 million a year.

managers and business leaders make decisions based upon those estimates.

Imagine if that cable guy coming your house said between 1 hour and 2 weeks. Good luck making anything approaching a good decision surrounding the cable guy's visit.

I've been a professional developer/engineer/programmer/whatever for 15 years. That doesn't seem stupid to me.

Reply on news.ycombinator.com