The longer it has taken, the longer it will take (2015)

Some usual questions that come to mind...

Are the individual metrics/incentives aligned with progress on the project? Is anyone panicking? Is anyone trying to appease someone else who's panicking, by giving an inaccurate appearance of progress/work? Is anyone stalling for time, while they try to move elsewhere?

Is there a project plan with work breakdown (and interdependencies, and preferably resource allocations) that shows what you've done, and what you still have to do? Are the incomplete tasks broken down to the resolution of 1-2 days, or to hours? Is the plan complete, or is there substantial work to do that doesn't appear in the plan? Is the completion on the plan accurate? How frequently is slippage being checked, and how does that happen? Does anyone have incentive for the plan be inaccurate at this point?

Are people working on unimportant things because they're blocked on important things by dependencies on other people, but don't want to say it?

Is everyone still putting in full effort, and committed to the project success? Or have some given up on the project, and are focused on shielding their careers?

(Even if the project effectively isn't working from a plan, has bad morale and panicking, there are conflicts of interest, etc... it might not be too late for a good manager to rally everyone around an achievable new plan, possibly including revisiting the requirements. Given that the project is in trouble, I suspect it would need believable buy-in from upper management and the "customer" for this project, or people will still feel doomed, rather than focused on making it work.)

I think a non-trivial part of why deadlines are always wrong is something akin to the Coastline Paradox [1], where the closer you are to something, the more details there are.

At some point you have to just say, "Fuck it. It's good enough" and leave it as the terrible flawed pile of crap it looks like at the detail level you're at.

Often once you step back and look at it from a customer perspective, you had just gone in too deep and they didn't need half of what you were preparing for anyway.

[1] https://en.wikipedia.org/wiki/Coastline_paradox

I’m beginning to doubt the reality of anything ever being done in the first place. Done software is dead software that nobody uses anymore.

It seems more fruitful to talk about gradients and equilibriums of desired changes.

A reference to Jaynes. :) Have an upvote!

So in my limited understanding, the exponential distribution popular because it's very easy to work with -- you can actually analytically solve some queueing problems, for example, that wouldn't be possible with other distributions.

But power-law distributions show up over and over in things that people here care about: file size, network traffic, process lifetimes, etc etc. In these cases the exponential will drastically underestimate the fat tail.

Worse, is when folks take a late project and add people to it. They don't realize this is just adding to the work, so will almost certainly just increase the time.

You may get lucky and one of the new additions throws out the current plan. Cuts from the sunk costs. Probably, though, that will be its own trap. You'll think you understood your success for the next time.

That’s Hofstadter’s law: “It always takes longer than you expect, even when you take into account Hofstadter's Law.”

Probably a year or more ago I compared some data on FOIA request processing times reported by US government agencies against an exponential distribution (not power law). I used the given mean and compared the predicted median against the actual median. Was acceptable as far as I was concerned, though I'd have to redo the analysis to actually get quantitative.

With a power law there's no mean or median starting at t = 0, so this sort of comparison isn't possible given the FOIA request data available. I'd be interested in seeing the data on the tails to see if those are power law distributed.

Edit: I'll do a quick quantitative check. Using data from the CIA's 2018 FOIA report (p. 18): https://www.cia.gov/library/readingroom/foia-annual-report

Simple track:

Mean: 32.21 days

Median: 12 days (exponential prediction: 22.33 days, 86% error)

Complex track:

Mean: 368.49 days

Median: 306 days (exponential prediction: 255.42 days, 17% error)

Not the best fit but could be worse.

The life example is weaker than the project example.

In the project example, not only does the total project time increase with spent time, but the time left increases with spent time.

That is true for any distribution. The mean (expectation) of the part after X+epsilon has to be larger than after X. There is no possible age distribution where your claim isn't true.

My guess is it is a statistical artefact. If you have a project and you pick a random time in that project then you expect to pick the midpoint.

So, if the only information we have is that the project has gone on for 2 years we expect that to be the midpoint and that the project will continue for 2 years.

In particular if the project manager has lost control of the project (eg, not scheduled in a contingency, missed requirements, etc) then there is no reason to believe that anyone knows what % of the project is done. So assume 50% because that is the Most Likely Estimate. And be surprised at how often that is the right guess in my cynical and not inconsequential experience :p.

If the project manager is in control (usually evidenced by people opining that the project will finish early) then expect the project to finish exactly on time when something unexpected goes wrong.

Not just intellectual endeavours, the planning fallacy shows up everywhere.

I have a book called Industrial Megaprojects by Murrow[0] which gives a lot of examples of financially disastrous multi-billion dollar projects. His conclusion? It's not really the doing of the project that was wrong, it was that the projects shouldn't have been done in the first place. The estimates and preliminary investigations were underdeveloped and this typically leads to overoptimism.

Bent Flyvbjerg has also done a lot of work on megaprojects[1] and his basic conclusion is that well-estimated projects don't get built, because almost no megaproject is ever viable or cost-effective in itself. So there's a survivor bias towards "bad" projects, ones that are poorly planned in the first instance, magnifying the effect.

In terms of software, agile/lean has the advantage of limiting commitment. It's easier to terminate something that's cost very little and not gotten far than to terminate something that's several years and millions of dollars into going nowhere. The former can be dismaying and annoying. But by the time serious time and money have been spent, there's a sunk-cost fallacy and often, personal pride or status of powerful individuals is involved.

[0] https://onlinelibrary.wiley.com/doi/book/10.1002/97811192010...

[1] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2424835

Your "exponential timer" page 404s, unfortunately.

Hofstatder's law: It always takes longer than you think, even after accounting for Hofstatder's law.

Hey mentions the Lindy effect. I don't think I learned anything new here, except there's a scientific name for something pretty intuitive.

I think that's definitely a factor. I came into a project once where that had happened. The initial estimate was a mistake and the contractor decided to up the estimate incrementally whenever they reviewed status. The client became annoyed because "every time they estimate completion, it is taking longer." (I had worked for both and came in as a third party and due to that, was able to get things back on track.) The initial situation brings me to the other major factor I see in these situations. Anything that was overlooked or comes up unexpectedly results in more work. The situations where unexpected situations result in less work are as rare as hen's teeth. This results in underestimating more often than not.

Sell a lot low, deliver less late

I think what this points out, which may be counter-intuitive, is that if there is sufficient complexity in the project that has driven it to be late, there is likely hidden complexity that will push it even further.

Complexity grows exponentially (a graph of inter-relationships) and so even cutting a feature isn't enough to curb the ballooning complexity of an over-scoped or poorly estimated project.

I could be projecting from past experience ;)