Show HN: Monte Carlo simulation for predicting project timelines
quotes.vistimo.com
quotes.vistimo.com
"At a county fair held in Plymouth, 800 visitors had participated in a competition to guess the weight of an ox. While most people's estimates were too high or too low — falling an average of 37 lbs. away from the true weight of 1,198 lbs. — the median of everyone's guess was off by only 9 lbs., or less than 1 percent of the true weight of the ox."
Curiously there is also some "within individual" averaging that helps too. If you would ask the same person to estimate the project timeline a few times, spaced temporally, then take the average, it'll probably be more accurate than a single estimate.
[0] https://www.npr.org/sections/13.7/2018/03/12/592868569/no-ma...
But Vistimo is a full project management tool; that space is full and switching project management tools is expensive, so it's hard to get people to adopt.
Vistimo Quotes is a much lighter-weight way to get many of the same benefits without having to talk your whole team into signing up.
For each task you have one distribution, and one sample. What's the likelihood that sample was actually from a different distribution?
Instinctively it feels easy to judge "your usually closer to the worst-case than the best-case" but mathematically it's not so straightforward, especially when the estimate doesn't include sick days, splitting time with other projects, etc, but both the projection and the reality do.
Getting a data scientist to estimate how long the front end will take to build will not be as helpful.
37 lbs seems astronomically accurate unless this is a county fair of farmers...
https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
I'm working on the assumption that it hasn't caught on because it's a pain to do without proper tooling; but it's also true that many engineers believe estimation is useless, impossible, and a waste of time or worse. It can be hard to overcome the pain of having been burned by a too-tight timeline.
It gave managers answers they didn't want to hear.
In my experience, statistical modeling of potential timelines is not the bottleneck to accurate quotes: it’s the ability to actually give good guesses for “min time” and “max time” on critical tasks. I.e. the majority of timeline variability is due to just a few “tasks” that totally blow out of proportion because I underestimated how much work was actually involved. Idk, maybe that’s just me.
It would be nice to track my estimates vs actual time spent to improve the “human error” part.
Big red flag on putting anything serious into it.
It was actually quite powerful: the UI and simulation were done by Microsoft Project (https://www.microsoft.com/en-us/microsoft-365/project/projec...), which I would describe as "the Excel of project management" in that it was an ENORMOUS bag of features for building Gantt charts. The coolest part was that it integrated with Outlook's calendar, so if a task started on June 1st, was assigned to Bob, took two weeks, and Bob was on vacation from June 8th-22nd, it calculated the end date for the task as the end of June.
The Monte Carlo part was run by Crystal Ball (which was a startup then, but has since been acquired by Oracle). It was a mature tool, so there was a big library of probability distributions (https://docs.oracle.com/cd/E57185_01/CYBUG/define_assumption...), you could enter correlations (if X task takes a long time, Y is going to take a long time too), and you could analyze the output in all sorts of ways (how sensitive is the deadline to this task, how frequently is this task on the critical path, etc.).
The implementation was quite a hack - a hidden instance of Excel running Crystal Ball and lots and lots of VBA to manage everything. I'm not sure that it was ever used for real :(
- https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
- https://support.fogbugz.com/hc/en-us/articles/360011258994-E...
- https://blog.fogbugz.com/evidence-based-scheduling
There’s at least one Jira app for it:
- https://github.com/handshake/pystradamus
And Trac also used to have EBS but I’m not sure if it still does, can’t find it atm:
It turns out the project management space is full of neat tools with clever abilities, but it's very hard for a team to switch between those tools. It's expensive and time consuming!
Vistimo Quotes is an attempt to pull just statistical estimation out on its own, so you can use it no matter what you use for your project management.
(A note: this and EBS are pretty different in their actual math -- they just seem similar because the next closest thing is "everyone agrees on a single number per task and we add them up" which is so limiting that just having a statistical model puts EBS, 50/90, and monte-carlo into their own category.)
https://blog.vistimo.com/post/169546687605/estimation-math
Note that Monte-carlo is not used prominently anywhere on the site, just here because I thought HN would find that part of the implementation interesting.
The only thing I would worry about then are events which have low probability but potentially very high "weight" in whatever measure of risk you are using. Sometimes the weight can be so large that your MC estimates seem to converging, and then the addition of a single trajectory completely blows up your variance. These are related to the "black swan" events people sometimes talk about. I don't think this is a problem in your model.
Thanks again for ansnwering.
One question that I do have about these types of tools... how does one evaluate how well it worked? I guess in the end as long as the timeline wasn't too far off then you call it a win.
I think a useful tool for project planning is not necessarily simulating possible time-lines (or a estimated window of time for the project to be completed) however to intelligently identify a set of critical tasks within the list which would have a high probability of totally blowing up your time line their deadlines are missed. In project management / scheduling there are often critical path items listed. However often these critical items are semi-arbitrarily picked by the PM and not backed by and deep insights.
The audience reaction was very positive, and I was pleased to have spread the word.
No one actually put it into action, though. It turns out if you need better estimation tools, you don't have time to make them -- you're in crunch time, because your estimates were bad.
The benefit is not so much slick UI -- that part is just me having fun. The benefit is that the work, however easy, is done.
If folks "steal" this idea because it's easy to implement, that's great! It's not a new idea, just an underused one. I just want to stop being hired onto contracts where the first thing I have to do is explain that the deadline is impossible.
I agree with you that a linear backlog is not sufficient to do really sophisticated planning -- Vistimo Quotes is a pared-down tool designed for easy adoption, because my kitchen-sink version requires too much buy-in for many teams.
(And honestly, sometimes I worry the UI is too slick -- that it will give people expectations the tool won't live up to.
But I have too much fun making fancy buttons, I can't bear to get rid of 'em :) )
This describes about 99% of all successful companies. Exceedingly few companies invent truly new technology, and that's perfectly fine.
On a more serious note, good idea and nice site. I may give this a try. I have been thinking along these lines myself, using simulations to complete different dev related tasks
Lots of people have expressed interest in Jira integration, and that's definitely on my list to accomplish one way or another. The Atlassian marketplace is like most app stores, though -- if people already know your name, it's easy for them to find and install your app. But if you're a relative nobody, there are 50 other estimation apps, and very little room to pitch what makes you unique.
Biggest issue I have is with the task list, for any reasonably complicated project we would at least need ability to group tasks into features or milestones. Task lists can get pretty granular, but communication to stakeholders or clients does not usually need to be that detailed, higher level groupings are preferable (and also makes it easier to organize the tasks).
As others mentioned, integration with Jira and such would be nice, but not a deal breaker.
Also I think I broke my demo project, I changed one estimate and the projections just say "New projections are being calculated". There are only 4 tasks :)
I totally agree, personally I want more hierarchy; but user testing with milestones was pretty rough. Vistimo proper (https://www.vistimo.com) is less slick-looking, but has milestones and story-mapping and bugs and enhancements and all kinds of hierarchical organization tools. Most users I've tested with have found Vistimo Quotes' approach easier to pick up and use on existing projects, even if it's less powerful at expressing the full richness of the work.
Re: the long running calculations, I will take a look, but it's possibly just a traffic jam. There are a few more "test" simulations running simultaneously than usual this morning.
You can fix those things, have fully signed off design docs, have a CCN process for every little change etc. and watch everything grind to a halt while you deliver the wrong thing on time.
I’m not saying don’t estimate. Do! Have a plan and realise it won’t survive contact with reality.
A question though, isn't Monte Carlo a bit overkill? Would you lose much by modelling each deadline as a normal distribution? Then you could just calculate things analytically.
And once you're committed to using monte carlo, you're free to include one-off events -- sick days, vacations, social loafing, etc, that are really hard to model analytically, since they are roughly uniform in time -- the longer the project runs, the more days off make the project run longer.
As another alternative, could you convolve the probability distributions numerically? I’m not sure of the trade-offs between that approach and Monte Carlo, but if it’s something you’ve considered I’d be interested to hear your thoughts.
And everything else, all the statistics etc, can stay exactly the same. And I can have a lot of trust that it's producing sane results because I can look at a run, and say yes, that's a plausible number of sick days.
Monte carlo requires you to be clever once, setting up the simulation, and then you almost never have to be clever later when you want to introduce new twists. Any other statistical model you have to keep being cleverer for every additional complication.
But sure, Monte Carlo is more flexible, and not really harder to implement I suppose, since you're already on a computer!.
Are you able to share the task list so that each project member is able to fill out their best and worst case estimates on their own in an async fashion, preferably anonymously?
Vistimo Quotes is specifically for producing rapid estimates that don't require team buy-in, sign-up, etc.
Vistimo (https://www.vistimo.com) does anonymous, asynchronous team estimation -- but the additional labor costs (sufficiently describing tasks to async folk understand them, managing team permissions, requesting reestimate when requirements change) have made it not particularly viable as a product.
Finding a balance of power and simplicity is never easy.
It's possible that people may give better estimations when thinking about the number of tickets because then you immediately have to think about the number of individual components whereas with time, you're going more by feeling.
As a side note, have you seen/heard of (sic) "Vistima" parody? https://youtu.be/4jVQhdlCmjc
Congrats, Stay Thirsty.
If you mean you want technical details of the math involved, https://blog.vistimo.com/
(Fetching those projections uses websockets, so you may need to refresh if you're behind a particularly restrictive firewall.)
If you're still having trouble, drop me a note at sam@sambleckley.com and I'd be happy to help.
The "how does it work" section doesn't even mention
* modeling assumptions
* training data (if any)
* validation (if any)
Why is this better than, say, giving the 25-75% range around my "best guess" assuming log-normal duration and the variance of prior project completion times? Can you demonstrate that it's better than that?
If you're already doing log normal modeling of your estimates, and including sick days, social loafing, etc, then you don't need Vistimo Quotes. Just keep doing what you're doing!
This is something I would buy if I had confidence in the simulator.
Why not write a blog or a brief white paper giving some hint of what's going on behind the scenes?
Generally speaking, every bit of energy someone spends being cool (good UI) comes at the expense of energy spent being good. Maybe you've done some serious work on the simulator, but I can't tell from your marketing material.
Why would you assume that? If your model is already pretty much perfect, not much more you can spend your time on than presentation.
I'm not saying you can't make money selling an RNG with a slick UI, you certainly can.
I can't comprehend I'm expected to write them out one by one with no ability to import that I can see.
It’s one thing to type in a few tasks on a small project but it’s completely unrealistic to expect me to manually re type 100s of tasks.
I'm working on CSV export, first, but I will absolutely keep this in mind.
And the great thing about being able to run estimates quickly is that you can tell the CEO how much their new idea will send the timeline off course as soon as they propose it, so they can decide if it's really worth the cost.
I've been using this tech for years as a consulting engineer; this is just the dressed up version. It's been useful and effective for me, at least.