Forecasting with uncertainty | Hacker News Reader

Forecasting with uncertainty

causal.app

84 pointsLukas199420 comments

Hey folks, I'm one of the authors of the post. Would love to hear people's thoughts, and happy to answer any questions :)

If you liked this, then I'd definitely recommend reading "The Flaw of Averages" by Sam Savage. It talks about this stuff in a lot more detail, and the style is very entertaining — not at all what you'd expect from a statistics book.

Is there anything causal as in Pearl’s graphical models or the potential outcome framework?

Afraid not, sorry — right now it's more of a Monte Carlo simulation tool. Can see how the name might be a bit misleading!

If causal.app does not make causal statements of that kind, which kind of causal statements can it make then?

Not involved with causal.app, but I see it as going the other way as far as causality goes. Pearl-style models are trying to infer causation from data. Here, you give it a causal model that encapsulates how you think a scenario works (which variables affect the outcome, how, and a range of likely values for them), and it simulates your model to give you ranges of likely outcomes, plus some other things like sensitivity analysis (which variables impact likely outcomes the most). I like the comparison they make to spreadsheet models. It's that style of modeling but with Monte Carlo simulations, so you can put ranges instead of single numbers in cells, and the ranges propagate through the model to output ranges.

Couldn't have put it better myself :)

This is great. I think you're well positioned to help people understand this very important concept in a familiar way.

Have you thought to incorporate the "Minimise Maximum Regret" concept? I suppose it is quite an opinionated strategy, but it is a very interesting approach to decisioning, and supposedly leads to robust decisions.

In the visualisation it would be very useful if the "best/worst" cases included the inputs that were relevant to make that happen. It is somewhat obvious, but would drive the point home that should X & Y occur, you'd be left in scenario Z.

I like the concept. Who is this targeted at though? Having been an analytics mgr at couple of ad agencies my common questions were...

Given current impression share for my campaigns, what would incremental spend do for CPA and position?

What would be the expected CPA to move from avg position of 1.8 to 1.4

Can I tie CRM data to get value of leads/opportunities and optimize for that?

I would imagine it is targeted at a similar audience, it just lets you answer a better question.

Given the current impression share for my campaigns, what would incremental spend do for CPA and position with 95% certainty?

What would be the expected CPA in 50% of cases to move from avg position of 1.8 to 1.4? This question can be asked in reverse again with a degree of certainty attached.

The answers to the questions you posted are likely to be averages and possibly medians if the posterior is Gaussian (or close enough). Mixing uncertainty into the equation allows you to provide a floor on probabilistic expectation, it also allows you to ask what the best case is, say the 95th percentile.

I've stuck to 95 and medians here but you can use any percentile you'd feel comfortable with.

Glad you like the concept! The post, and Causal, are targeted at anyone who needs to work with numbers —

With those ad questions, there's no getting around the fact that you need to come up with a 'model' for how, e.g. incremental spend impacts CPA/position. Once you have a model you're happy with, there's always going to be uncertainty around aspects of it, and you'll ideally want to account for it. Simulation is a good way to do that.

Causal has data connections that you can use to pull stuff from your CRM. Depending on what you mean by 'optimize', we might be able to do that too — happy to chat in more detail if you're interested: taimur @ causal . app

yeah i suppose i was just wondering how flexible it was and how it fits in the day-to-day of addressing client inquiries. and by optimize in this sense i guess i meant analyze - as in tie that data into forecast projections. i'll play with it and see!

I'm an advertiser, worked agency & in-house, and have used approaches like Causal's in the past.

As the other comments mention, simulations aren't going to let you answer these questions if you can't already answer them, but they can provide you with uncertainty ranges which are valuable in their own right.

Particularly if you're agency-side, being able to go to a client with an uncertainty range for some forecasted KPI is so much better than having to give a single number. Works for them because they can understand the risk, and works for you because you're not going to be held to a single number (which you're never likely to hit in practice).

True - i wasn' really asking about the methodology in question moreso the linked-to SaaS offering.

I use these techniques for day-to-day estimation in software development, usually for things like project timelines, system requirements, etc.

I often run into developers who think "there is no way to estimate when we will be done". What I think they usually mean is that they cannot estimate a single time in which it will be done, which is completely fair. However, if they are being honest with themselves, I think they can often estimate a range of possible outcomes. Adding a new "Contact Us" form may take a few hours or a few days, but it isn't going to take 1 year.

One of my bigger successes with this kind of model was sizing the resource requirements for a brand new high-performance system that was in it's early stages of development. I created a model that started with what we knew so far from out performance testing, with uncertainty for everything, and projected forward to different confidence levels of sizing. We ended up being pretty having just a little extra, which was perfect considering the multi-month procurement process.

Very cool! What tools are you using for this right now?

For my more routine analysis of LOE/budget, where it is mostly a straightforward summation, I have a template spreadsheet that I just reuse. For more complex cases, I hand write the equations and error-propagation into a custom sheet. A little time intensive, but it gives me an excuse to practice multi-variable calculus.

too bad casual.app is so tightly coupled to the google ecosystem, even enough to say it's optimized for chrome (terrible for privacy/confidentiality). we really need deeper and wider understanding of decision-making under risk and uncertainty (see over-reaction to covid by many, under-reaction by some). tooling like this can help.

it's hard to evaluate the modeling capabilities from a few pages and blog posts. how does it compare to crystal ball on excel, for example?

Causal's a bit like a spreadsheet and a bit like programming — you can use it to construct arbitrary models by creating "variables" and linking them together with "formulas".

Compared to Crystal Ball, it can do all the distribution stuff — variables can take on any kind of distribution you define. But we (try to) make it trivial to use these — you can write formulas like "5 to 10", and we map that onto a sensible distribution for you, so you don't need any probability/stats knowledge, whereas CB is a lot more technical. The CB suite also now includes some other tools that can do things like optimisation, which Causal doesn't do yet.

Causal works in most modern browsers, and we're working on handling non-Chrome browsers better :) Not sure what you mean by "coupled to the google ecosystem"!

There's also Guesstimate, which I think is similar: https://www.getguesstimate.com/?ref=producthunt

Reply on news.ycombinator.com