Against predictive optimization

OP here. Many people are reacting to the title of the paper. A few thoughts:

* The paper is 35 pages long and it's hard to convey its message in any single title. We make clear in the text that our point is not that predictive optimization should never be used.

* We do want the _default_ to change from predictive optimization being seen as the obvious way to solve certain social problems to being against it until the developer can address certain objections. This is also made clear in the paper.

* The title is a nod to a famous book in this area called "Against prediction". Most people in our primary target audience are familiar with that book, so the title conveys a lot of information to those readers. That's one reason we picked it.

* Despite its flaws, when might we want to use predictive optimization? Section 4 gets into this in detail.

Thanks for reading.

Is the problem really the predictive optimization algorithms themselves, or rather how we think about these problems / solutions in the first place?

I don’t know, it may just be that I’m getting old (45), but my impression is that there has been a substantial shift in our culture around these issues over my adult lifetime (at least in Sweden where I live). For example, only 20 years ago it was extremely rare that the Swedish police shot anybody, and e.g. holding a knife and being non compliant with police instructions was not considered enough to warrant getting shot. There has been no change to the relevant law AFAIK, but there has been a cultural change where many people now think the police is responsible if such a person suddenly runs away and attacks someone with that knife. And there’s also a constant stream of cases where the police has shot someone essentially for being non compliant (mentally ill, addicts, distraught people and so on).

I think in general 20 years ago in Sweden most people in positions of influence were a lot more principled in thinking about these issues, and the principles were the classic ones ironed out by Voltaire et al. Nowadays, not so much… The thinking seems much more primitive.

One of those principles was that it was wrong to punish (or take similar action) against someone for what they could potentially do in the future. Another was that each person is (with very few exceptions) morally responsible for their own actions. Pretty good principles in my view.

I read the intro and skimmed the paper. The title is a bit misleading.

The authors are not "against" using ML to make predictions about human beings for automated decision-making (e.g., whether to approve a loan, offer college acceptance, offer a job, reduce a jail sentence, etc.). What the authors are against is using ML for such purposes without explicitly addressing multiple common-sense issues that unfairly impact groups of human beings.

Specifically, the authors recommend that those organizations which use ML for automated decision-making to specify how they are dealing with these issues, instead of sweeping them under the rug. This strikes me as a sensible recommendation.

As someone in the field, I can absolutely vouch that stakeholders generally want the issues raised swept under the rug, and junior folks often don’t even consider these issues. So I absolutely agree with the authors recommendation that each of the points raised should should be assumed to be an issue unless specifically addressed by the developers.

It's not that we can’t do this stuff, it’s just that doing better than a human is amazingly hard, and a lot of Good Decisions are AGI-hard (or harder, since philosophers through history still struggle at it). Solving it well at scale might be fundamentally expensive.

If a sales person tells you “just trust me our algo is great,” the burden of proof is on them to demonstrate that they’ve addressed these issues.

This is a fantastic summary of the potential issues by some of the most respected people at the forefront of research on bias and harms in AI.

Some people are criticizing the short page for not being more constructive - there are some constructive things you can do and the authors know them and probably discuss them in the full paper. But there are also cases where using a predictive system is inherently unjust (or it is not feasible to fix its issues) and sometimes you have to make an ethical decision not to deploy (example: automated sentencing or bail decisions).

This is an important argument and I agree with many of the problems they mention. The problem is always identifying an alternative. You could say just don't do it but people will probably do it anyway in a black box kind of way, and so the question becomes what then? What is the implicit alternative, and can it be made explicit or replaced?

The problem is that ML models are somewhat inscrutable and this inscrutability creates a problem for decision-based software. In particular you don't know why the algorithm made the decision. As the author notes, this makes the outcomes hard to contest because it's not possible to know how the decision was made (only in a general sense). For example, you can't know if there is any bias that was taken into account. ML models can absolutely be biased. For example face recognition software has high false positive rates for groups of people that are less familiar. The author goes on to say in the paper that the outcomes from ML seem to not be much better than a human or a simpler statistical model doing the same role - but without the drawback of inscrutability.

Related:

https://aisnakeoil.substack.com/p/the-bait-and-switch-behind...

https://www.sciencedirect.com/science/article/pii/S026736492...

Honestly, the title threw me off at first—I used to work in supply chain optimization which very much follows a predict → optimize pattern, and I think there are interesting directions[1] for preferring a different approach, but the abstract and title made it pretty clear that the work focuses on a fundamentally different area. Supply chain forecasting is qualitatively different from the predictions in this essay because it applies to aggregate behavior, not to individuals.

[1]: In particular, what we did (and what most people do?) is predict first, then optimize. Prediction and optimization were done by different systems, built and maintained by different teams, optimizing for different performance metrics. But, over time, I became increasingly convinced that this separation was not effective and that we could get substantially better performance by jointly optimizing the prediction and decision-making models. I saw a paper[2] demonstrating this idea mathematically, but now I think some variant of this would make sense from an organizational and systems-design point of view as well. If I ever get the chance to build a supply chain optimization system from scratch, I'd want to start with the forecasting and optimization components much closer together technically and organizationally, even if we don't jump into trying joint optimization from the beginning.

[2]: "Smart Predict then Optimize", https://arxiv.org/abs/1710.08005

I think that if you read their list of "Flaws of predictive optimization" as a primer for what developers of automated advisory or decision making systems ought to be thinking about, and ultimately be accountable for, then this is very useful. But then concluding that we should be "Against predictive optimization)" (Their choice of headline), and provide a gauntlet of 23 ways to de-legitimize a predictive algorithm (as in their linked rubric), all of which, in all likelihood, no algorithm can survive, it goes way over the top.

Predictive optimization is a necessary fact of life in many functions. The question is often not "should we do it," but "should we do it with a particular algorithm, in a particular setting?" The fundamental questions we should be asking, from the top down:

1) does making the prediction systematically, and well, serve a useful, desirable social purpose. That is, do accurate predictions actually do us a net good? This can be a hard question - using predictive algorithms to screen for adverse futures, given that all predictions, including human-based decisions, throw both false positives and false negatives, you have to weight whether the harm in false predictions is outweighed by the good in true predictions.

2) Does a machine algorithm improve on human judgement in the question and system in question? Machine algorithms don't have to be perfect to be better than human judgement, which is often abysmal when viewed on a system-wide basis.

3) Does the system into which the algorithm is being deployed provide reasonable mechanisms for detection, recourse and compensation for false predictions? Because, again, there will be false predictions.

A lot of the use of AI in social decision making that the authors criticize would flunk these questions. Not all, though.

Yeah, black boxes belong no where near these fields. Rubric plus human review is much more transparent and flexible

The issue I see is that humans seem to have many of the same problems as the ones they point out ml systems can have.

I initially rolled my eyes at the title, but then the subtitle adds “On the Legitimacy of...” and the rest of the article raises good points, so it’s just the misleading title I dislike. I don’t think there is a good argument against predictive optimization itself, but that’s not what the article is about—rather it is about how we are implementing it incorrectly and misusing its results.

I’ve posted on here before that I think we as a society need to carefully distinguish between predictive ability, causal inference, and ethical policies.

You can predict things without understanding the cause. And you can determine the cause of something without being able to make future predictions about it. But neither causation nor prediction are sufficient to inform an ethical policy.

A simple example might be a genetic abnormality that causes disease. Suppose we can use someone’s DNA to predict early in life and with high accuracy whether they are likely to eventually develop a disease that entails exorbitant medical costs. And suppose we can even identify the combination of genes that fully determines the outcome of whether a person will develop the disease or not. Health insurance companies might decide it’s not in their financial interest to insure these people, at least not without significantly raising the affected customers’ premiums.

But as a matter of morality, society decides that because this disease is not the fault of any individual or within any individual’s ability to control, insurance (and subsequently, all customers) should accept this additional cost as a necessary loss in order to ensure fairness.

To expound on the article’s other point, what is frequently claimed as “predictive” often is not actually so. As a data scientist, one of the things that I’ve noticed within my industry that is really glossed over is evaluating the accuracy of accuracy evaluation. Determining the quality of a model’s predictions is to some degree extremely difficult to do correctly. As a motivating example, consider a collection of models from different seismologists for earthquake prediction. Could you easily tell me which model is “the best” and to what extent each model is wrong?

Any ethical policy that includes predictive ability as one of its components should require intense scrutiny of the quality of the predictions. Not to mention that policies that depend upon predictive ability should already be extremely rare—I can think of perhaps one example: “what value of n to use for Blackstone’s ratio?” In this case, the true value of n (as opposed to our desired value for n) depends on how accurately we can predict whether someone is legally guilty or innocent. And even in that case, we never have ground truth data, so our ability to determine precision/recall is fundamentally limited.

This presentation would be more effective if the key information were explained more clearly and not hidden in dropdowns and modals.

You want to make loans without predictive optimization? Good luck with that. This is truly one of the most childish criticisms I've ever seen published.

Anyone who works on forecasting or "predictive optimization" understands all of these issues. The question isn't "is predictive optimization perfect", the question is "Is predictive optimization better or worse than your practical alternatives?" And sometimes the answer to that question is yes, and sometimes the answer is no.

First, love the presentation. I wish more papers had such a nice, succinct presentation of the main arguments.

Second, a disclaimer, I did not read the full paper but only went through the link here.

The site should be more specific that it refers to the use of predictive optimization in social setups. I thought I was about to read a finding in how predictive optimization is fundamentally flawed, but that is far what this presents.

Even within the social scope, the argued flaws seem to be more like a list of things to check against, rather than fundamental flaws:

- Good predictions may not lead to good decisions: In a setup where the prediction model is trained independently than the optimization one, this should be obvious. How you craft the optimization model and what you feed it is what matters.

- It's hard to measure what we truly care about: This is true in way too many cases, and still we have to do something. We constantly seek proxies to be able to act instead of raising our hands and give up.

- The training data rarely matches the deployment setting: A well known issue in ML, not really insightful and just something that should be taken into account when developing models.

- Social outcomes aren’t accurately predictable, with or without machine learning: This way too broad a claim. The variance from person to person may make it hard to predict a specific outcome, but it may be possible to provide a model that gives a distribution over outcomes in an accurate way. For example, if you are born in income percentile p, it is hard to predict _exactly_ the income percentile your kids will be born into, but it is feasible to predict a _distribution_ over p. Moreover, if social outcomes were truly random, that is we can't say anything about the future, then there is zero signal between our interventions and the outcomes down the line... which again sort of implies we should never do anything.

- Disparate performance between groups can’t be fixed by algorithmic interventions: Again, not a fundamental flaw. There's nothing an algorithmic intervention that makes it inherently biased, rather it will simply reflect the design of the creator. This is a known issue and there is plenty of literature on the matter.

- Providing adequate contestability undercuts putative efficiency benefits: This is probably the best argument of the list. The design of predictive optimization models should take into account that they may need to be explained on individual cases, which probably reduces the model space. Or we should get better at explaining complex models!

- Predictive optimization doesn't account for strategic behavior: Again, this is just something to take care when designing these systems, not a fundamental flaw. This afflicts human interventions as well, as the examples show.

I worry that people in public policy read papers / summaries like this, take it as absolute truths, and then sentence our systems to be fully manual for the foreseeable future.

Very annoying to read papers like this. It’s easy to write a paper pointing out that things are wrong. It’s also unhelpful. Do i care about your opinions that this is a bad idea, if predictive optimization is still the best idea we’ve got?

It’s much more useful to create a better alternative and show people how to improve.