I feel anything that we develop for AI would fundamentally always be inspired by our own experiences and hence curve fitting is something we understand to be the best metric to optimize for.
No, the book actually makes an important point about this.
For instance, let's take a counterfactual. How do you know what would have happened if Barcelona had played Lionel Messi in goal over the latest season?
They've never done this, there are no data points for you to fit. The situation almost never arises that an outfield player plays in goal, and when they do it is always in a situation where someone's been sent off or injured, which is also rare.
And yet you and I and everyone else who can think knows this would result in Barcelona having a much worse season.
Just to be clear, it's not only because you'd be taking a top player out of offense where he is worth a lot, which you can surely fit some curves to show.
We can all guess that he'll be a worse keeper than Ter Stegen, but what curve would you fit that shows this? There's no data about Messi in goal.
Pearl does give a way to work it out via counterfactual analysis though.
The fact is that everything in the world can be reduced down to curves. It's just a matter of your perspective.
That's kinda the problem here. Once you have a model fitting curves is fine. But you need a structure from somewhere.
> players require practice to excel in their position
The problem with this is there's no data. Nobody gets to play a position they haven't practiced for. Yet you still somehow came to the right conclusion.
Anyway Pearl is much better at explaining this than I am.
I don’t even think educators mistake that for intelligence. It’s, at best, a proxy for intelligence mixed with other factors.
We have been trying to manage governments, public services, and education the same as corporations, creating numerical targets for institutions to optimize for. Education itself was formulated as an optimization problem about how to create more jobs. Public services like healthcare were privatized and became a target for profit optimization. Half of the stock market is controlled by High-Frequency Trading supercomputers, which will do virtually anything to gain an upper-hand in profits. Those methods were all inherited from the management styles of corporations that began in the neoliberal era. As fundamental parts of our society are replaced by those systems, the society now curve-fits the systems rather than the systems curve-fitting the society. We now hyperoptimize ourselves to fit in this neoliberal landscape; our time is told as something to be optimized between work, socializing, exercising, and self-improving, with no space for "actual free time" of our own. We go to college not to learn but to pass exams and get ourselves a good job that can sustain us. And the faults of our systems are now blamed to be individual problems: "You didn't optimize towards the current trends of the job market, it's your fault." And from the view of the corporations, we are just AI agents waiting to be optimized for cash, and we're now becoming one through fitting our bodies and minds to the social media that tries to maximize engagement and ad revenue no matter the real societal cost.
Now, the real problem of AI compared to the algorithms of the past is its data-driven nature: it can only learn from what data you give it for training. We can only accumulate data from the past and never from the future, so the AI systems will just keep repeating the past, no matter what unseen change will come. We will lose the ability to imagine new political, economic alternatives, we will just be feeding ourselves the status quo, and societal advancement will stagnate at the hands of automated systems. The cancellation of the future: this is what I'm ultimately afraid of.
that said I think if you made its context recurrent in some way between responses and queries you could probably build a really interesting chatbot that could ramble about just about anything and nothing but do so in a sufficiently coherent way to scare up venture capital. Just a guess but it is my guess.