AI has hacked the operating system of human civilisation

I mean, maybe. Sure, LLMs can currently code shockingly well, and LLMs are just programs, so an LLM writing an LLM is def much more likely than a nuke building a nuke. But we don't _know_ if there are limits to what AIs can do. Or, to pick on the word "exponential," we don't know if there are limits to growth rates of AIs improving AIs!

FWIW, I personally think that AIs will be able to invent more powerful/intelligent AIs. But we should be stating this as as conjecture, not a fact - mostly b/c my experience in saying things like this to people just causes them to be _extremely scared_. And for most people (myself included) extreme fear is a really bad place to try to make decisions from.

> We need an equivalent of the Food and Drug Administration for new technology, and we need it yesterday.

I wish there was more detail on this. Does Yuval Noah Harari really understand how the FDA works and is it really an appropriate model for the AI Safety Administration (AISA)? I mean - the FDA is not really batting 100% - and can we afford to miss swings when it comes to AI safety?

Maybe we can. Maybe we can't. Anyone saying we can afford to miss swings is probably selling you something (or their livelihood depends on it). Anyone saying we can't afford to miss swings is probably terrified. I honestly don't know how to possibly feel other than ???!!??

So, for example, in his book _Sapiens_ he got a warm reception to a "zinger": Humans did not tame wheat - wheat tamed humans! It's essentially a "punch line" to a long, analytical story he develops. There are several other such punch lines in that book.

So, when he concludes AI has done anything, keep in mind that he thinks that because we are (mostly) addicted to carbohydrates and this is not great for our lifestyle that a plant with no ability to cognate has "tamed" our species.

A funnier example of this is a Usenix Keynote delivered by Van Jacobson in - I think 1996. He tells an involved story about language acquisition in humans and Unix and networks and ends with a "conclusion" that using Windows makes you literally brain damaged. I wish I could find an online video of that talk, if anyone knows of one. (I'm not sure it was even recorded.)

Where was this attitude when social media was taking over the internet? We just stood back while it dramatically reordered society. But now we're expected to wring our hands about entirely hypothetical and likely imaginary risks related to AI because they echo of dystopian sci-fi themes? Seriously?

I've yet to see any AI catastrophist address the fact that the training set for current state of the art LLMs is already effectively "the whole internet". It's difficult to see how an LLM which sets out to improve itself overcomes this limitation. We haven't even seen the early stages of models able to reason or develop and retain new knowledge independently. Maybe at that point I would agree it's time to apply the brakes, but at the moment regulation would be tantamount to strangling AI in the crib.

> Yuval Noah Harari argues that AI has hacked the operating system of human civilisation. Storytelling computers will change the course of human history, says the historian and philosopher

For a moment there, I was worried The Economist went off the rails.

A point of note: Hinduism (specifically Sanatana Dharma) does not only talk about Maya (which for some reason the Western scholars who study Hinduism are fascinated with). This is just one branch of Vedanta (specifically Advaita Vedanta/Monism). However, two other branches do not agree with the concept of the Creation being an Illusion/Maya (Vishishtadvaita/Qualified Monism as well as Dvaita/Tattvavada/Dualism). So can't really generalize and say that "Hindu sages pointed out that all humans lived trapped inside Maya-the world of illusions". An equal number (if not more) opposed the idea.

Is not the most major problem with maintaining a healthy society keeping the power of governments and massive corporations in check? How are closed source models supposed to be the solution? I simply don't see it, so to me it reads as propaganda from those powerful entities. Leaving the most powerful software to be only understood, researched, and used by the most dangerous groups is likely the worst thing you can do.

I am saying that just as an hypotheses: I don't think we can control the daemon or alien as Harari suggest. The AI technology is different from nuclear technology, "everybody" has the technology to run it. We don't need enriched uranium.

All this challenge anthropocentrism. At the end is the complex reality who will shape the next iteration of the world.

From a more HN social perspective, I think it is important that everyone could potentially create and access this technology based on open source software and computing power.

AI hacked nothing. We've been doing it to ourselves for ages. This is just another way of doing it, from ourselves, to ourselves. We hacked every single kind of language there is. The AI just trained on that previous work.

This could have been a nice piece about how we need to be less inclined to believe in stories and be more skeptical, but I guess no storyteller wants to write a story about _that_. Instead, they blame it on the machines.

Can it?

"Storytelling AIs", i.e. LLMs, are not autonomous. Their goal is to predict text. They predict text. They are predictors.

So, in the article, we casually drop the claim: "AI can make exponentially more powerful AI." Is it feasible for a predictor to run in a loop and exponentially improve, by itself? No. Is it potentially feasible for a predictor to run in a loop and generate outputs that may be used by a human to further improve its own outputs, or Rube Goldberg'd together by a human to steadily improve predicted outputs in general or for niche? Yes. But exponentially, and realistically, no. Hardware constraints, architectural constraints, performance constraints, monetary constraints, correct?

Let's go back to fear, for a second. Nukes cannot invent more powerful nukes, but a nuke can cause a "Buck" Turgidson can nudge an organization or machinery in the direction for more powerful nukes. LLMs can potentially produce the output to feed into more powerful "AI" in that way, with a "Buck" nudging it. But that's with the assumption that the predictor's outputs don't reduce performance. Can the predictor spit out a novel description of some architecture that may be used by a human that is "exponentially" more powerful? Unlikely.

After re-reading the article, perhaps the joke is on me, because I detect some vague satirical wit. But when someone wants a government, the elected congressional or federal body in which elected individuals are knocking on dementia, catatonically voting on party lines, and are still figuring out email, to regulate for some vague idea of "safety" for some "AGI" entity they cannot even define, I smell bullshit. Not to mention, regulatory capture. Maybe that's the real "death of democracy."

We've been through this over the past few years. Hysteria leads to regulation and government intervention, leads to large corporations to be able to plug their nose and survive via acquired capital, and/or use their resources to comply with the bureaucratic bullshit, while smaller businesses die, people lose their life savings, income inequality grows, a generation is set back for the rest of their life: we're worse off and outcomes are not improved as per the original goal of the regulation, in the end.

Capitalism is undoubtedly a sort of AI that only seeks to find equilibrium points in markets, and the idea is running on almost everybody’s brain. I hardly imagine that any sort of AI could be more dangerous than something that currently let’s millions die due to malnourishment because it’s trying to satisfy some loss function - equilibrium.

This is dumb and lacks basic education. No, ai cannot make an exponentially more powerful AI.

Humans, however can use AI as tool to create a more powerful AI. But that would not be different from humans crating first nukes with enriched uranium, and then using that uranium to create plutonium for even bigger nukes, and then figuring out that this plutonium could be used as a mere activator for even more devastating thermonukes.

Not sure I agree here. Who watches the watchers? Who sets the safety checks? Who gets to see behind the curtain? Do we trust our current governmental and regulatory structures?

Writing changed humanity.

Compared to this AI doesn’t even register on the scale.

No, they are based on instinctual responses (https://www.youtube.com/watch?v=pXyZ0kEwvqY ). Telling stories about them and writing laws, at best, codifies them (and often enough codifies exceptions to them).

> What gives money value is the stories that bankers, finance ministers and cryptocurrency gurus tell us about it. Sam Bankman-Fried, Elizabeth Holmes and Bernie Madoff were not particularly good at creating real value, but they were all extremely capable storytellers.

I grant him this.

Yes, language is important. But it is not the foundation of human civilization. It is merely an important pillar (especially for the subset of humanity, such as the author, who are very language-use prone).

> The catch is that it is utterly pointless for us to spend time trying to change the declared opinions of an AI bot, while the AI could hone its messages so precisely that it stands a good chance of influencing us.

I seriously doubt the AI will influence the person it is arguing with. This happens among humans occasionally, but not frequently. Maybe the AI will get very good at reverse psychology, though.

So why is it utterly pointless for us to get better at arguing a point by arguing with an AI that we will personally not convince, but the reverse is not also true for the AI? Bystanders are the audience. If a human's arguments are more convincing than the AIs, then most bystanders will be convinced by the human. It may take a bit longer for the human to reference the arguments of other humans, than the AI to cite whatever (if it can cite, as LLMs are currently not good at doing that), but an AI who argues too fast risks being seen as a bullshitter by the bystanders.

> In a political battle for minds and hearts, intimacy is the most efficient weapon, and AI has just gained the ability to mass-produce intimate relationships with millions of people.

Until people stop being intimate with other people, and are intimate solely with AI, there will be a countervailing force. Intimacy is also more compelling when coupled with in-person interaction.

> People may come to use a single AI adviser as a one-stop, all-knowing oracle.

This is a risk. In-person interaction is a countering force. Few people would trust an AI, or even an expert, over the feedback of those they know. Because the feedback of those who know us is more likely to take into account our personal experience. Maybe intimate AIs would be a problem, here.

> Previous tools like the printing press and radio helped spread the cultural ideas of humans, but they never created new cultural ideas of their own. AI is fundamentally different. AI can create completely new ideas, completely new culture.

They never created new cultural words of their own, but they surely created new ideas. Just as humans get new ideas from the lives and interactions of other animals and plants. Do the ideas and culture of AI matter if humans don't adapt them to our purposes?

> And the first regulation I would suggest is to make it mandatory for AI to disclose that it is an AI.

This is a very neat idea. Of course bad actors won't care. But in a future where AI is generating new AI maybe this is something that would be worthwhile.