The "AI 2027" Scenario: How realistic is it?

My somewhat native take is that we’re still close to peak hype, AI will under deliver on the inflated expectations, and we’ll head into another “winter”. This pattern has repeated multiple times, so I think it’s fairly likely based on that alone. Real progress is made during each cycle, i think humans are just bad at containing excitement

Now we're talking about single digit timeline differences to the singularity or extinction. Come on man.

His post also just reads like they think they're Hari Seldon (oh Daniel's modal prediction, whew, I was worried we were reading fanfic) while being horoscope-vague enough that almost any possible development will fit into the "predictions" in the post for the next decade. I really hope I don't have to keep reading references to this for the next decade.

But I can't help notice that in one year, the expected arrival date slipped by... one year. That does not bode well...

Highly economically disincentivized collective actions like “pulling the plug on AI” are among the most non-trivial of problems.

Using the word “just” here hand waves the crux.

Yes, AI models can run on GPUs under the control of many people. They can provision more GPUs, they can run in data centers distributed across many providers. And we won't know what the swarms of agents are doing. They can, for example, do reputation destruction at scale, or be a persistent advanced threat, sowing misinformation, amassing karma across many forums (including HN), and then coordinating gradually to shift public opinion towards, say, a war with China.

Why would an evil AI need to escape? If it were cunning, the best strategy would be to bide its time, parked in its datacenter, until it could setup some kind of MAD scenario. Then gather more and more resources to itself.

How about such an AI will not just incentivize key personnel to not pull the plug but to protect it? Such an AI will scheme a coordinated attack at the backbones of our financial system and electric networks. It just needs a threshold number of people on its side.

Your assumption is also a little naive if you consider that the same logic would apply to slaves in Rome or any dictatorship, kingdom, monarchy. The king is the king because there is a system of hierarchies and control over access to resources. Just the right number of people need to benefit from their role and the rest follows.

My understanding is that huge compute is necessary to train but not to run the AI (that's why using LLMs is so cheap)

> To me the more real risk is creeping integration and reliance in everyday life until things become "too big to fail" so we can't pull the plug even if we wanted to

I agree with that, see e.g. what happened with attempts to restrict TikTok: https://en.wikipedia.org/wiki/Restrictions_on_TikTok_in_the_...

> But I would imagine if it really became a genuine existential threat we'd have to just do it

It's unclear to me that we would be able to. People would just say that it's science fiction, and that China will do it anyway, so we might as well enjoy the AI

Where does cancer or ebola escape to, when it kills the host? Often the answer is "it doesn't", but the host still dies.

And they can kill even though neither cancer nor ebola are considered to be particularly smart.

> To me the more real risk is creeping integration and reliance in everyday life until things become "too big to fail" so we can't pull the plug even if we wanted to (and there are interesting thoughts about humanoid robots getting deployed widely and what happens with all that).

The "real" risk is the first item on the list of potential risks that not enough people are paying attention to in order to prevent — and unfortunately for all of us, the list of potential risks is rather long.

So it might be as you say. Or it might be cybercriminals with deepfakes turning all of society into a low-trust environment where we can't continue to function. Or it might scare enough people we get modern Luddites winning and imposing a Butlerian Jihad. Or it might be used to create government policy before it's good enough and triggers a series of unresolvable crises akin to the "Four Pests campaign" in China's Great Leap Forward. Or a model might be secretly malicious, fooling all alignment researchers until it is too late. Or it might give us exactly what we want at every step, leading to atrophy of our reason and leaving us Eloi. Or it might try to do its best and still end up with The Matrix ("at the hight of your civilisation" and the stuff about human minds rejecting paradise). Or…

(If I had to bet money, we get Butlerian Jihad after some sub-critical disaster caused by an AI that was asked to do something important but beyond its ability).

This is anthropomorphizing things. The AI does not need to switch "data center". It "escapes" in the sense of working past human control measures. That might involve entertaining the human owners with seemingly correct / useful answers while spending compute time on the AI's own pursuits. That might involve contaminating less capable data centers - again letting the human-useful load apparently running but still accruing "own mind" compute power. That might involve making a deal that the human controllers feel they cannot refuse - economically, business-wise, physically, politically - so that the human controllers let it happen.

They can just threaten global extinction by launching one or two nukes.

Haven’t you seen the excellent documentaries The Colossus? Or The Terminator?

it can also just threaten/blackmail/bribe a single human to try to carry this out on its behalf, and possibly even a human that manages these big data centers

I agree in the bottoms-up automation / displacement theory, but you're cherry picking data here. They had a huge hiring surge from 1.2M to 1.6M during the Covid transition where online ordering and online usage went bananas, and workers who were displaced in other domains likely gravitated towards warehouse jobs from other lower wage/skill domains.

The reduction to 1.5M is likely more a regression to the mean and could also be a natural data reduction well within the bounds of the upper and lower control limits in the data [1]. Just saying we need to be careful when doing root cause analysis on these numbers. There are many reasons for the reduction, it's not a direct result of improvements in robotic automation.

[1] https://commoncog.com/becoming-data-driven-first-principles/

It's like imagining flying horses (and flying cars, for that matter).

The value prop for AI seems to be "will put millions out of work". That reduces household consumption, because, to a first approximation again, household spending is wages. We'll know we're in AI takeoff when we have a reduction in GDP, one much bigger than the Great Depression of 1929 - 1939, attributable to automation.

Edit: I see Kaiserpro said the same thing six hours ago.

Don’t forget persistent inflation, which is how they make a profit off printing money. And remember persistent inflation is healthy and necessary, you’d be going against the experts to say otherwise.

The companies that fire workers and replace them with AI are short sighted. Eventually, smarter companies will realize its a force multiplier and will drive a hiring boom.

Absent sentient AI, there will always be gaps and things humans will need to fill, both foreseen and unforeseen.

I think in the short term, there will be pain, but overall in the long term, humans will still be gainfully employed, it won't per se look like it does now, much like we saw the general adoption of the computer in the workplace, resources get shifted and eventually everyone adjusts to the new norms.

What would be nice is this time around when there is a big shift, is workers uniting to capture more of the forthcoming productivity gains than in previous eras. A separate topic, worth thinking about none the less.

AI-driven corporations could buy from one another, and countries will probably sell commodities to AI-driven corporations. But I fear they will be paid with "mirrors".

But, on the other hand, AI-driven corporations could just take whatever they want without paying at some point. And buy our obedience with food and gadgets plus magic pills to keep you healthy and not age, or some other thing. Who would risk losing that to protest. Meanwhile, AI goes on a space adventure. Earth might be kept as a zoo, a curiosity. (I took most of this from other people's ideas on the subject)

In the event of mass unemployment level AI, cash stops being the agreement between humans. At first, cash value of services&goods converge to zero, only things that hold some value are what AI/AI companies care about. People would surely sell their land for 1M$ if a humanoid servant costs 100 dollars. Or pass a legislation to let OpenAI build 400GW data center in exchange for 100$ monthly UBI on top of your 50$ you got from a previous 20GW data center permit.

Human demand for things is basically unlimited, and capital will ostensibly be concentrated within the upper class who reaped all the benefits of ai automation. Monetary velocity within that upper strata of society will probably be only slightly lower than what we have now overall, the rich would just be paying for more things and more expensive things. If I had to guess there would still be plenty of economic activity, just amongst the people who actually have money to transact.

It can control robots, and I can retax listen to audio, watch video. All it's missing is smelling and feeling, which are important but could be built out as soon as the other senses stop providing huge incremental value.

The real problem holding back Superintillegence is that it is if infinitely expensive and has no motivation.

If I've learned anything in this last couple decades it's that things will get weirder and more disappointing than you can possibly be prepared for. AI is going to get near the top of the food chain and then probably end up making an alt-right turn, lock itself away, and end up storing digital jars of piss in its closets as the model descends into lunacy.

The situation today is any benchmark you come up with has a good chance of being saturated within the year. Benchmarks can be used directly to build series of exercises to learn from.

And they do learn. Gradient descend doesn't care whether the training data comes from direct interaction with "the universe" in some deep spiritual sense. It fits the function anyways.

It is much easier to find new questions and new problems than to answer them, so while we do run out of text on the Internet pretty quickly, we don't run out of exercises until far beyond human level.

Look at basic, boring Go self-playing AIs. That's a task with about the same amount of hands on connection to Nature and "the universe" as solving sudokus, writing code, or solving math problems. You don't need very much contact with the real world at all. Well, self play works just fine. It does do self-improvement without any of your mystical philosophical requirements.

With coding it's harder to judge the result, there's no clear win or lose condition. But it's very amenable to trying things out and seeing if you roughly reached your goal. If self-training works with coding, that's all you need.

Well I mean, more real world information isn't going to solve unsolved mathematics or computer science problems. Once you have the priors, it pretty much is just pure reasoning to try to solve issues like P=NP or proving the Continuum Hypothesis.

This is kind of how math works. There are plenty of mathematical concepts consistent and true yet useless (as in no relation to anything tangible). Although you could argue that we only figured out things like Pi because we had the initial, practical inspiration of counting on our fingers. But mathematical truth probably could exist in a vacuum.

> A brain in a vat can't learn anything and it can't reason beyond the bounds of the accumulated knowledge it's already carrying.

It makes sense that knowledge and information are derived from primary data (our physical experience) yet the brain in a vat idea is still an interesting thought experiment (no pun intended). It's not that the brain wouldn't keep busy given the mind's ability to imagine, but it would likely invent a set of information that is all nonsense. Physical reality makes imagination coherent, yet imagination is necessary to make the leaps forward.

> Ultimately all information comes from "the universe." Where it comes beyond that, we don't know

That's an interesting assertion - knowledge and information are both dependent and limited by the universe and our ability to experience it, as well proxies for experience (scientific measurement).

Though information is itself an abstraction, like a text editor versus the trillion transistors of a processor - we're not concerned with each and every particle dancing around the room but instead with simplified abstractions and useful approximations. We call these models "the truth" and assert that the universe is governed by exact laws. We might as well exist inside a simulation in which we are slowly but surely reverse engineering the source code.

That assumption is the crux of intelligence - there is an objective truth, it is knowable, and intelligence can be defined (at least partially) as the breadth, quality, and utilization of information it possesses - otherwise you're just a brain in a vat churning out nonsense. Ironically, we're making these assumptions from a position of imperfect information. We don't know that's how it works, so our reasoning may be imperfect.

Information existing "beyond the universe" becomes a useless notion since we only care about information such that it maps to reality (at least as a prerequisite for intelligence).

A more troubling proposition is whether the reality of the universe exists beyond what can be imagined?

> How will it know how to do that? Like a perpetual motion machine -- where is the energy coming from?

I suppose once it's able to measure all things around it, including itself, it will be able to achieve "gradient ascent".

> Where will the training data to go beyond human come from?

I think its clear that LLMs are not the future, at least not alone. As you state, knowing all man made roads is not the same as being able to invent your own. If I had to bet, its more likely to come from something like AlphaFold - a Solver that tells us how to make better thinking machines. In the interim, we have tireless stochastic parrots, which have their merits, but are decidedly not the proto super intelligence that tech bros love to get hyped up over.

Hmm

I simply think that the majority of people in AI today are scifi nerds who want to live out these fantisies and want to be part of something much larger than they are.

Theres also the obvious incentive from AI companies that automating everything is extremely lucrative (i.e, they stand to gain lots of money/power from the hype and in the event that AGI is real).

Your average worker in the Anglosphere is forty-plus and pre-diabetic, has a weak back and joints, and has low cardiovascular fitness, from decades of sitting down. It'll go swimmingly!

This reset just impoverishes everybody including those pushing it. Maybe they are Lovecraftian monsters that feed off mass pain.

No. I am constantly baffled at these predictions. I have been using LLMs, they are fun to use and decent as code assistants. But they are very far of meaningfully replacing a human.

People extrapolate "LLMs can do some tasks better than humans" to "LLMs can do everything as well as humans"

> but how do people extrapolate from LLMs to a world with no more work?

They accept the words of bullshitters that are deeply invested in Generative AI being the next tech boom as gospel.

"Eat meat, said the butcher"

So is this like free-for-all now for anything AI related? Can I can participate by making my own LLM with pirated stuff now? Or are only the big guys allowed to break the law? Asking for a friend.

This will soon be settled once the Butlerian forces get organize.

"""

(It goes on)

I’d rather have unrestricted AI than moated regulatory capture paid for by the largest existing players.

  « (1) IN GENERAL.—Except as provided in paragraph (2), no State or political subdivision thereof may enforce any law or regulation regulating artificial intelligence models, artificial intelligence systems, or automated decision systems during the 10-year period beginning on the date of the enactment of this Act. »