Ask HN: How did we go from wanting AGI to asking a 6 month pause on it? | Hacker News Reader

Ask HN: How did we go from wanting AGI to asking a 6 month pause on it?

14 pointsbehnamoh26 comments

I know the letter might be fake, but the overall tone seems to agree with the content of the letter. How did we go from desperately looking for a way to build AGI to a point that we plead everyone to just stop/pause achieving AGI level models? I understand the concerns written in the letter, but isn't the genie out of the bottle? How can anyone be sure that governments won't do it behind the scenes? Isn't AGI a first-mover advantage?

View on HN

"We" might not have changed our ask at all. "We" might be referring to two divergent groups with minimal defectors.

Many people have been urging caution for a long time. As we get closer to more features and capabilities those people are going to ride on the publicity of those achievements to call out their warnings. This will garner more interest because the threat seems closer or more real as the capabilities increase (or at least appear to be so due to company hype).

Then there are many others who are excited and still pressing ahead.

AGI isn't even imminent it's just the Twitter AI hype cycle has gone out of hand and is spilling over and producing this kind of silliness.

I don't see it as hype to be honest. It's a real problem we need to solve now.

Also, it seems like the letter was faked?

Anyways, it doesn't matter if it's faked. It does make people notice that it's a real problem now.

General consensus amongst data scientists of when each AI breakthrough will happen have almost always been wrong. AI breakthroughs tend to happen earlier than what the consensus predict. I don't see AGI predictions as any different.

> It's a real problem we need to solve now.

We need to solve AGI now? Don't get me wrong, it would be an incredible scientific breakthrough, but I cannot see how AGI would solve our current societal and human problems. In fact, it would just upturn our entire world.

AGI is a tool we want, not the solution we need.

And I agree with OP anyway. It is pure hype. Thinking we are closer to AGI because of LLM is just like thinking we are closer to the Moon because we have conquered the Everest peak.

>We need to solve AGI now?

No, we need to solve the problem of AI alignment, regulation, etc now.

https://arxiv.org/pdf/2209.00626.pdf

I believe the idea of a pause is ridiculous right now. AGI is the biggest thing since the iPhone in 2007 and Google in 98… anyone thinking about pausing is asking to be left in the dust . It’s go time like never before.

Think back in history to the interaction of the natives of the America with other humans, from Europe, with only slightly better technology.

AGI would be make us the natives, only the disparity will grow geometrically.

There is good reason to worry.

I've been saying it's more like the Watt steam engine. Which led to physical labor being cheap. Leading to railroads, cars, tanks, planes, drive bombers, battleships, tractors, the end of subsistence farming, people being forced to migrate to cities, child labor, 80-100 hour weeks, and so on. Maybe indirectly leading to things like the violent rise of communism, Cold War, and such.

AI will lead to attention being cheap. AGI will be a wrecking ball to the education system. And it will mess with the creative industry who are suddenly realizing they were not that creative in the first place.

I don't expect things to be as rosy as when Google and iPhone came in.

AGI isn’t a thing yet.

I haven't seen many people change position. The people who want to build AGI and the people who want to prevent AGI are different.

Actually, it seems like the people who are most worried about AGI are the ones building it, because they don't want to see it "raised" the wrong way.

Here's a blog by Sam Altman, about 10 months before OpenAI was founded: https://blog.samaltman.com/the-software-revolution

"Two of the biggest risks I see emerging from the software revolution—AI and synthetic biology—may put tremendous capability to cause harm in the hands of small groups, or even individuals. It is probably already possible to design and produce a terrible disease in a small lab; development of an AI that could end human life may only require a few hundred people in an office building anywhere in the world, with no equipment other than laptops.

...

I think the best strategy is to try to legislate sensible safeguards but work very hard to make sure the edge we get from technology on the good side is stronger than the edge that bad actors get. If we can synthesize new diseases, maybe we can synthesize vaccines. If we can make a bad AI, maybe we can make a good AI that stops the bad one."

I don't really see how AI itself will end human life. It would have to interact with something else to achieve that.

You just answered it :p

Microsoft trained AI on the internet and it became a racist asshole. OpenAI released ChatGPT and people had fun jailbreaking it to get it to do evil things.

Someone could just put AI on a weaponless drone and it could start slashing kids at a playground, deliberately or not. I think what Sam Altman wanted to do was to make sure that if someone created an AI library that was as popular as MySQL, it wouldn't easily develop slasher drone behavior.

Ending a human life is a little different than ending human life. Especially given the implications posed by his inclusion of mentioning pathogens created in a lab.

Even if you have a "slasher drone", the threat itself is not really any different from a manually controlled drone where the operator has similar intentions.

Asking for a 6 months pause "to protect x" is just the excuse to further the interests of certain individuals.

If something may cause a hike of my chance of losing job I don't really care how rosy it sounds. Yes I use it very often but I prefer it to stay where it is. By the same principle I wish graphics tech to be fixed approximately in the level of late 90s. Well but I don't expect to get what I want.

So this is basically different interest groups competing with each other. Usually the more technically advanced ones won towards the end but the more important thing is how long the other groups can drag it down.

Under the current social-economic world order, any large scale tech advance such as AGI will almost surely bring destruction to a whole generation of workers.

You think AGI, nuclear fusion and such will bring prosperity to us? Too naive, mon ami.

These thoughts of a unified “we” are the first sign of self deception. We must all get along, yet there will not be found a universal consensus among humanity.

Keep on going man, the future shall arrive, and not everyone will be ready.

We must continue this debate of our human competence, relevance of public narratives, and true potential of outcomes. All the meanwhile the future presses upon us.

Humans are still by far the greatest threat to our humanity.

"Humans are still by far the greatest threat to our humanity."

This, but replace "humans" with "intelligence". One of the risks is augmented humans as well. A human with a chainsaw is more dangerous to other humans, even when that chainsaw is a tool.

There's really no research done on what humans can actually do with AGI. Part of the fears are similar to industrial age.

I think it's good though. We got through COVID from the lessons of past pandemics and we got through 2008 and 2023 from the lessons of past recessions and bubbles. We've already planned for AI not to be Skynet and Asimov has run several scenarios.

I am shocked and mortified that you say “ We got through COVID from the lessons of past pandemics …”

We collectively stumbled through incompetence, hysteria, delusion (on every side), and catastrophic failure in ways obscured by our ability to bs over the details.

Which resulted in humanity's best response to a pandemic thus far.

You do mean the pharmacy companies, most ingenious taking of profits…

And authoritarians running roughshod over us all…

regardless of effectiveness and outright harm by their ‘solutions’.

Some authoritarians wrote a blog post.

AGI (any reasonable definition of it) is not happening in our lifetime.

We have a few very specialised "AI"s and still struggle since 30 years and trillions of USD investment to even just let an AI drive a car (in all conditions not just when everything is easy).

The step from ChatGPT to an real AGI is much bigger then the step from going to the moon to colonising other planets that are lightyears away.

Both AGI and colonising other planets lightyears away will not happen in our lifetime.

One word - hype !

That being said, once we really reach the AGI level - I suppose it will be like this - very sudden.

Anyone with a bit of prescience would know well to follow the butlerian doctrine.

In any way I think overhype is playing a role here as well. There is a lot of money to be made here, so motivations are high, tends to make people polarize their opinions.

Who is "we"? There are thousands of nuanced opinions on what AGI is and what it should be.

So someone wrote a letter. That doesn't mean anything. It doesn't even mean that the average opinion has substantially changed.

There is no consensus on anything here. Beware of so-called "experts" on this subject. There are no "AGI experts". There are AI experts, but speculation by humans on what a hypothetical superhuman intelligence might be like will never be more than, well, speculation. It's like ants trying to wrap their minds around what they imagine human philosophy to be.

Your opinion on AGI is as good as mine, or as Elon Musk's, or as Eliezer Yudkowsky's. Nobody knows anything. Get comfortable with uncertainty, and don't jump to the conclusion that anything has happened just because somebody somewhere said something.

Reply on news.ycombinator.com