Hacker News Reader

Hacker News

Comment by angst | Hacker News Reader

Parent Full thread

angst

“I don’t think Sam is the guy who should have the finger on the button for AGI.”

- Ilya Sutskever, Co-founder, co-lead of Superalignment Team , Departed early 2024

- May 15, 2025, The Atlantic

Anyway, I concur it's a hard choice as one other comment mentions.

View on HN

Is this a button any one person should have their finger on?

They don't make buttons large enough for multiple people to press. It's always going to be someone in the end.

Yes they do?

There's also plenty of buttons that can't be pressed unless unlocked by multiple keys which cannot be turned by a single person.

Exactly. AGI is something that will significantly affect all of humanity. It should be treated like nuclear weapons.

Effectively kept secret and in the shadows by those working on it, until a world-altering public display makes it a hot politically charged issue, unaltering even 80 years later?

Edit: Honestly, I bet that "Altman", directed by Nolan's simulacrum and starring a de-aged Cillian Murphy (with or without his consent), will in fact deservedly win a few oscars in 2069.

International co-operation to control development and usage. The decision to unleash AGI can only be made once. Making such a decision should not be done hastily, and definitely not for the reason of pursuing private profit. Such a decision needs input from all of humanity.

> International co-operation to control development and usage.

Non-starter. Why would you trust your adversary to "stay within the lanes". The rational thing to do is to extend your lead as much as possible to be at the top of the pyramid. The arms race is on.

With these things, the distrust is a feature, not a flaw. The distrust ensures you are keeping a close eye on each other. The collaboration means you're also physically close and intellectually close. Secrets are still possible, but it's just harder to keep them because it's easier for things to slip

It's rational only if you don't consider the risks of an actual superhuman AGI. Geopolitical issues won't matter if such a thing is released without making sure it can be controlled. These competition based systems of ours will be the death of us. Nothing can be done in a wise manner when competiton forces our hand.

And quickly proliferated around the world to other superpowers and rogue states…

Remember the soviets got the nuke so quick because they just exfiltrated the US plans

> It should be treated like nuclear weapons.

Seeing how currently nuclear weapon holders are elected, that would be a disaster

The disaster will happen if AGI is created and let loose too quickly, without considering all the effects it will have. That's less likely to happen when the number of people with that power is limited.

>It should be treated like nuclear weapons.

Either you get it or you're screwed?

It's already being treated like nuclear and that's a problem

  - highly centralized
  - lots of misinformation
    - lots of fear mongerng
  - arms race between most powerful countries
    - who can't stop because if the other gets a significant lead it could be used to destroy the other
  - potentially world changing
    - potential to cause unprecedented levels of harm
    - potential to cause unprecedented levels of prosperity

Sometimes things are just done better with your enemy than in direct competition with them. "Keep your enemies closer" kinda thing.

As a parallel, look at medicine and gain of function research. It has a lot of benefits but can walk the line of bioweapons development. A mistake causes a global event. So its best to work together. Everyone benefits from any progress by anyone. Everyone is harmed by mistakes by any one actor. That's regardless of working together or not. But working together means you keep an eye on one another, helping prevent mistakes. Often ones that are small and subtle. The adversarial nature is (or can be) beneficial in this case

Regardless of who invents AGI, it affects the entire world.

Regardless of who invents AGI, you can't put the genie back on the bottle (or rather it's a great struggle to that's extremely costly, if even possible)

Regardless of who invents AGI, the other will have it in a matter of months

Good point. I like George Hotz’ philosophy on this - paraphrasing badly - if everyone has AI, nobody can be subjugated by AI.

That's all fine and dandy if we skip the development phase and AI is already invented.

But this doesn't work during the transition. During the development. "The button" here is for AGI. As in, when it's created and released.

its a button everyone should have finger on. if you can get it to do something good, go for it?

I want Sam to win more than I do Zuck, just based on the proven negativity generated by Meta. I don't want that individual or that company anywhere near additional capital, power, capability or influence.

The hypocrite who violates everyone else’s privacy to sell ads, or the scammer who collects eyeballs in exchange for cryptocurrency and whose “product” has been banned in multiple countries…

Yeah, there’s no good choice here. You should be rooting for neither. Best case scenario is they destroy each other with as little collateral damage as possible.

Between the trio of Thiel, Zuck and sama I’d pick the fourth option - I don’t want to be on that train anymore

It became clear in 2024 and 2025 that they're all dangerous.

All these tech billionaires or pseudo billionaires are basically believing that an enlightened dictatorship is the best form of governance. And of course they ought to be the dictator or part of the board.

Fourth option is Musk with xAI.

Fourth option is butlerian jihad

At the start of the “LLM boom” I was optimistic that OAI/Anthropic were in a position finally unseat the Big 4 in at least this area. Now I’m convinced the only winners are going to be Google, Meta, Amazon, and we are right back to where we started.

What makes you think so? They got the Chatgpt.com domain and the product seems to be growing more than any other (check out app downloads: https://appmagic.rocks/top-charts/apps). They got the first mover advantage - and as we know around here that's a huuuge advantage.

> They got the Chatgpt.com domain and the product seems to be growing more than any other

And still haemorrhaging money.

I still have hope that Anthropic will win out over OpenAI.

But… Why put Meta in that group?

I see Apple, Google, Microsoft, and Amazon as all effectively having operating systems. Meta has none and has failed to build one for cryptocurrency (Libra / Deis) and metaverse.

Also, both Altman and Zuck leave a lot to be desired. Maybe not as much as Musk, but they both seem to be spineless against government coercion and neither gives me a sense that they are responsible stewards of the upside or downside risks of AGI. They both just seem like they are full throttle no matter the consequences.

Yes, I’ve never seen a more sociopathic company than Meta. They are so true to the cliched ethos of “you are the product.” Sickens me that society has facilitated the rise of such banality of evil

> Sickens me that society has facilitated the rise of such banality of evil

American society. Those are uniquely products of the US, exported everywhere, and rightfully starting to get push back. Unfortunately later than what it should’ve happened.

Does any of those OpenAI spinoffs have something to show already or are they still "raising money"

Reply on news.ycombinator.com