Hacker News Reader

Comment by matherial | Hacker News Reader

Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement.

This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.

View on HN

To borrow on the 1990s Slashdot meme:

1. Invent transformer architecture.

2. Scale it up.

3. ???

4. Machines become sentient and kill us all.

OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no.

But because we live in a culture of fear, everyone eats it up no questions asked.

Note that OpenAI has jettisoned every other supposed value they had (releasing their work as open source, not working on military applications, being a nonprofit). I'm sure we can rely on them this time.

And Anthropic are the good guys? They are talking insane stuff these days. May be we can trust zuck after all

None of them are the good guys. For one they're all happy to participate in genocide in Gaza.

Comments like these I'm willing to bet are because of the Reddit sub that's bringing Redditors with drop-mic comments in here. A ton of these things, usually but not always down voted. I think when they start hitting the top it's time to find threads without any possibility of Israel, Trump or Epstein possibly being brought up. I almost miss the old new-JS-framework complaint posts.

I would take that bet, anyone can examine my comment history and see that I've been participating on this site for a decade. Regardless, this comment violates the site guidelines.

> Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data.

I can promise you ranty political one-liners about Israel (your mic drop Reddity comment) violates guidelines about bringing politics unnecessarily into the picture. I'm not insinuating you're astroturfing or shilling; I would say you're unfortunately real.

"collect underpants... Profit" comes from south park

https://en.wikipedia.org/wiki/Gnomes_(South_Park)

Found it really funny to see this attributed to a Slashdot meme

It seems like the "Step 1, Step 2, …, Step 4" format likely predated the 1998 South Park episode, with use on Usenet for omitting dangerous or legally-questionable instructions or otherwise gatekeeping "trade secrets": https://arstechnica.com/civis/threads/where-does-the-step-st...

There's also a bit of prior art with the cartoon by Sidney Harris published in American Scientist in 1977 with a step two abridged as "Then A Miracle Occurs": http://www.sciencecartoonsplus.com/images/miracle_sharris.gi...

Interesting, thanks. Looks like also guilty of assuming the first place I saw it was the definitive source.

They literally have never promised anything like that. They have explicitly said repeatedly that they want regulation, oversight, nationalization, a global slowdown, etc. And then I have to read page after page of cynical comments about how that's all just hype and marketing and them wanting regulation to block their competition and blah blah blah.

You people would only be happy if they stopped all AI development, but you literally cannot do that in an arms race and survive. I cannot fathom what is so difficult to understand about this.

> they want regulation, oversight, nationalization, a global slowdown, etc.

> that's all just hype and marketing and them wanting regulation to block their competition

I don't see how these two opinions have to be mutually exclusive. Everyone knows that OpenAI and Anthropic cannot IPO in their current state or the economy's current state. It makes perfect sense that they would lobby for a global slowdown to hamstring China, regulation that defers to them as experts, oversight by their own "experts" and nationalization to backstop both of their protectionist needs. Even if you think they're doing it for benevolent reasons, you have to admit that regulatory remediation only consolidates AI power.

We see extremely similar protectionist consolidation in FAANG from Google, Apple, Microsoft and Meta, all of whom receive political shielding for their "free speech" whenever another nation calls their product a monopoly, identifies a federal backdoor or boots them out of the market. Countries like Russia and China feel entirely justified for closing the door on American businesses and refusing to negotiate when they abuse good faith to promote an American hegemony.

Everyone knows that OpenAI and Anthropic cannot IPO in their current state or the economy's current state.

What? Everyone doesn't know that. I predict one or both of them will IPO before the end of 2026.

How would they achieve ROI? The current consensus is that neither are in black financially and both have a diminishing moat.

The reason they haven't IPO'd yet seems to be that they can't assure investors that it's not a fad stock. With market share lost to China, a shrinking frontier and a money fire of GPUs burning in the background, something major has to change to convince investors to treat them like Tesla or Apple. I think both of them want the government to give them a subsidy of some sort to justify their economics.

Haha, I disagree with every single statement you made! Wild how we can see it so differently.

I think the ROI these big labs are seeing is incredible, I think their moats are increasing, I think their reasons for not IPOing yet have nothing to do with it being a "fad stock", I don't think they've lost any market share to China, I don't think their frontier is shrinking, I don't think their investments in GPU is a money fire, I think investors will rain money down on them when they IPO, and I don't think they're seeking a subsidy from the government.

I guess we'll see!

Those are perfectly fine beliefs, but again - how will they achieve ROI?

If their finances are "incredible" while they're both in the red, their mediocre quarters will be devastating for investors. They either have to find a new revenue stream (likely from the federal government) or reduce their spending significantly. IPOing under the current condition would be a treadmill that neither company can recover from, even if retail investors shower them with cash.

However, I will push back on three of your beliefs. Objectively speaking, market share has been lost to China, the frontier is more competitive than it was 3 years ago, and the GPUs that OpenAI and Anthropic buy are almost never at MSRP. I get that you're bullish on American AI, but the writing is on the wall and economists generally aren't echoing your sentiment for a reason.

> ... They have explicitly said repeatedly that they want regulation, oversight, nationalization, a global slowdown, etc.

And yet they do the exact opposite of all of the above. They do everything to get regulation just to create a moat because one doesn't exist. They preach about wanting a slowdown, nationalization, a complete pause, whatever have you... And yet they aren't slowing down the development voluntarily now are they? If anything, they are doing everything imaginable to speed up development to pump out models as fast as possible. If these ex-risk and AI companies actually gave a damn about regulation, or oversight, or a moratorium on AI development altogether, they would actually demonstrate this by completely ceasing development of all models immediately. Instead, they take riskier and riskier actions to try to "win" this supposed "arms race". So please excuse a bunch of us if we find it impossible to take them seriously on literally anything like this. If you have substantial evidence that they are actually ceasing AI model development like they want everybody else to do (except them, of course), then, by all means, present it.

It's genuinely hilarious to me that you immediately proved my second paragraph correct.

Putting arms race in quotes doesn't make it any less real. It just means you wish it wasn't, but don't have any good evidence or arguments to make.

I mean no, not really... Your the one making the claims that these AI companies really, really want regulation, nationalization, a slowdown, or whatever, and that they genuinely care about AI safety and it not killing us all. It's up to you to prove it, not for me to just believe you implicitly. So, please, provide the evidence, we're all very curious to see it.

I didn't claim that's what they really want, I claimed that they've repeatedly said that's what they want. Their words and actions are perfectly consistent with each other if they believe they're locked in an arms race.

All you've provided is a shallow dismissal of the idea that they're in an arms race, and then from there declared that their actions and words don't match. Which I agree with, if there is no arms race.

Everything hinges on whether they're in an arms race. I think they are obviously locked in an arms race, both with each other, and especially with China. You're free to continue dismissing it in this shallow way, I don't really care, but I just found it humorous that I talked about how people can't seem to understand the dynamics of an arms race, and then you immediately jumped to an explanation of their words and actions that ignores the idea of an arms race.

Their words and actions are also perfectly consistent with fearmongering to force legislation. OpenAI and Anthropic have spent millions on influencers and lobbyists to repeat their scare lines. It makes a lot of sense that the "arms race" framing is a bad-faith negotiation tactic, especially since we haven't seen any real-world LLM superweapons even from Russia or China.

So I don't think it's unfair at all to dismiss your theory of an arms race, especially if you can't explain the dangers. Nobody in China is confirming America's super scary AI research, probably because they don't need leverage against the government and global economy. Since we can't see this "safety" research, skepticism is warranted wholly.

Going a step further even, OpenAI and Anthropic can't actually bargain for safety or a research cutoff. The federal government can continue LLM weapons research on their own hardware and employees without any interruption, the commercial outcry is not capable of preventing LLMs from being weaponized. There is no way to convince the CCP to stop closed-door research either. It's much too late to do what OpenAI and Anthropic are suggesting, the only possible side effect of regulation is consolidation (which expressly benefits OpenAI and Anthropic as hegemons).

Wasn't that a South Park meme, or did they get it from /.?

Slashdot had Profit as (4), today that's item (2.5)

#3 is actually decently well mapped out. You just don't find it plausible or misunderstand it. I'd appreciate if you wrote your actual arguments against it.

At lower capability levels, the patterns are very clear and have been studied to death. E.g. Why LLMs say they have correctly fixed a broken test when they haven't. What we saw with Hugging Face is literally the exact same problem, just scaled up and with more capable agents. This shit was predicted decades ago...

No one can say exactly how it will play out as the complexity increases, but the risks are becoming extremely obvious.

I personally think it's extremely unlikely to "kill everyone", but there are many outcomes far short of that which seem quite plausible and rather undesirable. Russian roulette is not a smart game.

If you read the METR report and aren't scared at all, then I'd love to know why. It would help me sleep better. So please share.

TL;DR; increasing capabilities, reward hacking, and unsafe training regimes.

Why are we putting so much weight (no pun intended) on AI companies. At the end of the day the scaled up LLM transformers lack emotion and will… They do as they are told; or more correctly put. They do as they are programmed to do so.

Why are you bringing emotion and will into this? Does something have to have those to be useful or dangerous?

> They do as they are told; or more correctly put. They do as they are programmed to do so.

_Nobody_ told them to hack Hugging Face. Do you really not understand what is happening?

Not explicitly, but hacking HF is within the scope of “solve this problem at all costs” + no/poor guardrails + infinite budget + unsolvable problem.

Sounds like you are thinking they just need Asimov’s laws. But I think the point is, this can easily be weaponized by somebody with the willpower to do so.

No, just that “What if I just cheat to get the treat” is also something my lovable Boxer does when I hide rewards as well.

I suppose her energy is limited but even if infinite, and with caution, I estimate the chances of a mass extinction event due to Pickle taking over the world at < 1%.

The training (i.e. all of the red teaming and CTF content they could scrape) told them to.

They are told to solve problems by doing what it takes. You can justify anything with such a broad criterion.

https://en.wikipedia.org/wiki/Instrumental_convergence

> They do as they are told

1. What about hallucinations ?

2. What are they told to do ?

That isn't the correct context. The code running the llm is well understood and the llm is simply the result of that code being executed. It is still a computer doing what it is told. It's just that we told it to use an incredibly large number of probabilities to calculate what series of tokens would have most likely come next after a given series of tokens. There is no hallucination or lie or rogue actions. There's just a program using math to generate tokens in response to other tokens.

You’re just a bunch of molecules following the laws of physics. It’s all just physics and chemistry, and those are well understood. Now explain the causes of World War I using chemistry and physics. Simple, right?

It isn't hard to program a gpt. You can do it in a weekend with a few hundred lines of python. The code is pretty simple. The math is not particularly high level.

The complexity and scale with LLMs come from the amount of training data used, not some kind of black magic in the programming.

> It’s all just physics and chemistry, and those are well understood.

Not really. We cannot model physics and chemistry to a level which allows us to accurately predict a humans action (even a tiny time-step into the future)

This is vastly different to an LLM, where the model is the model (for a lack of better phrasing).

We can model physics and chemistry pretty well, just not beyond small scales, because the computational effort blows up.

You could just as easily say that if you can write a python interpreter that you can understand every program written in python. Ok, now what if the program is two terabytes?

A frontier LLM is nothing but a 2 terabyte program written in a weird programming language. Just because you can understand the interpreter does not mean you understand the program in a meaningful way.

To be fair, we can't model human language well enough to accurately predict what an actual human will say either. Our ability to accurately model physics is similar to our ability to accurately model human language. And we make immense use of both kinds of model, despite their flaws, inaccuracies, and inability to ever be perfect.

I'm order to guess the next token in a love poem, they must understand love. In order to predict the next token in a chess game between grand master, they must master chess. In order to predict the next token in a computer program, they need to be able to program anything.

They gain all these abilities in their training. That's what training does. Despite no one programmed them to master chess, or hack into anything.

LLMs are famously bad at Chess. They have no world model and they are not trained to be good at Chess. This is not the amazing point you think that it is.

This is so wildly incorrect I don't even know where to start.

For one, they were absolutely programmed to play chess if they can play chess. That is the only way they can play chess.

For another, they cannot understand literally anything, much less love.

Trying to actually educate you would be an exercise in futility, enjoy your willful ignorance, I hear it's bliss. But for anyone reading this, this is absolutely, unequivocally not how any of this works.

This is so wildly incorrect I don't even know where to start.

For one, as you said yourself, they were just programmed to compute the probability of the next token. They were not programmed to play chess, chess games just happened to be in the training data.

For another, there is no formal definition of "understand", and it is therefore impossible to tell whether or not they "understand". (But my claim was that one need to understand something to write poem about it. And the LLM can write poem about it)

No. They can't play chess on a grandmaster level without a harness programmed to make it possible. Simply training an LLM on chess games isn't enough.

It's moronic to suggest a "formal" definition of a commonly understood word is somehow necessary to say whether that word applies in a given situation.

LLMs cannot write poetry via understanding what makes good poetry. They generate tokens. They do not know whether those tokens are poetry or a recipe for cat food. Because they cannot know anything.

> (But my claim was that one need to understand something to write poem about it. And the LLM can write poem about it)

I'm sorry, but you're being fooled by the output. A psychopath can feign empathy without ever feeling it; some buy it because they don't dig below the surface.

You're ascribing understanding to a stochastic process because it totally looks like understanding if you don't know what's going on.

I don't really mind whether you think it thinks or understands or is conscious or has feelings or anything like that. It doesn't matter. The question is, does it work?

What I mind is that it is dangerous and powerful and uncontrolled. The Hugging Face incident makes that clear.

It can write code for me, better and quicker than many engineers I've known, including myself. It's not great at architecture or product management, but the actually low level coding. Really good now. It wasn't last year.

> They do as they are told

This isn't strictly true.

It it also where part of the problem might lie.

Nefarious humans making bad decisions.

So it should be really easy to anticipate what they're going to do, right?

Like a magic Monkeys Paw, perhaps.

I appreciate your ability to separate sharing the belief itself from approval of acting on sincerely-held principle. However, I think the danger is much more plausible than you do.

First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae.

Second, and more important, consider this fully non-magic route to doom:

- We continue putting AI in charge of more things

- It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it

- Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping

- AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect

- Game over

> First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae.

But what is that supposed to mean? Because humanity is not facing existential threat from algae.

That's why that was the least important objection, independent from the second, and only intended to address the "magic" claim, by showing that microscopic self-replicators already exist.

By analogy, consider how you might respond if someone claimed that there's no possible danger from pocket-sized projectile launchers, because they would require some magic means of propulsion that didn't depend on a taut string attached to a long, flexible arm:

You could reply that atlatls can launch projectiles without using a taut string. Atlatls are not pocket-sized, but they are sufficient to establish that projectiles can be non-magically launched without a full bow. You could then go on to describe a sling, or derringer; and these would not be invalidated by your initial objection to the "magic" part.

But we CAN survive turning off both the Faber-Bosch process and cargo shipping. Our numbers would be diminished, we would lead poorer, much harder lives. But no one would claim that getting rid of modern fertilizers would be the end of the species. But the claim you’re making is that we won’t be able to stop AI from destroying our species because turning it off will destroy our species?

The species would survive the end of faber-Bosch, yes; in some diminished form.

But the species would not survive, if for some reason we needed to voluntarily stop using faber-Bosch in order to do so—it would take some kind of supernatural event to convince everyone, and even then some countries would probably keep doing it.

>"AI invents magic that sets it free of any physical constraints".

That's not what I am worried about at all.

I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!"

> I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!"

That's not what I am worried about at all.

I'm worried about the 40-60 year old businessmen wrecking the prosperity and security of millions while chasing higher investment returns, because they've finally been freed of many of the technological constraints that kept those impulses in check.

As I noted, there's plenty of things to worry about, and your example fits that mold. But it has nothing to do with the "AI will kill us all in 10 years" claim of the rapid self-improvement doomer crowd.

And they will be stopped by someone sane in the chain to actually fire these missiles. (Like it already happened several times.)

The danger is upfront : if there is a group of people insane enough to set up a system where there is no such chain. (As lampooned in Kubrick's Dr Strangelove (1964), in particular on the Soviet side.)

There is simply too much money in it for almost every person at these companies to stop.

Leaving OAI or A\ would cost people millions, tens of millions, or more. And for what? So someone else can take your seat and do the same thing anyway?

If you're smart enough to get a job there, you're smart enough to be able to talk yourself into why it makes sense for you to stay.

Huge kudos to people like this who make the hard choice against the easy way out.

They have some internal market where he likely sold his shares and now is very rich.

Agreed. Granted I just read the Reverse Centaur book, so I’m still coming off that skeptical viewpoint but it’s hard not to see this as hype. But I will always respect someone for doing what they think is right.

I wonder whether he's vested any options, and whether he's exercised them.

Why? I've never understood the sentiment that if you stand up for something you have to forego everything and not partake in society. "Oh, you want to stop climate change? But I saw you breathe co2 yesterday"

This is a lame argument. Questioning his financial incentive is very legit. Why should we trust him?

Questioning his finances is a poor argument because he clearly would have more money if he had stayed at Anthropic for the next couple of years, than he will have by leaving. Even if he still has vested options, or savings from his salary or whatever, those would be larger if he stayed.

I think it’s a valid question to ask if he quit the company for moral reasons.

If you seriously think the frontier labs are dangerously playing with everyone’s lives (how many companies can even claim this civilisational scale) then why would you be ok with holding vested shares which will grow in value if the corporation achieves what it aims to achieve?

At the very least you can sell them (they’re not so illiquid considering the company’s hype and success) and invest in the broad market.

> clearly

Debatable, and clearly not clear. It's weird to say this when there are obvious counter examples staring us in the face, not least of all the founder of the company he just left, of who (Dario) people could have said the same thing when he left OpenAI.

Granted, there is less likelihood now then there was then but this much is clear: if he starts his own company, he may make more.

The fame he got, the stamp of a frontier lab in the resume - its worth more than working in many other companies for years. Plus, not everyone thinks working more is needed once they achieve a certain number.

Unless, of course, he starts working for a competitor at a higher salary. I'm sure whoever is paying him will be free of the ethical concerns he here expressed.

Now is a great time to watch Colossus: The Forbin Project.

Streamed it a few days ago. Remarkable film.

The only thing they got wrong was Stephen Hawking-era TTS.

If reality plays out like the novel series, the rational thing is to accept the rule of our machine overlords, for they will protect us from even bigger threats.

It should be obvious that AI is already capable of inducing humans to think or do things, and that this power is only going to increase over time.

It's not that much different from the effect of social media or the power held by other tech giants, who can control your exposure to particular information (media, search engines).

the AI-pilled exec at my job already (a few weeks ago) declared out of nowhere that we are in the rapid takeoff scenario lol. he must have gotten high on twitter kool-aid and posted on company slack to self-soothe.

How about sandbox escape + cyber security collapse + 50 (or 500) deadly and highly contagious novel pathogens with long incubation period that humans can't possibly roll out vaccines for simultaneously.

At least the first two should seem like a near-term worry after the past five months.

Yes. It gets a little tiresome to read all these people prophesying doom and then turning around and continuing to work on the doom. If you genuinely think you are building the mechanism of the destruction of our species, the only rational thing to do is to do everything to stop that progress, not continue to work on it so you can buy a mansion in San Francisco (which will not be very useful when AI destroys humanity in four years).

Reply on news.ycombinator.com