Hacker News Reader

I don’t agree it’s that small. I think that the tools have advanced to the point where there are maybe 100 people pushing it forward at each company but a shortlist of thousands who would love to take their place and would be more than capable of it if entrusted with the compute and resources the current heads are.

Hell I'd do it

> [...] but tons of people on all sides with all motivations [...] all seem to agree there are real, existential risks here [...]

Are you talking about the Jacob Coxon story and the people that came out agreeing with his post? It was a well coordinated campaign, not a genuine grassroots development.

It's a lot more than just Jacob Coxon's resignation, and many have been talking about this (some for years) before now.

Also, if you have actual proof Coxon's resignation "was a well coordinated campaign", show it, otherwise I'm calling bullshit.

Here you go: https://xcancel.com/ParkerThayer/status/2097759699626328575

What a complete and total joke. One random Twitter thread from a guy "illuminati-ing" the events does not a conspiracy make.

Besides, I would assume that someone who was genuinely concerned about the direction of AI research would want to amplify their voice, as he obviously did by his resignation post on Twitter and then having that picked up by prominent voices. That doesn't make it some sort of nefarious astroturfing campaign. And there was tons of genuine fear before this guy resigned when the details from the Hugging Face and other attacks came out.

Plus, that Twitter link you posted obviously has some weird tribal bone to pick against Democrats, as if there is some sort of Democratic conspiracy against AI. Yeah, does he think this guy is a Democrat?: https://x.com/HawleyMO/status/2098137180392604083

> What a complete and total joke.

The only joke I see are your comments which make absolutely no sensible argument in support of an even less sensible cure - the magic "slowdown".

What's "slowdown"? Everyone is free to slow down to their heart's content, OAI or Antro or whoever. What's hidden behind this deceptive word? Can you explain?

According to your claims, there are only "50-100 technical people" in the know, so just list their names and ask them to slow down, if "they're already rich" they will listen, there's no need for any other "slowdown", right?

> tons of people... all seem to agree there are real, existential risks here

I don't see anybody with a proven lack of conflicts of interest agreeing with this but I do see megatons of people disagreeing with the notion that a silly token generator can be anything like an "existential risk" on its own. And the disagreeing have a point, they have strong arguments to support their view, you don't.

There have been volumes written about this, just because you haven't read them doesn't mean they don't exist.

I think the AI futures project, https://www.aifutures.org/, is a good place to start. You definitely don't have to agree with all the steps in the various scenarios, but I think more than anyone Daniel Kokotajlo and gang have traced out, step-by-step, exactly how AI acceleration (and takeover in some scenarios) occurs, so it at least provides a good framework to discuss and disagree over the details. It's just beyond annoying at this point to hear people rail against him or his general ideas who have clearly never read those ideas. Like the number of times I've read people think that AI becomes like SkyNet and acquires "sentience" in a spark and then nukes us is mind boggling, but at least it shows folks who have 0 idea what they're talking about because this is nothing like what anyone rational thinks that AI takeover looks like.

And after you read through the various AI Futures scenarios, it then becomes blindingly easy if you read the METR report about the details in Hugging Face attack (and reports on the other attacks) how a slightly more capable and slightly less aligned AI could do more catastrophic damage if AI were hooked up to and running more real-world systems.

(btw the parent was flagged and dead when I tried to post this initially, no idea why. I hit vouch and it resurrected)

> there are probably around 50-100 technical people who are crucial to moving the frontier forward. So, if you're one of these people, just stop.

I don’t think this is true. I doubt the technical know-how is nearly as much of an impediment to progress as raw compute. If AI were something that could be trained end to end on consumer hardware then crowd powered open source would blow the “labs” out of the water. The moat is money, not ability or innovation.

Even at the scale of 100 people there is a coordination problem. If the 50 most conscientious researchers quit, then the 50 left behind would be the more aggressive group and be less interested in AI safety.

Personally I agree though. I would not be able to work on AI model development right now as I just don’t think it’s ethical. Fortunately for OAI/Ant I lack the relevant skillset anyways.

If the leaders want to slow down, but the companies aren't, then it isn't those 50-100 people demonstrating no agency - it's just the opposite. Or, the leaders words are empty.

> The number of folks with the technical chops to really push on the forefront of AI is just not that big.

Utter bullshit. We are still scaling transformer architectures initially introduced ten years ago. Ten years of phds trying to improve on what Google produced ten years ago, mostly failing.

What has actually changed and can explain the progress we’ve seen? Do you really think it’s just a collection of better and badder RL gyms? Get real!

The main thing driving progress in ai is massive amounts of capital investment and incremental improvements in hardware (mostly memory bandwidth/capacity) and computer networking (we got better at collectives). The AI labs, ironically, have nothing to do with it. They’re the vessel for capital. The people actually driving things forward mainly work at nvidia.

The reason the models are better today than they were 3 years ago is almost exclusively due to better hardware and infrastructure software. Not better data, not better model architecture. The evidence of this is pretty easy to feel: the reason opus 5 doesn’t feel much more capable than opus 4.6 did is because they run on the same hardware generation. The reason opus 4.6 felt much more capable than anything before it is because it coincided with the scale out of a new hardware generation.

This also explains why Gemini is lagging, they don’t have an nvidia training cluster.

Thank you very much for your sensible comment.

I'd like for more commments to contribute at least small things to the discussion instead of just mindless parroting.

Also, I can't follow your advice, as I don't work in AI at all ;)

Turns out a lot of people can be mentally ill and delusional at once-- especially when there is a vector to turn their sickness into both profit and an instrument of control.