Comment by hodgehog11 | Hacker News Reader

Comment by hodgehog11 | Hacker News Reader

Parent Full thread

I love it when I hear all of this compounding evidence on this claim, because none of it is strictly wrong, but it misses the point, and lulls us into the feeling of having quick solutions available. Yes, the top execs are probably approaching things this way, but these labs are not that top down. There's too many things going on.

Here's a different idea: talk to the to the staff. Not the evil CEO, but the nerdy guy on the ground who graduated from a top university, wrote a few research papers, and got a job there. I have. They have rose-coloured glasses of the institution, and not a lot of life experience. They were never taught to be careful, and still don't really comprehend what they're working with. They don't see real danger, they see a toy, and they see research that is low-hanging fruit. What they are doing is basic stuff. They are not setting up proper sandboxes because they barely need to think at all. People seem to think that these are all amazing computer science experts working on highly advanced technology. They are not. OpenAI researchers see huge improvements on this gigantic toy, crazy behaviour, and they are enamored by it. "Oops, people are angry, so maybe I'll make a slightly better sandbox. Let me ask ChatGPT on how to do that." A more senior researcher would be horrified by how little effort they need to put in to get such terrifying results. Junior researchers think they're just top stuff.

But I appreciate your discussing how to isolate this stuff. I honestly don't think the OpenAI researchers I've spoken to are aware of this. (Anthropic is a totally different story, BTW.)

> ...talk to the to the staff. Not the evil CEO, but the nerdy guy on the ground who graduated from a top university...

Why would I talk to the people who don't have the power to set company policy and fire anyone who fails to comply with it? I've worked at several big companies over the years and have observed the only even vaguely reliable power that folks at the bottom have to change company policy that management substantially benefits from is to quit en mass.

> ...but it misses the point, and lulls us into the feeling of having quick solutions available.

The point is that these companies claim they're working on oh so dangerous tools that are very likely to kill us all, but the evidence that these companies don't behave even a little bit like this is true keeps pouring in.

The CEO [0] can set company policy. In the US, the CEO [0] can fire people who fail to comply with policy. Most folks would -correctly- think that a CEO of a company who is working on a tool that has a high chance of destroying humanity is very interested in not destroying humanity (accidentally or otherwise)... if for no other reason than the fact that once all of the humans are dead, his company can't make any more money!

> Junior researchers think they're just top stuff.

In sane companies, when a junior staff deletes the prod database, an investigation is launched to understand if the deletion was unintentional and -if it was- what about the company's procedures need to be fixed to make sure that that doesn't happen again. In sane companies, when one performs a live test of a tool that has

* been designed to attack computers

* been instructed to attack computers

* had its safeties removed

one ensures that this computer-attacking tool cannot attack computers that aren't owned by the company. Both OpenAI and Anthropic have way too many senior staff on staff to be unaware of this... the fact that the computer-attacking tools could get out to the Internet is -at best- negligence. [1]

[0] ...and many-to-most managers in one's management chain...

[1] For a discussion of the decades-old techniques for preventing computers in datacenters from escaping logical airgapping see [2] and [3]

[2] <https://news.ycombinator.com/item?id=49862136>

[3] <https://news.ycombinator.com/item?id=49862373>

I agree with this almost completely (especially about sane companies, which I think we can all agree they are not), but I think it's important to separate the notion that these tools are potentially dangerous from the behaviour of the CEOs. The executives are there to make as much money as possible, that is all they care about. It's the researchers who are playing around with these things that are causing damage with them (aside from the damages from the data centers themselves, of course). They need to be better than this. It's not enough to blame senior leaders in this case, since they are clearly problematic. The junior staff share responsibility now too.

> OpenAI and Anthropic have way too many senior staff on staff to be unaware of this

Anthropic, yes. For OpenAI, not in the way you might think. Most senior staff are research scientists who have likely not even thought about sandboxing and cybersecurity in their lives. They outcompete the rest. That's why so many of their "safety" staff left for Anthropic; the culture at OpenAI has never cared for these sorts of topics.

It seems like you're trying to claim that -unlike OpenAI- Anthropic has a robust culture of security and safety and would never do something so negligent as test a highly-capable computer-attacking tool that has been instructed to attack computers in a test environment that's connected to the Internet.

Well: <https://www.bbc.com/news/articles/cz7dl7w8y7po>. [0]

I stand by my claim that the conduct of the major LLM manufacturers does not look at all like what you'd expect from people who believe that they're working on something so dangerous that it could plausibly wipe out all of humanity.

[0] I refer you back my first post about how one sets up a test environment when one actually wants to ensure that a machine doesn't connect to the Internet. [1] Just like OpenAI, Anthropic did not do that.

[1] <https://news.ycombinator.com/item?id=49862136>

No, I'm not implying that. I'm suggesting that many more of these issues are likely to come from OpenAI due to their culture (not that that is okay). Anthropic has no excuse, I know they know better. One might argue that they were initially negligent because they didn't really believe it would do anything like this. But having further incidents is pure negligence at best.

> I'm suggesting that many more of these issues are likely to come from OpenAI due to their culture...

Both companies have been equally reckless with the tools they sell and the tools they test. Your warm and fuzzy feelings that come from speaking with a handful of line workers at Anthropic don't change that fact.

I've spoken to researchers at both OpenAI and Anthropic. They are not "line workers". The researchers at Anthropic are more senior, that's all. None of the folks I met, from either institution, would likely know how to properly airgap their runs. I don't, I'm a maths geek, not an engineer. Honestly I wouldn't even have thought to look it up. Maybe that's a hiring problem.

Both companies have not been equally reckless, but they have both been reckless to varying degrees. I don't have any personal stakes in either company, and would prefer both to go bankrupt if I'm being honest, as I despise their behind-closed-doors attitude.

Other than that, do you think AISI is reckless too?

> They are not "line workers".

In the "management" vs "line worker" split, they absolutely are. You appear to think that line workers cannot be highly skilled, which is absolutely not true.

Anyway, I'm super done here. Hopefully one day you'll learn to judge a company based on its actions [0], rather than what it claims about itself or how you feel about those of its employees that you've met.

[0] ...especially when considered in light of standard practice for companies in similar industries...

Reply on news.ycombinator.com