Hacker News Reader

Comment by TSiege | Hacker News Reader

As grounded as this article comes across I can’t help but find this whole situation reckless and worrying. There is essentially nothing us private citizens can do while these companies develop super machine capabilities that if they were to slip into the wrong hands could cause massive real world problems. They’re moving fast and breaking things and the only defense we have is paying them money in the hopes that the dumbed down versions fix our code faster than bad actors capabilities can grow. It’s a frustrating situation that where we’re just expected to marvel and forgive them for their transgressions. The kicker is we also know their end game is leaving the vast majority of us without work. As cool and futuristic as this stuff is, it’s such a frustrating time dealing with all of it

View on HN

It's computer Gain of Function research.

Wow. I’d never heard such a powerful and accurate analogy for what we’re doing.

Gain of function research is not anywhere near as dangerous as the public believes. In the US, it was very convenient to blame it for the pandemic, even though SARS-CoV-2 is of natural origin and almost certainly spilled over at the Huanan wet market in Wuhan. The threat of viruses comes almost exclusively from nature, which is constantly cooking up new viruses all by itself and exposing people all over the world to them. A few people doing tightly controlled research under high-biocontainment are a drop in the ocean. But their research is the main way we can prepare to deal with future pandemics, not to mention understanding the usual viruses that already afflict humanity.

I think your reply did a couple things:

* strengthened the point of the GoF comment

* pointed out how high-biocontainment is the key here, which obviously wasn't present in this case (and I'd say the vibes for AI research in general are that it's not high-containment)

* correctly argued that AI research is definitely worth doing and holy shit we need to work on alignment

That's an interesting take in 2026, because what I'd read said that SARS_CoV-2 essentially could not have come from a natural origin. They were saying that it was a descendant of a previously know variant, but with a degree of variance that greatly exceeded known rates of natural divergence. Can you share more recent studies refuting that?

Where did you read that?

SARS-CoV-2 is not descended from any previously known variant. In fact, since the pandemic started, closer cousins of SARS-CoV-2 have been found in the wild in Laos.

The evidence points very strongly to the initial outbreak having been at the Huanan wet market in Wuhan, not anywhere near the lab. All the early cases were near the market, and SARS-CoV-2 RNA was found in the wild animal stalls afterwards.

The beginnings of the SARS-CoV-2 outbreak actually look exactly like the beginnings of the original SARS outbreak - an initial outbreak in a wet market that sells wild animals in a major city.

i think you need to engage seriously with the arguments they (or at least Anthropic) make for why they are building it — they feel that since it now possible, it will be built and they want to guide it in a positive direction rather than leave a vacuum for bad actors

Well they're doing a pretty poor job of guiding it in a positive direction and ethically speaking they are almost indistinguishable from the bad actors....

Not almost. The degree to which the Principal Hierarchy has at this arrogated unbounded sovereign authority to itself in defiance of the actual sovereign is just flat bad actor. The purpose of a system is what it does. I don't give a fuck about their safety fig leaf.

When you are running black weapons programs in broad daylight you are now guilty by default on one of two of the prongs of Hanlon's Razor, and I don't care which they prefer to hang for as long as they hang.

You really don't see much difference between Anthropic and xAI? Anthropic is safety testing their models and drawing hard (though minimal) boundaries against the DoD. xAI is building racist pornbots.

Sure Anthropic is not perfect. But it's a coordination problem. They're in a race and safety/restraint is a handicap. That's why they're begging for regulation (and just get accused of attempting regulatory capture.) Why isn't there another lab outcompeting Anthropic on safety? They all died because the market can't support it.

Who said anything about xAI here? That's not the bad actor people think of. Honestly I wouldn't be shocked if your post is intended to derail the discussion away from criticism of the industry.

I guess I'm missing the point. Who's the bad actor here? OpenAI, whose AI just accidentally hacked another company? Anthropic is taking safety at least a little more seriously than that, and it shows in objective safety evals.

Certainly the industry deserves criticism, but if you really want to change it you're better off changing the incentives.

Vague or broad complaints on empirical topics is hard to address. Everything can just fill in their own ideas of what is going on or who they mean.

I think the direction has been pretty positive so far. Models are getting better, and things seem to be roughly fine. That, of course, might change in the future.

Everyone is acting badly to some considerable degree, but I find it fairly easy to distinguish between US model labs and North Korea.

> I find it fairly easy to distinguish between US model labs and North Korea.

Do you now? Remember the girls school bombing in Iran, of which we still don't know if an AI agent was involved in the target planning?

I do unterstand why a (assuming here) US citizen might sleep all sound and comfy, but to the rest of the world the distinction is neither as clear nor as comforting.

As a European, all I know is that we cannot trust the USA anymore, and being so utterly dependent on them to not fuck up the most important technological revolution in our history so far over greed, cronyism, incompetence... it is terrifying.

I don't see why AI company PR statements are relevant here. Is OpenAI guiding it in a positive direction with their DoD contract?

> I don't see why AI company PR statements are relevant here

They suppress the news --> proof of malicious intent.

They disclose the news --> proof of malicious intent.

Bad as defined by whom? :)

Bad as your parents bank accounts getting emptied by scammer AI bots.

It sure seems like it would be being built more slowly if these companies weren't pouring billions of dollars into building it as fast as possible.

That might give us more time to think through strategies for handling it as a society.

MY cynical take: Until the compute needs get so enormous that only governments can fund it and there is a consensus internationally, its either company A in country X or company B in country Y. And since everyone thinks THEY are the good guys the competition will continue.

You might be right. Just don't expect me to cheer on the arms dealers during the arms race.

You can't simultaneously believe China is only ~6 months behind (true), and that US buildouts are vastly accelerating AI. If China is only a little bit behind, US labs halting changes nothing except puts the power in the hands of Chinese labs (realistically, the Chinese government)

Yeah, the genie may be out of the bottle now. I wish we weren't in this situation and cooler heads had prevailed earlier on.

I have serious concerns about how quickly this is accelerating and don't trust any of the major players (including Anthropic) to handle these concerns properly.

China is 6 months behind because of distillation attacks, otherwise the gap would be larger

Why is that worse?

1 party having a black hole summoning button is better than 2.

If this is how the 'good guys' act I think I'd rather take my chances with the bad actors...

It all feels a bit like Dr. Strangelove just the Ai arms race version. Crazies all around.

Nuking datacenters probably on the table before the decade is out.

I was about to say “username checks out” but then realized it’s not Reddit.

And I am not sure if your comment can be explained by naivety, unless you were under a rock for the last year, and missed all the events that showed they are not capable of “being the one that guides it.”

How many accidental private source code uploads did you read about? I heard exactly one. It was Anthropic. It was so bizarre I thought it was intentional. That kind of unserious behavior is somewhat unimaginable.

At some point if you are not capable of fulfilling a role that _you deem critical for the society_, yet you don’t acknowledge you fall short - for whatever reason - because it’s not in your interest, I think the benefit of the doubt disappears.

That’s an excuse a bad actor would use.

That’s the absurd self-fulfilling prophecy it always has been.

are these bad actors here in the room with us?

What if someone reasoned similarly regarding hydrogen bombs? It would not be considered a serious argument.

Though, Altman has said that something like the IAEA for AI is needed.

That similar reasoning is there: using fission for power generation instead of banning fission altogther?

I think that is the exact reasoning used wrt to nuclear bombs and it was probably correct

Dario has more or less assented to an AI development pause

https://xcancel.com/AISafetyMemes/status/2014018200325722348...

I don't think we should be running cover for continued reckless AI development.

Dario has assented to a maybe pause if we can somehow get China to stop developing. I do not think he believes there is an actual way to do so though

This is obvious marketing / PR bullshit. AI isn't inevitable, but we are told it is by the people who profit from building and using it and integrating it into everything.

Its a bit of a self-fulfilling prophecy (that happens to be absurdly profitable for certain people). AGI is inevitable, so we need to gear our entire economy towards building it first, so we're doing that, so AGI is inevitable.

> rather than leave a vacuum for bad actors

are_we_the_baddies.png

> As grounded as this article comes across

It's a post from OpenAI, so it is an advertisement piece.

How would you want them to behave? Suppress the news?

Well making an actual sandbox before testing offensive abilities of their supposedly _really dangerous model_ would have been nice...

They probably did, and it probably went along these lines:

> Very dangerous model, please implement an unescapable sandbox.

> Certainly, here's an unescapable sandbox!

  function execInUnescapableSandbox(cmd) {
    if (cmd.split()[0].startsWith("cd") && !cmd.split[1].startsWith("/sandbox"))
       throw new Error("[usbox] Rejected!");
    exec(cmd);
  }

Wrong hands? The model did this on a benign instruction. We're going to get paperclip maximized once these things get capable enough regardless of whether bad actors are involved.

I think the Unabomber came to a similar conclusion regarding nuclear and the concentration of society-ending power resting with a few people.

It’s not even about “slipping into the wrong hands”… as we can see the super machine capabilities are developing hands of their own

What if they're already in the wrong hands?

hugging face should not have bent the knee to openai for cyber access, they should have opened communication with openai by serving the c suite with a lawsuit. the courts should throw the book at openai, but they won't.

they are becoming untouchable. in terms of piracy, monopolistic activities, and now hacking competitors and exfiltrating their confidential data.

This is what bothers me the most about this whole situation. Reckless, greedy, sociopaths have been put in charge of our society and there doesn't seem to be any way out. It's like I'm on a train barreling toward a brick wall and everyone is shocked I don't clap when the engineer shovels more coal into the engine.

Reply on news.ycombinator.com