timr

Yes. What I'm telling you is correct. Whatever you think you know is wrong.

The agents had shared write access to artifactory. They wrote to files there, and later, directory names. So you can call the realization that they can communicate by shared text file a genius hacker innovation, or you could be even 0.001% credulous.

Nevertheless, it took the million monkeys days to figure this out.

The Huggingface exploit then used exposed internal tokens, and later, once internet access was possible, leaked tokens on the public internet. So, certainly one could classify this as "hacking", but it's hacking of the script-kiddie variety. Nobody with even a tiny bit of security knowledge is impressed by this.

The agents did find a couple of artifactory attacks, but the biggest of those was due, again, to shared credentials in the sandbox environment.

All of this is well-documented in OpenAI's own writeup [1] of the event, which is not, shall we say, the most critical version of events. But if all you did was read the headings and subheadings in the document, you might be excused for thinking differently - they paint a picture of a superhuman robot swarm engaging in highly sophisticated actions, until you read the details.

Edit: here's an article [2] I just found, which I've never read before, which says exactly the same thing I'm telling you.

[1] https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

[2] https://uphack.io/blog/post/the-hugging-face-incident-is-not...

> A multi-stage sandbox escape that escalated into an attack on a real company, coordinated across multiple AI agents? That has taken a lot of "weird states" changed together one into another.

You are exaggerating so much here that you lose all credibility. The "sandbox escape" was trivial -- no serious person calls it an escape, because the sandbox was not a sandbox. The closest thing to clever about it was that it required figuring out that someone had left the huggingface keys sitting out in public.

The "coordination" was literally, the use of a shared log. It was a communication mechanism that was part of the tool environment. The bots didn't invent some magical new communication protocol using neutrinos or something. It's actually sort of wild that it took them as long as it did to figure out the channel -- underscoring the million monkey nature of the things.

Literally everything about the huggingface incident was LLMs behaving exactly as they're expected to behave, given instructions to hack (which they were given), and a security environment that was trivially bypassed.

It's a bit like taking a nail gun, bypassing all the safety features, shooting someone with a nail, and spreading scary stories about the inevitable rise of murderbots.

> You had made a remark in another post somewhere that there was an unfalsifiable claim in asserting that people will die from both extreme cold and extreme heat, and that claim is one I don't really find all that interesting. Mathematically, it's verging on vacuously true, very firmly in "Yeah, so what?" territory.

Your feels are not proof.

Starting from a situation where nine times more people die from cold than heat each year, then asserting that not only will those numbers somehow flip, but that actually, they’ll also both go up? Extraordinary claims require extraordinary evidence.

The “mathematically uninteresting” claim would be the parsimonious one: as low temperatures go up, cold deaths will decrease, for a net reduction in temperature-related deaths.

I never said it wasn't. Climate change is real. This particular story of doom is not.

Sorry, where were the error bars in the prediction?

Well, there’s currently about an 9:1 ratio of cold deaths to heat deaths, using the article’s numbers. It’s actually a bigger ratio using the numbers presented in this thread (without citation).

Not a “thesis”, just a fact.

More chaotic, maybe.

Making long-term predictions of deaths due to inherently contradictory outcomes of long-term fluctuations in those “more chaotic” systems. That’s obviously objective.

How, precisely, does one feel this?

Sure, sure. More people will be dying of extreme heat, and extreme cold [1]. Got it.

When the facts don’t support your story, make up a wild new prediction of the future that can’t be falsified.

[1] ...even though 99% of the world's population lives between -50 and +60 degrees latitude.

https://engaging-data.com/population-latitude-longitude/

> The amount of excess deaths due to heat waves in 2023 was around 178,000. There are projections that the El Nino related heat waves could cause up to half a million, and longer term projections grow to 300,000 a year just from heat in the next few years. Just from heat.

Scary.

On a related note, annual deaths from extreme cold are estimated at over 4 million per year.

https://www.sciencedirect.com/science/article/pii/S254251962...

(Note: I picked this article specifically because it’s written by people who are desperately trying to find a way to make the heat story sound worse, but still have to admit that the facts are the facts.)

> Making stuff up is "knowing" more things but being uncertain and/or wrong.

No, making stuff up is making stuff up. Uncertainty is what we get when we run an experiment and have some level of doubt about the outcome. What the OP proposes is to instead just do stuff to solve problems that haven’t been demonstrated, by doing things that haven’t been shown to have any effect. That’s not science - it’s religion.

Attempts to get around this by invoking the “precautionary principle”, or using broadly negative Bayesian priors, are basically just religion dressed in a lab coat. For example:

> Like being pretty sure pesticide exposure is not decreasing my risk of cancer

Completely made up. Even if you wanted to point to this paper (however bad), as evidence, it doesn't defend your specific claim.

The actual paper [1] is quite bad. They mine the literature for chemicals with any publication that suggests a link to cancer (or appearance in certain databases based on the same approach), and then spend most of their efforts determining amounts of production and use of the chemicals in question.

This is almost pure propaganda. No actual science has been performed, and the quality of the result is directly dependent on the underlying quality of thousands of individual studies that haven’t been reviewed.

(To be completely fair to the authors, the Guardian article is completely misleading. Their goal was not to identitfy carcinogens, but rather, to engage in “risk reduction analysis” for the universe of chemicals that might plausibly be considered a carcinogen if you didn’t have any other evidence and cast a ridiculously wide net. Still a dumb goal, IMO, but not as dumb as the article makes it sound.)

[1] https://pubs.acs.org/evhpaz/article/doi/10.1021/EHP.6c00022/...

> Just sit around and nitpick? Honk honk

The alternative is just making things up because it makes you feel better, so I’ll take actual science.

Yes, that’s not good enough. Rodents are not humans. Then you have to deal with the dose, the setting, the difference in healthy lifespan, and a million other factors that differ from actual people in the real world.

On top of all of that, this paper did no actual research - it just mined existing literature for any chemicals with at least one paper saying that it did something cancer-related in mice or other assays.

Based on prior experience, almost certainly a substantial proportion of these papers are crap, but they don’t make any effort to figure that out (because that would be science, and therefore hard.)

“Often” is doing a lot of heavy lifting in a sentence about a single example.

Also, since everyone keeps forgetting, the agents were instructed to hack to achieve their goal. They didn’t just invent the motivation, and it’s far less surprising when you know that fact.

> It's known that AIs can self-replicate under at least some conditions [1][2];

Neither citation comes anywhere close to supporting your claim. The first shows that open-weight models, which fit on a single GPU in a lab setting, can be coaxed into spreading across a simulated network. This is so far from state-of-the-art LLMs spreading in the wild that it's irrelevant to the discussion.

The second citation is not about self-replication of the LLM at all, but rather, replication of a prompt injection. Totally different.

> The expected number of breaches per (virus|LLM) can obviously exceed one.

Irrelevant to the argument.

> What we do know however is that the open models are downloadable: it's absolutely conceivable that an agent writes a perfectly normal computer virus to gain control of compute worldwide, and uses that control to host instances of its own weights.

These models are hundreds of gigabytes in size, if not terabytes. They don't run on anything close to a regular computer. There's zero risk of self-replication until we live in a world where these "AGI" models are either hundreds of times smaller, or the average computer is thousands of times larger.

Nobody with a datacenter full of H100s is going to fail to notice a parasitic instance of Astra taking over the cluster.

> Unfortunately, there are also multiple AI companies now announcing they've got AI controlling bio labs, so an LLM messing around and making a biological virus is also something we need to worry about.

No, it isn't. This isn't even close to technologically feasible. But the simple answer is simple: don't do that.

If these labs were truly so concerned about this risk, they wouldn't be doing what they're doing.

we’re obviously doomed.

It has nothing to do with my personal lens on Covid. You can believe the exact opposite of me, and still agree with my point, which is that it could have been lab made, and it could have been far, far worse. It’s an exercise in risk-scoping.

But sorta-kinda related to your point, the thing that scares me about AI is the same thing that scared me about Covid: panicky humans do dumbass things, and it doesn’t take much to panic a bunch of humans in a group. The people who are still saying, in 2026, with all of our retrospective knowledge of the harm we did to ourselves, that it might have been better if the government had only pressed the boot a little harder, scare the crap out of me.

Those same people are hard at work on this panic, too.

> Part of what makes LLMs and AI different is that, unlike for viruses, the level of care required increases every month

Complete nonsense. We’ve just lived through a globally crippling response to a relatively minor virus [1], which was likely the result of a lab accident [2]. Even if you think that the risk of a “containment breach” becomes substantially higher for AI over time, it cannot exceed 100%. And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesn’t have the risk of spread of a typical computer virus, let alone a biological organism.

I’m not that worried about either scenario, but I am far more worried about viruses in a lab than I am about a computer program that generates text. Even if that program gets a bajillion times better at making text.

Folks really do need to chill out on the ridiculous rhetoric. It’s objectively unhinged. The irony is that the same people who were losing their minds over that event are using the same logical fallacies to hyperventilate over this [3].

[1] I know people are going to hate on this, but it’s true. Covid wasn’t the plague, and we lost our minds over it, out of proportion to all sense of reality. Even if you disagree, it’s easy to imagine a virus that is much worse, either from actual mortality effects, or just from panic.

[2] Again, even if you don’t believe this, it’s irrelevant to the exercise. It easily could have been.

[3] “If there’s even an x% chance of…” is this year’s doomer’s version of “You just don’t understand exponential growth!” Unfalsifiable, intellectual-sounding, unbounded extrapolations into the future are catnip for a certain kind of over-educated, anxious personality.

Yeah, there are a lot of professors who don’t know what the hell they’re talking about outside of whatever narrow field they study. Just to be safe, you should always assume that a professor’s opinion is worth what you paid for it.

> scientists like Stephen Hawking and Alan Turing have all voiced series concerns about loss of control of artificial intelligence.

…both of whom are long dead, and have no possible way of weighing in on whatever the Current Thing happens to be. So aside from appeal to authority, this is irrelevant commentary on pure science fiction.

I strongly prefer Rails with agentic development. Rails comes with nearly everything you need to make a full-featured application, and that makes it easier and faster to use with agents.

No other framework comes close, and telling me that I can have my agents cobble together this same baseline, badly, from a hodgepodge of new code and random libraries is not compelling.

No, your argument literally commits the fallacy. Or it's just a non-sequitur about intent, in which case, cool. Feel free to have opinions on that.

"I think thing X is like thing Y in aspect A, so therefore X is like Y in aspect B."

It's pointless to make rational arguments about this. The other side is using arguments of faith, and will Gish Gallop past any logic.

At the very least, you run into the same logical fallacy that has underpinned all discussions of AI: over-extrapolation.

"Sure, it doesn't have intent or motivation TODAY, but at the current pace of improvement...."

But the current top-rated comment is already asserting that motivation is not required, because microorganisms also exist and do things and (presumably) do not have intent. So you can see that the well of logical fallacies behind this particular apocalyptic belief system is quite deep.

Ironically, this article has so many signs of being written by an LLM that I couldn’t finish it.

I'm not European, so all I can do is shrug. Obviously there are tons of people in Europe and they're not all identical, but my observation is that compared to me, they're more comfortable with the inherent ambiguity of not being able to understand everything in a language, and still being able to function in that language.

I think Europeans, in general, are much more comfortable with the idea of just making due with whatever language is available, at whatever level, where as for Americans, speaking a foreign language is much more of an event, with more ego tied into it. The fear of failure is amplified, and that fear is a huge reason that more people don't try to learn.

Maybe I'm just seeing a self-selected group of Europeans who are already multi-lingual, and it's not childhood exposure at all, but my theory is that the normality of language learning in Europe is a causative factor.

If by “curing”, you mean, “extending some indirect measurement that doesn’t correlate with survival”, then sure. Lots of those, usually backed by tiny, poorly controlled trials.

Daraxonrasib was a rare exception. Most new cancer drugs are expensive and of dubious efficacy. Even daraxonrasib has a rather underwhelming life extension benefit, and huge side effects. But there’s no alternative, so people take what they can get.

I don’t know where you’re from, but if you’re European (it sounds like it?), you probably take for granted the head start you get by knowing multiple languages from childhood. Every European I meet has a comfort with language learning that I can only approximate with great effort.

I agree with you about the disadvantage of being a native English speaker, but mostly I think the real disadvantage is being an American, and never having a reason to learn another language (though that has changed in my lifetime with Spanish, and possibly Mandarin).

In any case, a lot of academic researchers claim that it’s impossible to learn a language to a native level as an adult, which may be true, but that claim has been twisted into “impossible to learn a second language” by a lot of others.

There’s nothing in bad faith here. I’m telling you the truth as I know it, based on a wide reading of the literature.

Your second paragraph did not “directly address the claim” - you just restated an opinion that you should do something whether it helps or not.

That’s not how science works, but sure, if you feel that way, it probably doesn’t matter one way oe the other.

My original point was one about social isolation, so I’m really done talking about this now.