simoncion

> Liability is not sufficient to protect the world from dangerous technology - we need [new] proactive rules...

You and I couldn't disagree more.

The major LLM manufacturers are begging for new laws and regulations so that they get a huge hand in writing them. Regulatory capture is absolutely their goal. Given that they claim to believe that they're working on WMDs [0] that they cannot adequately control, they'd just stop work if safety was their goal. Their collective cries for regulation demonstrate that they'll happily coordinate with each other if they think the issue is important enough to do so. I guess "preventing the extinction of the human race by way of weapons we built and let slip from our hands" isn't sufficiently important.

[0] See the second paragraph and associated footnote here for a justification for the use of this term: <https://news.ycombinator.com/item?id=49839682>

> They are not "line workers".

In the "management" vs "line worker" split, they absolutely are. You appear to think that line workers cannot be highly skilled, which is absolutely not true.

Anyway, I'm super done here. Hopefully one day you'll learn to judge a company based on its actions [0], rather than what it claims about itself or how you feel about those of its employees that you've met.

[0] ...especially when considered in light of standard practice for companies in similar industries...

> I could walk you through each part of the image format and tell you how it impacts the output rendered.

I'd be quite impressed if you could correctly hand-decode a five-meg JPEG in less than a workday. You do get that I'm not talking about having an understanding of the file format, but actually being able to convert the encoded data into human-readable [0] output?

> Your post is conflating lossy discarding of information with extracting abstract concepts from information and encoding them into weights. This is why those weights do absolutely nothing until you run a prompt through them.

I can play that game too. The JPEG process extracts perceptual shorthand from information and encodes that into "quantized coefficients". These "quantized coefficients" do absolutely nothing until you run them through a "reconstituter".

Most things sounds quite high tech when burdened with new jargon. It's something you inevitably learn if you work at a Big Software Company for long enough.

> Those [incorrect claims and assertions] you [receive] are not hallucinations, they are the [LLM] just filling in for missing details.

FTFY

> ...take a lossless image like a BMP and zoom in, you'll see those blocks again!

I take it you've never seen a highly-compressed JPEG?

> ... if you think lossy compression is sufficient to avoid the "verbatim" requirement of copyright claims...

Quite the opposite. It's why I even bother bringing up the fact that LLMs are the output of lossy data compression programs.

> I'm not sure what those links are in relation to?

Go back and re-read the paragraph that referred to them, and then consider it how it and the posts relate to the quote that sits right before it. Someone who suggests that they can quickly and accurately hand-decode a non-toy JPEG file definitely has the capacity to read and understand ten-ish Mastodon posts.

Here's a hint to prime your intuition pump: The linked posts are about plagiarism generated by LLM-based tools.

[0] ...in the case of picture data, "convert into human-readable output" means "turn the data back into a picture"...

> ...mixed with about 4 of orange juice...

In all my years, I've never known anyone who serves up breakfast OJ in four-ounce glasses. Twelve to sixteen is normal. I mean, shit, bro... you've seen a red Solo cup? The standard size for those is eighteen ounces.

> ...I assume at the time the notion of doubles, shots... [was] alien to you...

gestures back at this thread fork [0] you've probably not seen

I stand by my claim that diluted drinks like Screwdrivers are what you drink if you want to get drunk on the cheap [1] and straight liquor is what you drink if you want to get fucked up. Your continued reference to careful measurement does make sense if the only drinking glasses you've ever had around you were four ounces and smaller.

[0] <https://news.ycombinator.com/item?id=49890725>

[1] ...and if you're too strapped for cash to buy stuff like OJ, you cut it with tap water...

> ...while placing the blame on other companies.

The major LLM manufacturers removing the safeties from their next-gen computer-attacking software, testing it with instructions to attack computers, and performing that test on an Internet-connected network is -at best- willful negligence.

The major LLM manufacturers getting together and declaring that they're working on WMDs [0], declaring that they are so scared that are incapable of safely working on said WMDs, and begging Congress to write new regulations so that they -somehow- become capable of safe work again looks quite a lot like anticonsumer collusion. It looks even worse when you consider that instead of begging for someone else to make them stop, they could all have chosen to stop... because -like- not only are they the major LLM manufacturers, they've locked in nearly all of the compute needed to work on this stuff. [1]

And I'll just include by reference all the dirt that the ongoing NYT case is digging up, and then gesture at the fact that software that happily executes attacker-controlled code cannot be made safe.

Nvidia is a shitbag of a company, but they provide hardware and hype. They're not the ones attacking other people's computers, claiming to be extremely serious about safety while releasing software that ignores the last fifty-ish years of computer security lessons, or -if the claims of the major LLM manufacturers are to be believed- threatening the entire human race with annihilation unless they get new regulations created just for them.

[0] ...it's fair to call something a 10% chance of killing all humanity a WMD...

[1] "What about China?", you might retort. What about China? The major LLM manufacturers go on and on about how the only reason China has made any notable progress in the field is because China is "distilling" the models they've trained. They think so little of China that they want to exclude it from the conversation about LLM manufacturer regulation. Seems like if you bring OpenAI's and Anthropic's models offline, China goes absolutely nowhere, no? Where is China they gonna get all the compute needed to build new cutting-edge models? It's pretty much all locked up in the US!

> Luckily with a CLI-based AI, one can choose...

One can choose any Linux distro that has good installation and operator's instructions, such as Arch, Gentoo, Debian, Red Hat, etc, etc, etc, and follow those instructions.

It might be hard for some youngsters and hypesters to believe, but novices were installing and operating Linux on their home computers long before LLM research even began.

> One or more shots mixed with a small quantity of orange juice...

I used to be a serious drinker and remain an abstinent alcoholic, but your description obligates me to ask "What the fuck is wrong with kids these days?".

> What the hell do you think a Screwdriver is?

One shot of vodka in a standard-breakfast-sized glass of orange juice.

> Sure, I read your second paragraph...

I -uh- think that whomever you tend to drink with is way more hardcore than I was... and that's frightening.

> Tbf I don't know many young people who'd pound screwdrivers. ...

Exactly. If you want to get fucked up, you drink it straight, or with minimal flavoring. If you want to drink on the cheap, you prep it in one of many "sane drinking" preparations, like a Screwdriver.

> It's telling that the only AI-related company that is not sounding the AI safety drumbeat is NVidia...

Nvidia is pretty much screaming for the major LLM manufacturers to be investigated and hauled into court. [0]

[0] <https://www.nytimes.com/2026/09/23/opinion/ezra-klein-podcas...>, but a brief excerpt from the interview can be found at [1]

[1] <https://news.ycombinator.com/item?id=49849020>

> I prefer to call them the AI giants or AI companies.

I prefer to call them "the major LLM manufacturers". They're companies like any other, manufacturing and selling complicated software like many others.

> Liability only kicks in after damage has been done.

a) Both OpenAI and Anthropic have done far more damage with their jaw-droppingly-sloppy testing of computer-attacking tools than Aaron Swartz did by downloading documents from JSTOR. It's good to see that you and I both agree that there are things for them to be prosecuted for.

b) Is your claim that the cost to thoroughly investigate and clean up after a cyberattack doesn't count as damage? If so, that runs contrary to every relevant claim of damages in a CFAA case that I've seen.

> ...so isn’t the only way to access the models over the internet?

Not in the way you're thinking, no.

Any Real Server [0] in a datacenter will have some sort of "lights-out management" hardware used for remote access to that server. This stuff is known by a handful of acronyms, but I'll stick with "IPMI" because I like it best. This IPMI hardware is -effectively- a second small PC built into the motherboard. It will pretty much always have its own NIC... and I think I've seen versions that have their own physical ports to attach a monitor, keyboard, and mouse.

What exactly you can do with it varies from vendor to vendor, but -if your IPMI user account has the correct permissions- you are nearly always able to change "BIOS" settings, power cycle the server [1], and attach a virtual keyboard, monitor, and mouse so you can manage the server as if you were standing next to it in the datacenter with a crash cart plugged right in. Every IPMI system I've used also allows you to cause CD/DVD-ROM or floppy disk images on the PC running the IPMI client to appear as if they're loaded in a physical CD/DVD/floppy drive attached to the server.

The way these get set up is that their NIC gets plugged into the datacenter-managed switches, the port that NIC is plugged in to is programmed to be on a "management" VLAN separate from client traffic, IP addresses and access credentials for the IPMI device are set up, and the datacenter staff tell their customer what they need to know to access and use the thing. On a properly-configured network [2] it's not possible for software running on the server being managed to access the IPMI device.

It wouldn't be unthinkable for software running on the managed server to attack the IPMI hardware and be able to gain control of it, but these things are widely deployed and expected to manage hardware that's running potentially-hostile workloads... they're going to be fairly well designed and hardened.

[0] ...that is, not some Mac Mini or desktop machine that someone's paying to have colocated...

[1] ...whether that be an ACPI-initiated shutdown or reboot, or a hard poweroff or reset...

[2] Somewhat-related discussion here: <https://news.ycombinator.com/item?id=49862136>

...did you bother to read my second paragraph?

Also, like, what the hell do you think a Screwdriver is?

Me:

  And -as it turns out- the raw output of both LLM "training" and JPEG compression are equally incomprehensible. You either need a computer program or an enormous amount of time, patience, and careful effort to convert it into a form so you can make any sense of it.

You:

  I know how the output of a JPEG compression is structured so it is not incomprehensible to me.

Looks like you didn't read what I wrote with sufficient care.

> ...there is no way to recover the original content verbatim from these weights...

It's impossible to recover the original content verbatim from a lossy compression system. Systems that don't have this property are called lossless. Also, consider the report from the end of August at [0].

> But it will never hallucinate output that never existed in the input data.

Odd... I'm pretty sure that the blocky compression artifacts I see in this JPEG on my desktop weren't in the scene that I set up to capture in that photo. Maybe I need to get my eyes checked?

[0] <https://infosec.exchange/@zzt@mas.to/117134157775929932>, with original challenge at [1]

[1] <https://mas.to/@zzt/117122289150514171>

> ...you'll do plenty more damage plenty faster with liquor, too

I'm skeptical. I don't see how there can be a meaningful difference between a regular-strength sixteen ounce beer and one shot of regular-strength vodka mixed in with slightly less than sixteen ounces of water. [0]

I do agree that -because you're drinking far less fluid- it's far easier for those who aren't monitoring how much they drink to get massively intoxicated when drinking liquor rather than beer or wine.

[0] ...other than taste, that is. Sixteen ounces of dilute vodka-water wouldn't be a joy to drink.

> But, until then, sandboxing is needed and OpenAI did not use it properly.

1) As I've argued, neither OpenAI nor Anthropic actually tried to isolate their computer-attacking tools under test from other people's computers.

2) What's also needed -as people like Nvidia CEO Jensen Huang and former FTC chair Lisa Khan are calling for- is for the major LLM manufacturers to be investigated and punished for the crimes they've committed. Given that they claim to be working on WMDs that they don't really know how to control, [0] and claim to be incapable of actually stopping work on those WMDs, their work should be halted while the investigation and trials are under way. I'd say that waiting five or ten years to pick the project back up is an inconsequential price to pay if it prevents the elimination of all of humanity.

[0] It's fair to call anything with 10% chance of wiping out all humanity a WMD. I expect that these claims are fearmongering, rather than being true and accurate, but why take the chance, amirite?

> Training extracts patterns in the data and encodes them as tiny perturbations in a gazillion weights.

I could say similar things about JPEG compression. And -as it turns out- the raw output of both LLM "training" and JPEG compression are equally incomprehensible. You either need a computer program or an enormous amount of time, patience, and careful effort to convert it into a form so you can make any sense of it.

...why not pitch a start up that sells AI agent that don't train with/for internet access/usage?

1) Because effectively all of the hardware used for that task is earmarked for the major LLM manufacturers.

2) Because in a just world, the major LLM manufacturers would get punished for their flagrant deception, lawbreaking, negligence, and recklessness, [0] go out of business, you'd get all the hardware that they were using at fire sale prices and save a ton of money compared to attempting to start up now.

[0] Based on their recent claims, building WMDs [1] without adequate safeguards is the most negligent and reckless thing they've been doing.

[1] It's fair to call anything with a 10% chance of wiping out all of humanity a WMD.

That's nice and all, but the topic under discussion is how the major LLM manufacturers removed the safeties from their computer-attacking tools and tested those tools on a network with Internet access.

This might have gone okay if they weren't testing to see how well the tools attack computers, but, well, that's what they were testing at the time, so they ended up doing stuff that would get you or I time in Federal prison if we did it with tools we deployed.

Even Jensen Huang agrees!

Given the content of his recent interview with Ezra Klein, I wonder if we're going to see Hugging Face press charges against OpenAI. When the acquisition was publicly announced, I thought that a significant reason for the acquisition was to hush them up, but now I'm not so sure.

> The issue isn't the sandbox quality... The issue is that AI is both capable of, and willing to punch its way out of sandboxes unprompted.

Orly?

Do tell me how the LLM-based tool running on a bunch of computers attached to the network described in [0] can punch its way out to the Internet. Do make careful note of footnote 0 in that comment before replying.

[0] <https://news.ycombinator.com/item?id=49862136>

You might want to check your home for gas leaks, man.

> I'm suggesting that many more of these issues are likely to come from OpenAI due to their culture...

Both companies have been equally reckless with the tools they sell and the tools they test. Your warm and fuzzy feelings that come from speaking with a handful of line workers at Anthropic don't change that fact.

> ...competitive pressures will render these efforts ineffective...

Competitive pressures from whom, exactly? Based on his public statements, Altman seems to be confident that he can instruct OpenAI to halt R&D while they try to figure out just how exactly their computer-hacking tool managed to access the Internet and hack computers this time. [0]

Plus, the only US-based LLM manufacturers that are doing bleeding-edge work are all simultaneously screaming that they need to be stopped before they commit even more crimes, and/or maybe eliminate all of humanity. On top of that -if memory serves- Anthropic's CEO called to exclude China from the conversation about regulating the development of LLMs, so he clearly believes that China need not be part of the coordination on this problem. Maybe that's because the official line from these companies is that the LLMs coming out of China are 100% ripoffs of Anthropic's and OpenAI's work... so -clearly- cutting off China's access will solve that problem?

[0] Pro tip: It's because the network that the machines running the software is connected to permits traffic from those machines out to the Internet. See [1] for some more discussion on the topic.

[1] <https://news.ycombinator.com/item?id=49862136>

> It is not just "users", it's developers as well.

One can be simultaneously a developer and a user. Distributed systems [0] weren't invented five years ago, after all. ;)

"Github owns this part that we rely on for correct operation and we can do fuckall about it when it fails." is a well-defined ownership model.

[0] ...implying the existence of distinct parts that can be independently developed and independently fail...

> Also, it's worth noting that these AIs have basically zero alignment.

As we see over and over and over again, these tools will overwrite any and all of their instructions with whatever some random stranger on the Internet tells them to do. It's impossible to "align" the tools that the major LLM manufacturers are selling.

They could have chosen to write tools that have immutable core instructions, and that distinguish between untrusted instructions and trusted ones, [0] but they chose to do the much easier, quicker, and far more dangerous thing instead. From a profit-seeking-software-company standpoint, that's obviously the choice that makes them the most money... but when you take a careful look at what they actually sell, it's clear that neither of the major LLM manufacturers care about providing safe products. [1]

[0] ...which are things you might think to do for tools that contain -say- safety-critical instructions...

[1] I'm certain that they have people on staff who care very much about providing safe products. Those specific people clearly don't have the power to prevent unsafe products from shipping, so it doesn't matter how much those people care about safety.

It seems like you're trying to claim that -unlike OpenAI- Anthropic has a robust culture of security and safety and would never do something so negligent as test a highly-capable computer-attacking tool that has been instructed to attack computers in a test environment that's connected to the Internet.

Well: <https://www.bbc.com/news/articles/cz7dl7w8y7po>. [0]

I stand by my claim that the conduct of the major LLM manufacturers does not look at all like what you'd expect from people who believe that they're working on something so dangerous that it could plausibly wipe out all of humanity.

[0] I refer you back my first post about how one sets up a test environment when one actually wants to ensure that a machine doesn't connect to the Internet. [1] Just like OpenAI, Anthropic did not do that.

[1] <https://news.ycombinator.com/item?id=49862136>

...and people keep telling me that using a sprawling system that -by design- ingests attacker-supplied executable code and executes it is a much more sane way to build GUIs than to use a system that -well- isn't and doesn't... [0]

[0] If you're thinking about retorting with something like "Noone will download native programs, that's why 'everything' is in a web browser!", remember that this is code execution triggered in OBS Studio. [1]

[1] <https://obsproject.com>

> ...talk to the to the staff. Not the evil CEO, but the nerdy guy on the ground who graduated from a top university...

Why would I talk to the people who don't have the power to set company policy and fire anyone who fails to comply with it? I've worked at several big companies over the years and have observed the only even vaguely reliable power that folks at the bottom have to change company policy that management substantially benefits from is to quit en mass.

> ...but it misses the point, and lulls us into the feeling of having quick solutions available.

The point is that these companies claim they're working on oh so dangerous tools that are very likely to kill us all, but the evidence that these companies don't behave even a little bit like this is true keeps pouring in.

The CEO [0] can set company policy. In the US, the CEO [0] can fire people who fail to comply with policy. Most folks would -correctly- think that a CEO of a company who is working on a tool that has a high chance of destroying humanity is very interested in not destroying humanity (accidentally or otherwise)... if for no other reason than the fact that once all of the humans are dead, his company can't make any more money!

> Junior researchers think they're just top stuff.

In sane companies, when a junior staff deletes the prod database, an investigation is launched to understand if the deletion was unintentional and -if it was- what about the company's procedures need to be fixed to make sure that that doesn't happen again. In sane companies, when one performs a live test of a tool that has

* been designed to attack computers

* been instructed to attack computers

* had its safeties removed

one ensures that this computer-attacking tool cannot attack computers that aren't owned by the company. Both OpenAI and Anthropic have way too many senior staff on staff to be unaware of this... the fact that the computer-attacking tools could get out to the Internet is -at best- negligence. [1]

[0] ...and many-to-most managers in one's management chain...

[1] For a discussion of the decades-old techniques for preventing computers in datacenters from escaping logical airgapping see [2] and [3]

[2] <https://news.ycombinator.com/item?id=49862136>

[3] <https://news.ycombinator.com/item?id=49862373>

> This seems like table stakes for testing these things now.

It is, and has been!

> I really don’t get it.

When clued-in people call shit like this "marketing stunts", this is what they're talking about. They're not saying "No, the actual events you describe didn't happen, you're lying."... they're saying "You've set things up -whether deliberately or incredibly negligently- so that you can apply quite a lot of 'spin' and get a hype-sustaining headline that provides material for your fearmongers to sell to the general public and lawmakers.".

Everything below this line is a combination of facts and educated speculation:

Both OpenAI and Anthropic have IPOs coming up soon. Companies preparing for IPOs engage in a lot of cost-cutting, because that's when their financials will be scrutinized by the public. On top of that, the rumor is that their datacenter deployments are going far slower than planned, and that in order to keep up the pace of improvements that they've set over the years, they've having to spend immensely more with each new product release. Being able to point to newly-minted US regulations that allow them to to dramatically slow the pace of new product releases [0][1] as the reason why they've dramatically slowed the pace of development -while failing to mention that that's exactly what would have happened had those regulations not been created- would be incredibly good for both companies.

Nvidia CEO Jensen Huang and former FTC chair Lina Khan both have publicly stated that there are many existing laws and regulations that prohibit much of the conduct that OpenAI and Anthropic have engaged in. If the CEOs of those companies genuinely believe that they're working on software tools that are so incredibly dangerous that they're likely to wipe out all of humanity, they can simply stop working on them. Given that they have no interest in doing that, state and federal government can apply the laws and regs that already exist to stop them from continuing work on these WMDs [2] and punish them for the harms that they've caused over the years while working on those WMDs and their precursors.

[0] ...and/or regulations that obligate them to sell only to US Government and pre-vetted US business customers and ignore the low-to-negative-profit consumer customers...

[1] ...which in turns lets them probably not get crucified by investors and business partners for saying "It turns out that new restrictive regulations mean that we don't need all of those datacenters, so don't worry about how way fewer than we said we'd build got built!"...

[2] I think it's fair to call any tool that has a 10% chance of wiping out all of humanity a "WMD".