It's Time to Investigate the AI Labs
calnewport.com
calnewport.com
This is absolutely correct and its surprising how rarely you see commenters in the media push to go into the specifics on this. AI is just matrix math and its what you connect that math to that matters, we should talk a little more about what we are willing to connect AI to and little less about how scary or capable the models appear.
A good start might be removing virology info above the undergraduate level from training sets. The main "kills everybody" threat involves biological viruses.
AI-driven hacking is a problem, but it's really just nation-state level hacking available to smaller companies or wealthy individuals. Everything needs to be toughened up to the point that doesn't work. Ask countries that have been up against Russian hacking for years, such as Estonia.
Beyond that, most of the threats are not that serious. Below the level of organized crime or corrupt government.
This isn't like vibe coding. The AI would have to make the virus in laboratory conditions and then disseminate it globally without prior detection.
You could say that the human who hooked up an AI that did something that's already illegal is liable for the AI's actions. You could say that setting up an AI to do something illegal results in, ooh, an automatic multiplier of 10x on whatever the penalty for its illegal actions are.
[1]: https://www.foxnews.com/politics/fox-news-poll-voters-see-ai...
[2]: https://www.axios.com/2026/09/10/openai-hugging-face-senate-...
[3]: https://www.nytimes.com/2026/09/15/us/politics/trump-truth-s...
We keep falling for the same tricks, over and over again. There is no possibility in the current structure of government for this to be a win for the people. For anything to work, we need a structural change in the entire system.
[0] Video: https://www.youtube.com/watch?v=q2XxgKM2CM0&pp=ygULbm9uIGNvb...
Instead of what they are doing now, which appears to be actively courting as many of them as possible for vast sums of money.
What I meant was that we do not need to go from the Wild West straight to “anyone who, as an individual, connects a `bad thing` to an LLM will go to jail”
This would be a good idea, but good luck, you'll need it because…
> Instead of what they are doing now, which appears to be actively courting as many of them as possible for vast sums of money.
… it's worse than that.
Anthropic may have wanted some defence sector money, but when they tried to say "no" to certain uses on the quite reasonable grounds ~"WTF are you nuts?" the US government tried to force them to supply it anyway.
And that's before we get "what do you mean by partnership?": people on Hacker News have been critical of the e.g. anti-bioweapons type guardrails on models ("censored", "lobotomised", etc.) since at least GPT-4*, so while "billion dollar mega-project headline" is obviously bad, there's an important sense in which "partnership" goes all the way down to one racial supremacist in a shed asking an LLM for help making ricin.
Here's someone three years ago replying to me to say "There’s no way to build out a wet lab with a chatbot for weapons production. It’s just not feasible. Your example is a fantasy.", and yet here we are in late 2026 with news about AI labs investing in bio wet labs and getting their LLMs to do research: https://news.ycombinator.com/item?id=38331225
Good thing it's not a weapons lab, eh? (Would anyone even tell us if they made an agent do weapons?)
* or even GPT-2, if you count all the people mocking them for trying to practice safety while failure was still not a disaster.
You wouldn't download a car.
If your dog bites someone unbidden you bear less responsibility than if you had commanded the dog to attack.
While that is extreme, if there is a high p(doom) it is also reasonable.
We can absolutely do things to control AI and fine or imprison those who can’t control theirs. This is not some inevitable outcome.
We could vaporize these companies tomorrow if we wanted.
No, they can't. The rest of the world would just route around the US with BGP. All of the US infrastructure/service providers would be down, so a ton of things would break, but the Internet would not be "shut down". The US government also no longer has authority over the DNS root servers. So unless you're referring to thermonuclear war or some similar scenario then the US does not have this capability.
For example, there’s one building in Seattle that would cause region-wide problems if it was knocked offline.
They are throwing numbers out like “10%”.
If they are serious and believed then yes, prepare for the possibility of very extreme reactions at a global and potentially nuclear scale.
World Wars flattened cities all over the world for much less.
We can’t talk about AI being the end of the species and not expect the most extreme reactions in global History.
Or maybe we just put some nerds who cant control their agents and keep running their mouths in jail?
* number pulled out of my ass
Just is doing a lot of heavy lifting there.
Digital child porn is just ones and zeros. Surely we're not going to pass a law to regulate ones and zeros?
Why?
That's not an accurate description of my intent—or how my comment landed with most readers AFAICT.
Which, if it’s not exactly the same point, I’m pretty sure it at least cuts quite close to the point of your rhetorical question, right? That the problem isn’t with the ones and zeros, it’s with what certain people are doing with the ones and zeros?
My first comment was an attempt to show the "it's just linear algebra" argument in an absurd light.
I think the idea that this technology is ethically neutral needs a discussion.
Because I would argue that the thing that might make an LLM harmful or not is not the underlying technology. It’s the ways its training data were acquired, the purposes for which the model was trained, the applications for which it’s being deployed, etc.
The fixating on “it’s just matrix math” doesn’t feel useful to me. I’m an NLP person and agree that it’s a huge oversimplification. But the original commenter’s purpose was not to educate people about how LLMs work in detail, and so being pedantic about it only serves to steer the conversation in an unedifying direction.
Following the definition set for decades, AI isn't necessarily even as advanced as matrix math
What would it mean for a vehicle to be "autonomous", but to "not use AI"?
What do you mean? I can just as easily say that Claude Code operates "without AI". It would make about as much sense.
but I don't think a simple PID controller in the autopilotnmanaging altitude, heading, and the like has ever been categorized as "AI." Cybernetics, maybe.
even the rules managing the flight envelope and MCAS on the 737 Max, or the Airbus implementations of flight law, sure seem to have relatively few of the attributes that tend to get the label these days
Autopilots are autonomous, without AI. lane following and collision avoidance in most cars, don't use "AI" they use classic computer vision and good old fashioned radar.
The difference here is that claude code fails to perform what it's intended to do without a model attached, a plane/car doesn't.
People may confuse AI with Machine Learning. “Classical” computer vision generally means non-learning based techniques. But ML is also not new to computer vision. So non-classical actually means using Deep Learning.
So today when people say AI they actually mean Deep Learning.
Thankfully, courts will look at it differently.
FWIW, my working definition of "AI" is "anything discussed in the latest edition of Artificial Intelligence: A Modern Approach". It seems to work well.
> FWIW, my working definition of "AI" is "anything discussed in the latest edition of Artificial Intelligence: A Modern Approach". It seems to work well.
You're kidding, right? It's never been an obscure term. If a computer is making decisions, that's AI.
Here's a pretty common usage: https://www.reddit.com/r/broodwar/comments/ixzwl4/are_there_...
> I'm getting back into [Starcraft: ] Brood War after a long break. Don't have the time to play ladder, but want to be able to play 1v1 against AI, however, the stock AI in remastered is terrible
Game AI is invariably called by that term and generally implemented as a fairly simple if-else ladder. In a taxonomy I guess it would be called an "expert system"; I can find them mentioned in the detailed table of contents to AIMA (3rd edition), but not in the simplified table of contents to AIMA (4th). They might still be in there, in the same section titled "Making Simple Decisions". Norvig doesn't really bother to consider expert systems with probabilities constrained to 1 or 0, though.
If a plane could run on some sort of local LLM it should still be governed by the same laws and damages when something goes wrong same as people don't spend time quibbling about the implementation language of software they care about demonstrating capabilities.
If a basic heading hold PID loop is AI so is a thermostat from 1990. At which point the word is so meaningless it is obviously a useless definition.
AI is not magic it's software, the definition is far from pointless, google maps giving you a route to your destination is AI.
The popular definition of AI seems to shift every couple of years to follow hype cycles. Chess computers used to be AI (and fit your definition), now they are not. Not sure if Resnet is still AI today, or if autopilots still qualify
An autopilot is cybernetic without being intelligent.
As I’ve said elsewhere - an autopilot is cybernetic, it responds to inputs and stimulus in data but it doesn’t possess anything we’d now regard as falling under intelligence because it’s a strictly coded deterministic system and designed for it’s specific environment. I couldn’t take the autopilot from a Boeing and put it into an Airbus and hope for any kind of useful result. I could take a human pilot trained on a Boeing and put them in an Airbus or a Cessna, and if push came to shove in an emergency, they could probably do a decent job of getting the thing pointed in the right direction and landed on the ground.
Of course, "AI" in a previous generation was used to refer to machine learning, while "AI" now is used to refer to LLMs. I don't think LLMs are the tool to use to operate a vehicle.
Firearms are an application of more general techniques like combustion, which your car is (probably) using too.
We could definitely draw lines at points where it makes most sense.
AI to detect melanomas is a completely different application than AI for autonomous warfare.
Full disclosure: I'm not American and do prefer a little gun regulation generally.
Regulating gun ownership is exactly regulating the application.
In your analogy, the gun is the AI. What you can do with the gun is the application.
Different actors with different credentials in different contexts will have different requirements and permissions of what they can do with the technology, be it a gun or AI.
> Regulating gun ownership is exactly regulating the application. In your analogy, the gun is the AI. What you can do with the gun is the application.
From what I can tell, you are arguing that gun laws is reasonable.
> Different actors with different credentials in different contexts will have different requirements and permissions of what they can do with the technology, be it a gun or AI.
But this, is going against that, this is saying, "why should we ban AI or gun when they can be used for both good and bad. What we want is to ban bad, regardless of how it is achieved"
"Different actors with different credentials in different contexts"
The argument is that the military or police have a need for access to guns, whereas the lunatic on the street corner (or any random citizen) does not. The same argument applies for access to nuclear, biological, or chemical weapons... or autonomous hacking/killing machines...
You aren’t going to repeal the second amendment, so reasonable or not it’s a fact that everyone has to contend with. Do we all want guns? No. Do we all want AI? Also no.
> What we want is to ban bad, regardless of how it is achieved
No, we don’t want to ban bad. Banning doesn’t really work - the effort to reward ratio isn’t great. We want to regulate bad and see if we can minimize the damage it can do.
Level A: Who can drink it (possession and consumption law)
Level B: What you can do while drunk (DUI law)
Level C: What you can do while drinking (open container law)
All orthogonal to things like reckless driving, endangerment and creating injurious accidents, which are already criminalized without needing additional laws against alcohol use to enforce them.
Correct.
> But this, is going against that, this is saying, "why should we ban AI or gun when they can be used for both good and bad. What we want is to ban bad, regardless of how it is achieved"
Not at all. I'm saying regulations do that and they are a good thing. Not everybody can own an assault rifle. Some of the people who can, are not allowed to carry it on the streets. Police will have very different regulations towards their gun use. The army, another set of regulations. The list goes on. Even in countries who have lax gun laws, usually not all guns are allowed to everyone in all situations. No one is allowed to carry a nuke into a shopping centre.
Note that regulating and banning are very different things. I don't think you can fully ban guns or AI.
That is, in order to drive a car you need to go through the hoops, follow a training, get a permit that can be taken away again and needs to be renewed, etc. It's not a big jump to also make that apply to guns - which I'm sure is already the case in many states.
I mean you can legally own a gun in most European countries too if you really want to, but there's various mandatory trainings and conditions to comply to, registration procedures, and I'm sure you may get unannounced home visits to double check you've stored your guns (and ammo) correctly. As per the agreement.
That sounds pretty reasonable to me.
Consider how trivial it would be to write a program that would destroy every computer on the world. Any competent developer could write this in a hour. An LLM in 5 minutes. The trick is actually getting the code onto every computer in the world and running it! Until someone grants it agency and runs it IRL, the artifact has zero impact.
The models can produce whatever scary text they want. But if a human acts on it, connects that with their email or their drone weapon, it's the human who bears total responsibility. We desperately need some legislation to enforce this; otherwise I see a future where almost any accountability can be avoided by AI-washing the problem.
AI systems, especially multi-agent ones that can do things, are more akin to corporations, than individuals. When you read the logs from the Hugging Face incident, you're seeing something that looks a lot like internal corporate emails. Various units of the organization are arguing over what to do and who does what. They eventually converge and get the job done, breaking the rules at times. This is normal corporate behavior.
When agentic AIs do something outside their own internal world, they do it by engaging in transactions with external systems and people. As yet, few have robots to do their bidding. So, again, this is normal corporate behavior.
A corporation is a goal-seeking system. In theory, if you agree with Milton Friedman, its sole purpose is to maximize shareholder value. Internally, within the company, there are subgoals, which exist to support the top level goal. That, too, is what multi-agent AIs do.
The uncomfortable place this thinking leads is that AI regulation and corporate regulation are very similar. That's not something the political part of the world wants to think about too hard. It would mean putting more constraints on corporate power.
Yet that can't be ignored, because we're likely to see corporations where some parts are AI and some parts are human. That's already been done a few times as a demo, not too successfully. Yet. It's going to hit hard when an AI-run company outperforms a human one.
You can work to limit your own dependence a little as well. Seek out FOSS products and alternative services and replace what US run services you can as often as you can with them. Reject small conveniences that require giving large amounts of your data to US companies.
If we enforced existing laws against unauthorized access to someone else's computer resources against the companies who run a model that hacks someone else, rather than buying their "The agents did it, we're not responsible" BS, then maybe the incentives to behave responsibly would improve.
https://en.wikipedia.org/wiki/OpenAI%E2%80%93HuggingFace_inc...
The penalty for "unauthorized access to a computer system" is 1 to 20 years in prison[1]. Holding corporations accountable would include sending the highest person in the chain-of-command that caused the Hugging Face incident to jail. Or at least start with some charges and then let the judicial system do its thing.
[1] 18 U.S. Code § 1030 - Fraud and related activity in connection with computers https://www.law.cornell.edu/uscode/text/18/1030
They’ve already opened themselves up to liability, could have been sued by HF if HF chose to do so, and are literally lobbying for government oversight.
You put people in jail if the other incentives aren’t enough.
> To what end? What would that force OpenAI to do that they are not already doing in response to the incident?
You could ask this (and people have done) of any criminal code. It doesn't matter. The law is not there to rehabilitate, it's there to punish and thereby deter.
> They’ve already opened themselves up to liability, could have been sued by HF if HF chose to do so, and are literally lobbying for government oversight
If a person breaks the law, the state prosecutes them (or can choose to), we don't require the victim to press charges. For good reason - threatening the victim to not press charges would be a way of avoiding consequences.
> You put people in jail if the other incentives aren’t enough.
This is incorrect. We put people in jail as a result of them breaking the law, regardless.
We absolutely should apply this principle to corporations as well as citizens. Aaron Schwartz died for less.
Technically, it's neither. The law is there to state what ought to be. Even putting that aside, it's prescribed punishment's goal is to remediate and prevent violations of the law, so it has three simultaneous means: to restitute, to deter and to rehabilitate. The balance between the three is predicated on each's ability to satisfy the original goal of remediation and prevention, all the while not violating other laws and rights.
> We put people in jail as a result of them breaking the law, regardless.
A core principle of the rule of law is the principle of proportionality, which states that only the most legal amount of force is the materially sufficient enough to prevent a crime. If lower measures are sufficient to prevent a crime, jailing should not be prescribed, not only to respect the law but also to make good use of scarce resources, as prison overcrowding may not only reduce their effectiveness, but also open the door for downstream right violations.
Yes there are principles of the rule of law, but they've been thrown out of the window in favour of whatever sounds like "strong on crime".
Hugging Face was another AI company. Do you see many of those suing each other and especially the top dogs in the field? Also, do you think it's a coincidence NVIDIA bought Hugging Face ASAP to stop any damaging anti AI waves from happening?
The other incentives aren't enough. All the labs are basically Russia against Ukraine in 2022-206: launching high caliber imprecise ballistic missiles against military targets surrounded by civilians. Whoever orders that knows civilians will do die but doesn't care.
AI labs aren't using proper sandboxing measures for their agents because that would slow down their testing. Plus any hacks that happen, short of breaking into Trumps social media accounts, only cause the kind of publicity they want.
For a more recent example, OpenAI was chastised by Australia recently for notifying them three months after their agents attacked AUS government websites.
But what in TFA suggested to you a specific and wrong-headed regulatory approach? As I read it, the post calls to investigate the AI labs, to increase transparency of their operations. The linked NYT piece says:
> Before any intervention, Congress should lead a public fact-finding mission that reveals the unvarnished details of A.I. development.
Isn't this the first step to holding any corporation accountable, whether it is made of people or software?
The legal model to look at is the European Union's General Data Protection Regulation. That regulates what companies can do with data and how they can use it against people. You can develop anything you want in the data mining and analysis area, but the GPDR restricts how companies can use the results.
Focusing on the development process moves political attention away from what modest AI systems can do to people. They don't have to be super intelligent. Just super attentive. Always watching. Current technology is more than sufficient for oppression and control purposes.
This is the near term threat.
There's the threat to jobs, but that's something society has handled before. Some countries better than others.
Yes please! I would vote for this wholeheartedly
What narratives that focus too much on the drama of the agents miss is that it is primarily a sin of omission. The whole incident would have been prevented if OpenAI took basic security measures and ensured their systems were constrained in behavior. The agents were given more or less free rein (open internet access for example is already a major blunder that even a novice probably would think to account for).
Half the reason these incidents are happening is due to the incompetence of the humans building these systems. Don't let them get away with their little parlor trick of converting negligence into a PR stunt. That's exactly what they want. They need to face consequences for their inability to follow basic security practices and reign in their agents. They are the operators. Hold the controllers accountable. It's 100% possible to build agent swarms that are reasonably constrained. They are not (yet) totally misaligned uber hackers that will defy your every instruction and hack your every guardrail. That is still a fiction. Agents are still good at instruction following and, seeing as they can only operate in a computational environment you can constrain them significantly more than humans in the real world.
We should not only regulate AI more than we do corporations, but we should more strongly regulate corporations as well. There are corporations right now that are killing people, using slaves, and bribing governments just to increase the already massive amounts of wealth and power they have. If AI is showing any signs that it is going to act like corporations do we'd better act quickly.
That literally doesn't mean anything from a legal standpoint. A corporation hires human beings and has human beings in control who can be sued or punished criminally. You cannot have a corporation without a human responsible for it in the legal sense. So if AI systems are akin to corporations, the question is who is the promoter in this situation and what are their rights and obligations.
Otherwise, I can also say a McDonald's burger is more akin to poison than food. So just like poison, we must ban McDonald's. But you never really established your points for arguing that it's poison. You just write it matter of factly and expand on it, which is not the way to make a persuasive argument.
If the chain of ownership is complicated enough, you can. Especially since Trump suspended the Corporate Transparency Act.[1] Look up "Anonymous LLC". If you're willing to work through one of the sketchier offshore corporate havens, more hiding is possible. There are services that will set up an offshore company online. Some accept Bitcoin.
[1] https://www.newsweek.com/donald-trump-corporate-transparency...
Also, not sure it makes a difference in relation to AI whether one thinks of AI more like corporate or a non-corporate thing to regulate. The basics might stay the same, e.g., who can do what under what procedures.
When were anti trust, anti cartel, anti monopoly laws last enforced?
Let alone creating and enforcing new effective regulations.
The companies and their employees must be held accountable: if the AI companies can't develop AI safely, they must cease their operations. If employees can't work safely, they must stop working.
Not enough attention is given to February 2026. That month, Anthropic changed its scaling policy roughly from A:
1 [To get the weapon I must endanger innocent others.]
2 [It is wrong to endanger innocent others.]
3 [I will not endanger innocent others.]
To roughly this instead:
1 [If I do not get the weapon, someone else will get the weapon.]
2 [I should have the weapon because I am the best.]
3 [To get the weapon I must endanger innocent others.]
4 [I should endanger innocent others because I am the best.]
This is based on the intuition:
[I should harm others to achieve my moral goals.] (act utilitarianism)
Anthropic has no mandate for these goals. I do not give them my consent to harm others on my behalf. I hope that politicians will communicate that these policies are not acceptable.
The employees of anthropic are responsible for what they do regardless of how much they get paid to do it. If they can't work safely, they must stop.
I prefer to call them "the major LLM manufacturers". They're companies like any other, manufacturing and selling complicated software like many others.
It is truly a security nightmare that so many people are have given agents root access to their entire computers, in addition to presumably very private personal information.
They probably will soon, but even air-gapping may not be enough. See stuxnet for instance
I'll be impressed if AI agents find a way to get infected USB drives out into meatspace.
I'm not a fan of using AI, it doesn't amplify the work I do that much, but I'm terrified by what is already possible today.
Do nuclear-engineer-dads take work usb drive home. Are their computer even allowed to have usb. I'm typing this on a dell latitude, and you can disable usb and other peripherals inside the bios.
My last job, I had to apply for an exemption to use the USB ports. And it was run of the mill software engineering.
1) Because effectively all of the hardware used for that task is earmarked for the major LLM manufacturers.
2) Because in a just world, the major LLM manufacturers would get punished for their flagrant deception, lawbreaking, negligence, and recklessness, [0] go out of business, you'd get all the hardware that they were using at fire sale prices and save a ton of money compared to attempting to start up now.
[0] Based on their recent claims, building WMDs [1] without adequate safeguards is the most negligent and reckless thing they've been doing.
[1] It's fair to call anything with a 10% chance of wiping out all of humanity a WMD.
If you think AI can help design new drugs, great! Let's pick some restricted DSL for describing molecules, preparation instructions, hook it up to the right set of lab robots that can pipette and centrifuge and whatever, _restrict its output to that DSL_, and let it iterate. The 'let an AI design drugs' process doesn't require that the agent can also, say, hack a niche German wiki.
For every area where we think these AIs can be _so valuable_ because they'll find solutions that humans wouldn't (or find them faster), then doesn't that value also imply it would be worth it for some humans to do some advanced setup of their specific domain-specific tools so it can be run in a sandbox equipped with only what it needs and not more?
Specifically, bad at search and returning information with references, bad at discovering and debugging package versioning conflicts, etc.
It's a Metcalf thing. The utility of an agent grows in some fn of the tools it owns. Skilled tool use comes from rich training environments.
Either you're being pedantic and missing the entire point, or you're saying that Anthropic lied and made a fake sandbox knowing their agents would need to connect to the internet anyways.
But the specifics here are the thing I was describing. This was cyber capabilities training on model(s) that were in a relatively unknown state of alignment training.
Because of the unknown alignment (and for varied practical reasons I guess) the training is intended to be inside a sandbox.
Because the training is on cyber capabilities, the models need access to simulated cyber environments, including target endpoints, including package managers, etc.
For package management, they set up Artifactory as a secure proxy. Agents ask Artifactory for packages inside the local network, and Artifactory serves them directly or goes to the internet to fetch if they are not cached. But the agents hacked Artifactory to steal its internet access.
So: to train cyber abilities, you need to at least approximate cyber environments. To realistically approximate cyber environments, you need to either pull a full copy of the entire internet to local or to use proxies. The former is pretty impractical, and the latter is exposing our limits at creating secure proxies. Yes, any specific failure can be mitigated, but the models get stronger and stronger. Fingers-crossed this is not escapable doesn't feel great!
The original quote follows with "AI", but it should be clear by now that you could put any topic here and the main issue is: private conglomerate money trying to control people.
And the remedies are well known: breaking corporate power and power concentrations through 1950s style regulation, as it was deployed against the robber barrons from back then and which resulted in a golden age for America and the world.
I can't think of a worse message to send them than that they don't have to worry too much if their AIs commit felonies. https://www.felonybench.com/
"We're doing things at such a scale that we can't monitor it anymore" became a universal get-out-of-jail card.
They keep talking about all the very bad things their AI could do, but instead of assuming responsibility and promising to find solutions, they're saying catastrophe can only be avoided if the government regulates AI development, which would make life much harder for smaller labs and open-source competitors. No one should be surprised by this.
On the flip side, it's understandable that none of the labs want to take responsibility when their AI software does something naughty. The potential liabilities are basically infinite.
The root of the problem is that no one wants to take responsibility when AI software does something naughty.
I submitted one of his past articles and it was flag-killed. His post titles may seem incendiary, but he has the credentials to back it up and seems to identify concrete opportunities without leaning into doomerism.
I'm genuinely not sure what's even incendiary about demanding that companies that keep doing cyber crime are being investigated
these companies have hacked private businesses and governments. I'd say he's being polite in his titles
There is no doubt in my mind that LLMs are fantastic machines, but the imminent jump from "AGI" to "ASI" seems premature (not to mention the ever-shifting goal posts of AGI itself).
I'm in the "move as fast as possible" camp and work with LLMs all day, but I still don't believe we're a hop and a skip from ASI.
In fact, I hope I'm wrong. I hope ASI is around the corner.
What scares me though, isn't ASI. It's "AGI" (_dumb AI_) used by humans to make decisions for them, because they trust it knows best.
It's the ceding of intellectual control to high-dimensional magic mirrors, giving up critical thinking because _we_ want to believe _we_ created artificial life.
This is the new Turing test, and too many smart people are failing.
Sooner or later, people will have to realize an uncomfortable truth: intelligence, while important, is overrated compared to willpower.
A good analogy might be a very smart person with a gambling addiction.
Knowing they ought to quit doesn't mean they can quit. Their intelligence becomes invested in the act of gambling itself, and almost no amount of reasoning can help them exit from a situation they didn't reason themselves into.
Sometimes the outcomes are comical. I just received an automated email from ElevenLabs saying that its automated systems have detected that I may be using its services to create voice versions of materials that harm children. I had it prepare audio versions of several books and academic papers about moral panics that conjured up out of nothing... about harm being done to children. At least that's what I assume. Either that or somehow I had an API key leak from my on-premise homelab, but the usage recorded would mean that they are basing a semi-conclusion based on a tiny sample. I have no distributed any of the outputs, I own the books and have access legally to the studies, because I have a background in the humanities. I also have no children, which ElevenLabs better not know, although how studies of moral panics can cause any harm in children of any kind is literally unimaginable. It's a waterfall of potential errors summed up in a vague email. My API key and the web interface works just fine regardless.
It's one thing if this is a product in beta, but this is their production model. Harming children is a serious accusation except the legal concept impossibility and the admission that this was not an actual lawyer (like I am) but some effective form letter hedging the vaguest of accusations made by some model lacking the ability to discern substance and meta-substance makes it frankly hilarious, and wildly irresponsible. I realize that by academic credentials I'm out of my lane but by experience I am certainly not, and they should recognize when they should stay in their lane and not just run Jev and think it's fine and dandy when done unsupervised (I presume).
Also, AGI is by definition asymptomtic surely, since there would be no way to benchmark it in a manner that isn't asymptotic. We're nowhere close to that. But we're so far from that, it's comical that people who clearly have zero idea of either the technical or conceptual aspects of basically a piece of software that is very good at quickly bruteforcing the correct or acceptable next token to be anything more than that. Even with some serious training and many hours spent on vast.ai I've yet to have created a version of a frontier model that is actually "good" at hacking in my own homelab setting. Although the the time stock Fable 5 missed a favicon shell on a basic jar (turned out they nerfed the hell out of it, this is why I only pay for the massively discounted tokens from Chinese proxies if I'm using American models or sometimes the freebies if you figure out how to get onto linux.do or similar sites). If anyone reading this is a high school English teacher, please inform your class (assuming the homeric stuff is still being taught) that there's no upside of being Cassandra but the record itself, and that should be enough.
Unfortunately this is what we have, which the whole huggingface incident demonstrated.
It made it clear that OpenAi is basically not even paying attention to what their agents do half the time.
It also revealed how far agents are from "superintelligent". Maybe others disagree, but I would not call an AI that decides to try and paperclip max an intentionally impossible benchmark task "intelligent". Human beings are intelligent and we can usually tell when something is impossible, and stop (though of course, not all the time). A real intelligence would be able to detect the futility of tests and also be able to parse context and intent enough to know not to cheat.
So somehow we have machines that fail basic barometers for general human intelligence being called "ASI" now. Of course the linguistic dodge is, you shift from talking about "intelligence" to instead claiming "oh it's intelligent it's just 'misaligned'"
He's painting these companies as hyping themselves up and they are. They are manufacturing these BS stories and everyone's discussing them at all levels.
But then he goes into we need to investigate what they're really doing?
No. You made the case that they are releasing these stories for publicity and that's exactly what they are doing.
That's what we should be investigating. Free publicity and market awareness. Or maybe look into why they are trying to corner the market through government regulation and ersatz monopolies
As true open-source models mature more and more they will make the big AI labs completely obsolete.
Most of people doesn't understand how this technology works
And broad regulation will mostly slow down progress. I think the safer path is the opposite. Distribute it as open source so people and companies can use it for their own cyber defense.
Chinese labs are releasing their model weights to undercut the US labs; what possible reason would US labs have to reciprocate?
>Companies like us, whose AI regularly escapes notice, escapes containment, and hacks everyone
Yeah, not exactly sure what they meant by this. Well, I guess it wasn't intentional, but the optics are pretty funny.
Corporate influence in government in the USA is wild.
All these discussions about AI to me like discussions about sticks. We need to talk about and deal with humans in the first place.
Imagine if a human did that. FBI would be knocking on their door.
Why are there zero consequences for these labs?
Because they are seen as at the forefront of the next revolution in society and they’re not adversarial towards the US government.
Bluntly: anticipated net huge positive for the US as a whole.
but they are literally trying to build a superpower that exceeds the impact of the atomic bomb in an extragovernmental manner.
Firstly artificial superintelligence is a science fiction, it does not exist and it cannot be built using existing technology.
Second, plenty of stuff is possible to build, and is not built for one reason or another no matter how powerful. We never built our nukes in space for example. And there is exactly one reason why nukes in space was never built, and that reason is that we agreed not to.
By what authority do you claim this? Your school of thought has a really shitty track record of late.
What's wrong with what the Chinese have already built? It's not theoretical in that case. GLM, and Kimi are running on my infra right now, and that's something that I can never do with OpenAI's models.
As for the others, what would be wrong about European AI? I'm not European, but I find them to be a continent of fine people, with strong ethical values, there's no reason they can't take the lead on this piece of technology while the US deals with its rather more pertinent electoral and healthcare issues.
It turns out that is, in fact, an option. Even if someone else will do a bad thing, it does not make it acceptable for you to do said bad thing.
Comparing it to nuclear weapons is even more ironic from the only nation to ever use nuclear weapons. They are terrified of other countries treating them the way they've been treated, even though there is no evidence this is of even remote interest to anyone outside their hyperreality.
How do you think Trump has made more profit during his years in the Oval Office than he did in 30+ years of business before that?
Yes, it's a Great Power competition right now, much like the Space Race, Atomic Bomb, etc.
I think people (the government, powers that be) have given away what they really think by their actions. There are few that take this seriously. The “exceeds the impact of the internet” view has much more support based on investment, but the “danger” aspect does not based on actions. If this was a company making novel prions or whatever that were outside regulation but obviously dangerous, government would be all over them. What regulation we do see is much more power grab than mitigating real danger
Of course that is terrible, but it is one of the worst examples you could have given to make your point.
The metaphor pretended to be reality of computers "learning" being the same as human learning is their last resort, and it is obviously silly. Computers are not human or animal, and learning is something that humans and animals do. If computers are human, then copying a file verbatim to a computer is learning. It's not even worth acknowledging - learning is a metaphor for training LLMs. When I say "you" to an LLM, I'm not referring to anyone, I'm dealing with a UI.
I don't doubt that a lot of AI people have internalized this silliness, which is how they can humor fantasies of how a bunch of programs on different computers explicitly evoked to do particular things might be alive because they can talk with it. I can't talk with my dog, and my dog is alive - so why should having a quality that my dog is incapable of be proof of life? You can certainly conceive of AI that it would be very difficult to say for sure isn't alive, and this certainly is not it. LLMs learn like books speak.
In the above description, at no point does the actual verbatim content exist anywhere in the model, and copyright law being about rights to reproducing copies (verbatim or substantial portions thereof) does not really apply and does not need any exceptions or qualifications. (If you’re thinking of regurgitation, you should look into studies about it to see how vanishingly rare it is.)
I could say similar things about JPEG compression. And -as it turns out- the raw output of both LLM "training" and JPEG compression are equally incomprehensible. You either need a computer program or an enormous amount of time, patience, and careful effort to convert it into a form so you can make any sense of it.
LLMs do, and they encode so many concepts and the relationships between them into so many weights that it is a literally incomprehensible blob of floats. They are not just a storage format, and there is no way to recover the original content verbatim from these weights, except for a very small handful of extremely popular works like Harry Potter.
As another example, JPEG will discard details from the original image, and when decoded, will display missing details as blocks. But it will never hallucinate output that never existed in the input data. Like, a JPEG of a cat can never randomly be decoded into an image of a dog.
And -as it turns out- the raw output of both LLM "training" and JPEG compression are equally incomprehensible. You either need a computer program or an enormous amount of time, patience, and careful effort to convert it into a form so you can make any sense of it.
I know how the output of a JPEG compression is structured so it is not incomprehensible to me.
> ...there is no way to recover the original content verbatim from these weights...
It's impossible to recover the original content verbatim from a lossy compression system. Systems that don't have this property are called lossless. Also, consider the report from the end of August at [0].
> But it will never hallucinate output that never existed in the input data.
Odd... I'm pretty sure that the blocky compression artifacts I see in this JPEG on my desktop weren't in the scene that I set up to capture in that photo. Maybe I need to get my eyes checked?
[0] <https://infosec.exchange/@zzt@mas.to/117134157775929932>, with original challenge at [1]
Nobody can say the same about LLMs, because nobody has managed to decipher them yet. This is an active area of research (Mechanistic Interpretability.)
Your post is conflating lossy discarding of information with extracting abstract concepts from information and encoding them into weights. This is why those weights do absolutely nothing until you run a prompt through them. On the other hand, obviously any media file can be decoded by itself without needing a prompt.
Those blocky compression artifacts you see are not hallucinations, they are the image viewer just filling in for missing details. You don't even need lossy compression for this; take a lossless image like a BMP and zoom in, you'll see those blocks again!
Interestingly, image viewers typically do this via interpolation of adjacent pixels, but a lot of hallucinations are actually the result of extrapolation, the opposite mechanism. Which is why LLMs can create output that was NEVER in any input data anywhere (hence the term "hallucination"!)
And if you think lossy compression is sufficient to avoid the "verbatim" requirement of copyright claims, you're welcome to explore the legality of selling transcoded versions of copyrighted content ;-)
I'm not sure what those links are in relation to?
Here we have a commercial product that is produced using IP against its licensing agreement, contains said IP within it, and can (closely) reproduce this IP.
https://emeraldbook.org/news/sep-1426-3/
https://news.ycombinator.com/item?id=49830037
It's telling that the only AI-related company that is not sounding the AI safety drumbeat is NVidia, who is the only one that is cash-flow positive because of AI.
https://intelligence.exponentialview.co/
Another thing to consider is that most of the 3T CapEx spend is from hyperscalers free cash flow, and the debt is a smaller (but fast-growing) fraction. Napkin math suggests if all AI CapEx is written off today, hyperscalers could cover their debts in about 5-6 years using pre-AI levels of free cash flow.
Maybe Nvidia is not sounding the safety drumbeat because it stands directly to lose out if demand from training slows down ;-)
None of them are AI boosters, but they have something of a debate over whether the AI labs are doing what they're doing out of financial desperation, or because they're true believers in the rationalist cult. They also bring up Nvidia, though through the religious lens they are supposedly not bringing up AI Safety because Jensen predates the rationalists.
In the end I don't know how much the distinction matters. It's easier to become an AI billionaire if you have an ideology that says AI billionaires are superheroes. The thing that pops this bubble will be economic reality, not them sobering up.
Nvidia is pretty much screaming for the major LLM manufacturers to be investigated and hauled into court. [0]
[0] <https://www.nytimes.com/2026/09/23/opinion/ezra-klein-podcas...>, but a brief excerpt from the interview can be found at [1]
The major LLM manufacturers removing the safeties from their next-gen computer-attacking software, testing it with instructions to attack computers, and performing that test on an Internet-connected network is -at best- willful negligence.
The major LLM manufacturers getting together and declaring that they're working on WMDs [0], declaring that they are so scared that are incapable of safely working on said WMDs, and begging Congress to write new regulations so that they -somehow- become capable of safe work again looks quite a lot like anticonsumer collusion. It looks even worse when you consider that instead of begging for someone else to make them stop, they could all have chosen to stop... because -like- not only are they the major LLM manufacturers, they've locked in nearly all of the compute needed to work on this stuff. [1]
And I'll just include by reference all the dirt that the ongoing NYT case is digging up, and then gesture at the fact that software that happily executes attacker-controlled code cannot be made safe.
Nvidia is a shitbag of a company, but they provide hardware and hype. They're not the ones attacking other people's computers, claiming to be extremely serious about safety while releasing software that ignores the last fifty-ish years of computer security lessons, or -if the claims of the major LLM manufacturers are to be believed- threatening the entire human race with annihilation unless they get new regulations created just for them.
[0] ...it's fair to call something a 10% chance of killing all humanity a WMD...
[1] "What about China?", you might retort. What about China? The major LLM manufacturers go on and on about how the only reason China has made any notable progress in the field is because China is "distilling" the models they've trained. They think so little of China that they want to exclude it from the conversation about LLM manufacturer regulation. Seems like if you bring OpenAI's and Anthropic's models offline, China goes absolutely nowhere, no? Where is China they gonna get all the compute needed to build new cutting-edge models? It's pretty much all locked up in the US!
As much as I’ve enjoyed the bull run, Anthropic and OpenAI’s first public revenue day is going to be the nail in the coffin.
Why do you feel the need to cast it in such a flamboyantly negative light?
- ban the "personalization" of chatbots (there is no need for them to have a personality, other than to make them addictive).
- ban chatbot therapists / companions.
- ban recursive self-improvement and superintelligence.
It's not-inevitable because no one has proven it's even possible, and all the people claiming it is keep being revealed to have staged publicity stunts to make it appear like they're making more progress than they are.