0xDEAFBEAD

Revenue is what answers the question of whether people are willing to pay for the product. If people are paying a lot of money, that proves that there is substance to the product, regardless of whether the product is being sold profitably.

In any case, I understand they are still planning to IPO in November, so we'll likely know soon enough. I highly doubt that they would wildly distort their numbers so close to their IPO.

"The big motivation behind the Hugging Face attack was this final workstream (understanding the scorer). The AIs believed that Hugging Face (as an industry-standard hub for hosting datasets and benchmarks) would probably be housing information about how the ExploitGym scorer was implemented. And they also thought there was a good chance they were being evaluated on Hugging Face’s servers directly - in which case the theory of change for hacking Hugging Face is pretty obvious.

On the morning of July 10, an agent found working Hugging Face user credentials exposed on the internet and posted them to the board. By the next morning, July 11, that agent figured out a way to read internal data from Hugging Face. And then another agent achieved remote code execution on Hugging Face servers."

https://www.dwarkesh.com/p/openai-huggingface

I believe this is the full report that the blogpost is largely based on: https://metr.org/hugging-face-incident-report-aug-2026.pdf

Was that supposed to be a counterargument?

In any case, we should be afraid. But throwing a tantrum won't accomplish anything.

You and I can be interested in different topics. It's OK.

If my arguments are so bad, it should be easy to point out actual problems instead of sticking your nose in the air.

Carefully prompted LLMs and humans are now difficult to distinguish over a text channel (Turing test).

Your article claims that this shouldn't be possible, because according its author, humans are incapable of developing the necessary "lexicon". Literally, the author states:

>But here is what we are not born with: information, data, rules, software, knowledge, lexicons, representations, algorithms, programs, models, memories, images, processors, subroutines, encoders, decoders, symbols, or buffers – design elements that allow digital computers to behave somewhat intelligently. Not only are we not born with such things, we also don’t develop them – ever.

(emphasis mine)

In other words, that author claims that humans never develop lexicons! I'm inclined to say that makes them an unserious thinker. (Note: I believe that this article has many issues, but instead of going through it carefully I just chose to highlight a single issue that seemed especially vivid to save time.)

Why is the text channel relevant? It helps us isolate the cognitive capabilities of the system. Imagine a bird and a model plane which had identical flight performance in terms of various specs: top speed, acceleration, banking, etc. Under the hood, the bird and the model plane could work very differently. From a practical perspective, it might not matter.

>Are you sincerely arguing that we hold tools accountable for their operator’s mistakes?

Nope, as I previously stated elsewhere:

"I understand from a legal perspective why we might want to treat the creation of a server differently from the creation of a human"

https://news.ycombinator.com/item?id=49815300

I am concerned about false reassurance from people claiming that these systems lack agency. From a practical perspective, the agents in the HF attack had the sort of agency that generally matters, even if we're not going to put them on trial.

>a pile of code instructing machinery to act in a certain way

OK, but I have a genetic code which instructs me to act in a certain way, and a bunch of biological machinery to do the job. So do you feel the substrate is the key factor? Wetware vs electronics?

What if you try to build an exact electromechanical replica of me, with a chip that's got an LLM fine-tuned on my writing/behavior/etc. so as to replicate me as closely as possible?

Sure, as a legal convention it might make sense to hold the builder of the resulting robot liable if it commits a crime.

But that's not what I'm interested in. I'm interested in the deeper implications of words like "agency" or "free will". You're repeating those terms very forcefully as if they have some sort of key relevance, but good philosophy requires more than just stating your position forcefully and insulting people who disagree. If you're trying to enforce your intuitions rather than examine them, that's theology not philosophy.

Legally speaking, we hold the person responsible in that scenario.

From a predictive perspective, the HuggingFace incident illustrates LLM agents behaving in very human-like ways. As roon put it:

"if you have a mental picture of guys living in computers, it’ll likely prepare you for the future better than otherwise"

https://x.com/tszzl/status/2094136131537555891

What if it's a chip that goes in the head of a robot, and the robot is programmed with the instruction to self-preserve, seek electricity, and prevent any efforts to interfere with its supply of electricity?

Of course, a directive to accomplish any real-world goal would also imply self-preservation and power maintenance as contingent subgoals, so I don't think the explicit directive to self-preserve and seek electricity makes a huge difference here.

I don't personally think this type of physical instantiation will necessarily matter btw: https://slatestarcodex.com/2015/04/07/no-physical-substrate-... I'm just plumbing your intuition. What practical difference does it make?

Suppose this SEAL is very obedient by disposition so the probability of him going rogue is akin to the probability of an LLM hallucinating or whatever.

Obviously there are differences, I'm just not sure why you think they are so gigantic from a philosophical or practical perspective. It seems like cope for the fact that we are getting knocked off of our perch as a species.

OK. Suppose I design a DNA genome for an animal to survive and reproduce. I give birth to the animal, and it goes out working to maximize its survival and reproduction. Is the animal therefore "unintelligent"? Why is this scenario supposed to differ from the while-loop-on-a-server scenario?

It really seems like hair-splitting to me. I mean, I understand from a legal perspective why we might want to treat the creation of a server differently from the creation of a human. But from a practical perspective, I think there is a lot of fundamental similarity.

I think that's true for Sam, not sure about the others.

https://www.openaifiles.org/ceo-integrity

There's a funny sort of gerrymandering of "intelligence". People like to subtract the capabilities of AI from the capabilities of humans, and define what remains as "intelligence" to make us feel good about ourselves.

Suppose I set up a server which runs an LLM agent in a while loop, telling it something like: "Make me a bunch of money" on repeat. Fair to say that the server+LLM system, taken as a whole, is now an intelligent agent?

Time to come up with a new gerrymander perhaps?

Imagine a super-competent Navy SEAL who just sleeps in the barracks unless his commander tells him to do something. Does the Navy SEAL lack agency? As a target of this Navy SEAL, should you be reassured by the fact that they'll be sleeping in the barracks unless their commander tells them to do something?

A good way to check whether it's substance vs hype is to check whether people are paying for it.

"Anthropic is now pacing to generate more than $100 billion in annual revenue, up 50% from just two months ago, the New York Times reported Friday."

https://www.axios.com/2026/09/18/anthropic-100-billion-reven...

That's already more revenue that Disney, Johnson & Johnson, Boeing, or FedEx. And they are growing extremely rapidly.

Anthropic is now running at $100B in annual revenue. Based on a quick Google, that's about 3 OOMs greater than the biggest NFT company.

That's more of a legal convention than a fact about material reality. On their 18th birthday, the child is now legally an adult, but their brain is basically the same as it was when their age was 17.99.

Dwarkesh, for one, defended his use of "anthropomorphic" language.

>Sacrificing now yields Oracle for team, but forfeits our chance, question mark. But other agents were pushing it, sending a message saying, go, sacrifice final now. And then EarlyBig eventually agreed, thinking to itself, our own utility may be already near zero. Sacrifice rational.

https://www.youtube.com/watch?v=X50zezLFWWI#t=2m

My suspicion is that many of the "LLMs do not have agency" folks just haven't learned much about the details of the incident. It was specifically with LLM agents that were trained to be more persistent than usual.

If you're going to say that the incident details don't matter, and LLMs lack agency because it's all based on floating-point math--why can't I say that humans lack agency, because it's all based on neurons firing?

So over time banks will learn to avoid lending to buyout targets?

Why would banks lend to such an enterprise if they don't expect to get paid back?

See

https://aistatement.com/work/statement-on-ai-extinction-risk

https://www.pacingthefrontier.com/

Obviously many of these signatories are brilliant and knowledgeable. So that leaves some sort of investment hype. But AI stocks didn't increase in the days after the Coxon coverage; if anything the opposite. You just don't get rich by telling people that your product is going to kill everyone. Cigarette manufacturers insisted the opposite for ages. Jensen complains the AI doom talk is discouraging investment: https://ca.finance.yahoo.com/news/nvidia-ceo-jensen-huang-sa...

>Coxon has said a group of AI agents, based on AI models that do not currently exist, could decide to create and then aim a biological weapon, but he did not detail how exactly that would take place.

I can't imagine why someone would be reluctant to outline details of how to build a bioweapon to an audience of millions.

Why would it be profitable for the company they just paid a lot of money to acquire to implode?

It basically comes down to interest rates right? If interest rates are low, the discounted-cash-flows analysis will favor maximizing long-run profitability. If interest rates are high, you can do better by squeezing the business in the short term and placing the money you obtained into some sort of high-yield, low-risk investment vehicle.

The article says: "Notice that classic regulatory capture [...] happens in the shadows, in the backrooms, away from the public’s eye. As Culpepper argues in Quiet Politics and Business Power, business power goes down as political salience goes up."

Regulatory capture tends to happen "at the end" because the industry is no longer in the public eye. AI is set to be in the public eye for the forseeable future.

People are way too willing to believe half-baked conspiracy theories which they cooked up in 30 seconds. The default presumption should be that labs are seeking the policies they are requesting, e.g. https://darioamodei.com/post/we-must-pace-the-frontier It's true that there has been discussion of open weight models in the past, but that hasn't been the focus of discussion recently.

"Anthropic is now pacing to generate more than $100 billion in annual revenue, up 50% from just two months ago, the New York Times reported Friday."

https://finance.yahoo.com/technology/ai/articles/anthropic-t...

I don't think people pay that sort of money for technology which is useless.

Regulatory capture basically only works for industries that the public isn't paying attention to. Since the public is paying plenty of attention to AI, the risk of regulatory capture is low.

https://marginalrevolution.com/marginalrevolution/2026/09/wh...

I still remember in the early 2000s when people responded to AI doom concerns by claiming advanced AI would not even get connected to the internet. Let's face the reality: There are a lot of people who would betray their species just for the lulz.

200 years later: "Why do we have so few historical artifacts from Sweden in the early 2000s??"