Rudybega

You don't actually believe in the core tenets of the USA if you think that the government should have or should exercise unchecked, abusive power.

Then the government should end their contract with Anthropic. The terms of the contract were clear.

Designating them a supply chain risk is unprecedented authoritarian strong-arming.

Well yeah, they're running a small, outdated, older model. That's not really the point. This approach can be used for better, larger, newer models.

If only there were some way to test it, like swapping the two nouns in the sentence. Alas.

If a passenger says to you, "go around this car by using that private driveway", are they driving the car?

There aren't remote "drivers" in the Philippines, that's not how Fleet Response works. You can see how it works here if you're curious: https://waymo.com/blog/2024/05/fleet-response but the TLDR is that they give the Waymo driver options in confusing situations (things like, you can go use this driveway on the side to pass this blocked traffic).

MMLU performance caps out around 90% because there are tons of errors in the actual test set. There's a pretty solid post on it here: https://www.reddit.com/r/LocalLLaMA/comments/163x2wc/philip_...

As far as I can tell for AIME, pretty much every frontier model gets 100% https://llm-stats.com/benchmarks/aime-2025

I mean, Waymo gives a lot of examples of the situations, in their blog post about Fleet Response where they detail this, released May 21, 2024. They're very explicit that the Waymo Driver autonomous system is in control the entire time.

This isn't something new.

https://waymo.com/blog/2024/05/fleet-response

TinyCC can't compile a modern linux kernel. It doesn't support a ton of the extensions they use. That Rust compiler similarly can't do it.

There are two compilers that can handle the Linux kernel. GCC and LLVM. Both are written in C, not Rust. It's "in distribution" only if you really stretch the meaning of the term. A generic C compiler isn't going to be anywhere near the level of rigour of this one.

I think they immediately corrected their median timelines for takeoff to 2028 upon releasing the article (I believe there was a math mistake or something initially), so all those dates can probably be bumped back a few months. Regardless, the trend seems fairly on track.

The denial of this overall trend on here and in other internet spaces is starting to really bother me. People need to have sober conversations about the speed of this increase and what kind of effects it's going to have on the world.

Direct transpilation would create a ton of unsafe code (this repo doesn't have any) and fixing that would require a lot of manual fixes from the model. Even that would be a massive achievement, but it's not how this was created.

You're comparing two different benchmarks. Pro vs Verified.

It wrote the compiler in Rust. As far as I know, there aren't any Rust based C compilers with the same capabilities. If you can find one that can compile the Linux kernel or get 99% on the GCC torture test suite, I would be quite surprised. I couldn't in a search.

Maybe read the article before being so dismissive.

Yeah, I'm having the same issue. Disappointing limitations.

That's probably more a function of more people being in the road than people not understanding what object they're about to hit.

"Oh that obstructing object doesn't look like a child? Gun it, YOLO." Lmao.

I suspect the cars are trying to avoid running into anything, as that's generally considered bad.

I wonder if you could get around this with a slightly more sophisticated harness. I suspect you're running into context length issues.

Something like

1.) Split audio into multiple smaller tracks. 2.) Perform first pass audio extraction 3.) Find unique speakers and other potentially helpful information (maybe just a short summary of where the conversation left off) 4.) Seed the next stage with that information (yay multimodality) and generate the audio transcript for it

Obviously it would be ideal if a model could handle the ultra long context conversations by default, but I'd be curious how much error is caused by a lack of general capability vs simple context pollution.

Holy crap. That's actually kind of incredible for a first attempt.

I think it's probably still useful to distinguish addictions with hardcore substance related barriers to quitting (think withdrawals) from addictions where the barrier is a lack of dopamine or serotonin or simple habituation.

For people with normal executive function, the second category of problems should be fairly tractable to overcome, whereas the first is still quite difficult.

The second only really becomes an issue when you have a bit of executive dysfunction.

Maybe that distinction is important and one merits the term addiction while the other doesn't? Though both categories seem to be relatively treatable with drugs that massively improve executive function, so the parallels are pretty glaring.

Scaffolding typically just refers to a larger state machine style control flow governing an agent's behavior and the suite of external tools it has access to.

This is actively being worked on my pretty much every major provider. It was the subject of that recent OpenAI paper on hallucinations. It's mostly caused by benchmarks that reward correct answers, but don't penalize bad answers more than simply not answering.

E.g.

Most current benchmarks have a scoring scheme of

1 - Correct Answer 0 - No answer or incorrect answer

But what they need is something more like

1 - Correct Answer 0.25 - No answer 0 - Incorrect answer

You need benchmarks (particularly those used in training) to incentivize the models to acknowledge when they're uncertain.

Yeah, this is a pretty major error. They already have a higher revenue than the article's quoted "jump next year" (which was this year and was an underestimate).

> what about the fact frontier labs are spending more compute on viral AI video slop and soon-to-be-obsoleted workplace usecases than research?

That's a bold claim, please cite your sources.

It's hard to find super precise sources on this for 2025, but epochAI has a pretty good summary for 2024. (with core estimates drawn from the Information and NYT

https://epoch.ai/data-insights/openai-compute-spend

The most relevant quote: "These reports indicate that OpenAI spent $3 billion on training compute, $1.8 billion on inference compute, and $1 billion on research compute amortized over “multiple years”. For the purpose of this visualization, we estimate that the amortization schedule for research compute was two years, for $2 billion in research compute expenses incurred in 2024."

Unless you think that this rough breakdown has completely changed, I find it implausible that Sora and workplace usecases constitute ~42% of total training and inference spend (and I think you could probably argue a fair bit of that training spend is still "research" of a sort, which makes your statement even more implausible).

I wonder if you could dramatically improve these results with some relatively simple scaffolding and tool access.

If a ton of these mistakes are genuinely simple calculation errors, it seems like giving the models access to a calculator tool would help a fair bit.

Higher token throughput is great for use cases where the smaller, faster model still generates acceptable results. Final response time improvements feel so good in any sort of user interface.

> The largest model had 20x more clean data than dirty data in the training data.

Yeah, I think this is the main misinterpretation. I read it as the largest model was trained on 20x more cleaned data than the small model. I don't think the ratio of clean to dirty data was 20x. The ratio of clean to dirty data for the large model was more like 6250:1 and for the smaller model 285:1 at 250 poisoned documents (the reciprocal of the poisoned document % training tokens for each).

I'd love to see a blind study comparing a wide spectrum of these AI tracks to lesser known real artists (so the participants don't just recognize the songs) to see whether people can tell or if knowledge of the source biases them. I'm genuinely curious as to the results.

I don't think people would think anything strange of a lot of these tracks if they just randomly heard them on the radio.

I actually think the vocals from ~2:00-~2:35 are pretty impressive there. It's wild to me that the models can play with tempo like that.

I've been listening to this across a variety of genres though, maybe these lyrics and vocals are more to your taste:

(similar to Opeth) https://suno.com/song/9ab8da05-c3f2-412d-80b4-c7d0b3ae840f?s...

(indie rock) https://suno.com/song/756dd139-4cba-4e40-b29c-03ace1c69673