lxgr

It's at least highly implausible. Why would they engage in pretty uncontroversially illegal deception/fraud if they have so many other legal ways of gaming benchmarks, selling more tokens etc. available to them?

It's like arguing that your bank is scalping you by rounding down interest math on odd days of the month when they can just introduce a perfectly legal bullshit fee or otherwise change their terms to your disadvantage instead.

Yes, but then the model wasn't nerfed, the harness/overall product just had a plain old regression.

This is very different from a nefarious inference-side degradation to save cost, promote the new model or anything else frequently proposed as motivation.

Methodological mistakes can and do happen despite good faith.

Fair enough, but if you can't answer them, you can arguably not use their work to support your point either.

Glad to see good old human cognitive biases (or, in very advanced cases, just sloppy methodology) are still highly competitive with SOTA model hallucinations.

> We use the latest available Codex release with GPT-6 Sol.

This alone makes the benchmark unsound.

It's absolutely based on demand: If you increase the number of experiments, you also increase the number of statistically significant-looking results. See also: https://xkcd.com/882/

No wonder you're convinced about there being an effect if you can just discard any criticism of your theory as being coordinated by a cabal.

Where are these careful measurements? Are you 100% sure they don't change the harness between runs and have a large enough sample size to be statistically significant?

From that site:

> We always use the latest available Claude Code release and the SOTA model (currently Opus 5.5).

Changing the harness can have a big impact on performance even when leaving the model completely unchanged.

I guess there's one thing that could really bias my anecdata: I almost exclusively use Gemini in a professional/niche domain and ChatGPT in a personal/idle curiosity context, so there's a chance that ChatGPT is just as wrong but I am not knowledgeable in a given domain to tell.

Gemini is unbelievably bad for research in my experience. It hallucinates like it's 2023, doesn't use its own search, makes up fake rationalizations for why it didn't need to etc.

It's baffling that OpenAI managed to get better at web search than the company literally synonymous with web search.

> You don't really care that another company on the other side of the globe or street continues working.

As somebody in need of a doctor, on a plane etc. I'd definitely consider it much worse if all hospitals, air traffic controllers etc. stopped working at the same time instead of just a local subset.

Obviously I don't expect them to intentionally help me move to the competition, but I do wonder whether obscure data formats as a moat are a thing of the past.

Your second point is where I'd imagine the future moats to live: Exclusivity deals with service providers. Things are already in motion with Amazon banning and Shopify explicitly inviting Muse; we'll probably see much more of that.

It's obviously a tragedy when non-sophisticated users provide their credentials without understanding the repercussions, but there are sophisticated users as well, and these hopefully only provide access and data they can bear to lose.

> When I moved from crush to Zcode , the first thing I did was tell Zcode to migrate all of my crush customizations to Zcode and to set things up in a harness agnostic way going forward. It did.

This works for hosted mass-market solutions just as well! ChatGPT and Claude allow me to download all of my data, and these days data formats are less of a moat than ever given that you can just hand them to an LLM and have that worry about importing it into your new thing for you.

I've even seen explicit "offboarding prompts" to hand to your old agent, e.g. in Meta Muse.

It feels like the opposite to me. Moving providers with my Linux VM is incredibly annoying, but with these things, I can (so far) literally ask them to create me a tarball with a README.md and hand it to an agent on the new provider and have it do the rest.

We swear, We Really Wanted To Make An "ASI" Model But This Is Literally The Way The Weights Dragged Us This Time

Which children have $100/month of disposable income?

Muse is pretty bad branding in that it first was the name of a model, then that of a harness/agent/consumer product.

Claw is also an existing name for an existing harness/agent, but at least that would be the same category as dots.

If they wanted to address that market, the play would be to make them relatively cheap first, then constantly raise the price. (See also: cable TV and ironically cable TV "alternatives")

Interesting, so they're not available in the "plus" plan?

$100 is a pretty tough sell when the competition starts at free (Meta Muse).

Wanting to have the newer model than the competitor, presumably.

Neither electricity, nor the Internet is provided by one or two organizations globally, both with a blast radius that quite often takes out all of their services.

Not sure how long the latter will remain true – centralization is well underway there; try not to think of Cloudflare too much...

That certainly makes sense at an individual level, but it seems implausible as an explanation of the dramatic shift over the past decades from TFA, unless basic character traits have shifted dramatically over the past decades as well.

I can almost hear the next agent swarm furiously brainstorming how to most credibly steganographically write like middle schoolers.

Altitude does impose a lower bound for the number required for global coverage but they have much, much more than geometrically needed. Iridium gets by with about 70 and is only marginally higher.

As for steered terminals, you actually only really need these once you have more than one satellite visible in the sky at the same time. Without that, they’re purely a power efficiency feature.

This really seems bananas. Why? What is the legitimate use case of the remote agent being able to run code on the local client's machine!?

It doesn't have to happen at exactly these specific ground stations, though (although any outage will increase the load on in-space forwarding capacity and downlink capacity of some other satellites, though).

> Starlink’s special sauce is the low earth orbit.

Arguably Starlink's special sauce is quantity of satellites and massively higher bandwidths combined with high-gain, electronically steered beams on both sides (i.e. terminal and satellite).

None of these are net-new as technologies: We've had LEO based telecommunications satellites for over two decades now (Iridium and Globalstar, the former of which even has in-space forwarding as well), as well as directional beams on both sides, but rarely all in one system and never at this scale.