dom96

I built a brand new language to test this[1]. Not only is the language different to basically any other language but it also tries to be adversarial against LLM understanding.

The best models can still make sense of it[2], though the tasks so far have been pretty basic. But I do think it gives some evidence that languages which aren’t well represented in an LLM’s training can still be reasoned about and written well by LLMs.

1 - https://killswitch-lang.org

2 - https://bench.killswitch-lang.org

I'd love to run it on my benchmark but alas, Google not making it public prevents this.

Why announce this if it’s not available yet? Why not at least announce when it will be released to the public?

None of the other AI labs do this. Really frustrating.

Well, it is genuine.

Opus 5.5 fails the "understanding" tasks which Opus 5 passes. I feed it a script which takes two numbers and prints the max of the two numbers. Opus 5.5 thinks it prints 1/0 instead of the max numbers. Opus 5 gets it right.

Here are the outputs from both: https://gist.github.com/dom96/b5bce82b6e6c1ebd5271ed70ad941b....

Looking at that Opus 5.5 fails to deduce that the "hack statement" is actually an if statement in disguise, but Opus 5 gets this right. I feel like this is a pretty good test and shows Opus 5's greater intelligence.

Based on my benchmark[1] it is the same price as Opus 5.5 and just as capable.

1 - https://bench.killswitch-lang.org

Surprisingly (or maybe not) it matches the performance of Astra on my benchmark[1], but is much cheaper. It is also head to head with Opus 5.5 on both the price and pass rate, but edges it out slightly.

1 - https://bench.killswitch-lang.org/

Funnily enough the Fable safeguards are the worst and testing Sonnet 5.5 didn't trigger them as much as it even did for Opus on my benchmarks[1].

1 - https://bench.killswitch-lang.org

I built an adversarial esoteric programming language to benchmark LLM models and just ran it on Sonnet 5.5 It does worse than Sonnet 5. Mainly because it is more reluctant to keep going to get an answer, instead it returns to ask the user questions whether to keep going.

https://bench.killswitch-lang.org/

    Claude Sonnet 5    17.8%
    Claude Sonnet 5.5  7.4%

There are multiple phrases that can be used[1]

1 - https://github.com/dom96/KillSwitch/blob/main/SPEC.md#abrupt...

Agents are fairly good even at languages designed to trick them. I built one[1] and it does make for a good benchmark[2] to see which LLMs are actually good. I think that a language which is largely similar to others will be a piece of cake for most and any advantage that an existing language will have will be minor enough to not matter.

Btw if folks have ideas of how to make Killswitch even harder for LLMs I’d appreciate them.

1 - https://killswitch-lang.org

2 - https://bench.killswitch-lang.org

The Democrats are not a left-wing party, I'm not sure why you think "the Left" would be in love with him after all the other horrible things he's done if he just gave money to a political party that is at best centrist.

If you want him to get the benefit of the doubt about his "hand gesture" then it would help if he wasn't promoting far-right parties all over Europe.

It's surprising but MiMo V2.6 Pro performs better and is cheaper than GPT 6 Sol on my benchmark[1]. Open weight models are really snapping at the heels of the major western models.

1 - https://bench.killswitch-lang.org

Just updated KillSwitch-Bench with this new model: https://bench.killswitch-lang.org/

It does perform slightly worse than Opus 5, but it is significantly cheaper and faster.

Not yet. But can do so if there is interest.

I’m using openrouter which I think is a fair representation of what the typical user will experience.

It is an impressive model. Agreed on most that is written on this page, with the exception of it being fast. I ran it on my own LLM benchmark suite[1] and it is faster than DeepSeek but still much slower than leading models. But it's pricing is where it really shines.

KillSwitch-Bench 1.0

  Claude Opus 5           66.9
  GPT-6 Astra             57.9
  Claude Fable 5.1        46.7
  MiMo-V2.6-Pro           38.8
  Muse Spark 1.3          36.5

1 - https://bench.killswitch-lang.org/

How do you think other benchmarks work? Every single one is going to have a budget and time limit. There has to be a cap on those, you can't just let it spin forever and use unlimited funds.

I would argue that this benchmark is uniquely suited to how most people use LLMs because it actually tests common harnesses and it is a true coding benchmark for a language that is unseen, thus testing the LLMs actual ability to understand nuance and learn.

Internet access is restricted to ensure that over time models cannot just look up the source code of KillSwitch, which would allow them to cheat.

Very capable model. I just ran it on my own LLM benchmark suite[1] and it matches Muse Spark 1.3 in pass rate but is significantly cheaper.

KillSwitch-Bench 1.0

  Claude Opus 5           66.9
  GPT-6 Astra             57.9
  Claude Fable 5.1        46.7
  MiMo-V2.6-Pro           38.8
  Muse Spark 1.3          36.5

1 - https://bench.killswitch-lang.org/

Why not? In my own benchmark Opus 5 does in fact come out on top[1]

1 - https://bench.killswitch-lang.org/

(I'm one of the authors of this post)

> Being tied to use only one version of Python/Pyodide (the one that Workerd embeds)

This isn't quite the case, you can choose between different versions using compatibility flags. For example, `python_workers_314` is the compat flag for Python 3.14[1]. You've also got compat flags for 3.13 and 3.12. Though it is worth noting that by using those older versions you will also be using older Pyodide versions too, which have fewer features (for example they lack JSPI support).

> Architecturally tied to the JS/v8 world, which may show some challenges as they aim to reduce cold start times

That is indeed a challenge. But our memory snapshot implementation has improved the cold starts significantly already and we will be working to reduce these even further. We also have sharding these days which reduces cold start frequency a lot. We wrote about cold starts (and sharding) in a previous blog post[2] which includes some numbers.

1 - https://developers.cloudflare.com/workers/configuration/comp...

2 - https://blog.cloudflare.com/python-workers-advancements/

Gyeongjae, who is one of the authors too, is also a Pyodide core developer. Definitely couldn't have got Python Workers this far without both of their help.

It looks like that link is incorrect, it should be pointing at https://cold.picheta.me/#packages.

It’s a shame this model has such negative political baggage associated with it. It’s the only one I decided not to run in my LLM benchmarks[1].

1 - https://bench.killswitch-lang.org

We’ve got some cold start numbers in our previous blog post: https://blog.cloudflare.com/python-workers-advancements/.

There is still more to do here and it’s what we will be focusing on next.

Glad you like it :)

In case you missed the link on the page, the benchmarks themselves are pretty interesting too and give a good indication of the performance of different models: https://bench.killswitch-lang.org/.

How is this different to Content Credential and why isn’t Apple implementing that?

It's pretty good, but it would have been good enough as a non-VR title. The thought of digging out my VR headset to play it makes me shudder. I never even managed to finish it, played through most of it and then couldn't bring myself to get my VR headset out again.

This is great. I've been building my own model benchmark lately and it has indeed been so easy to mess up the scoring. It's simply much harder to come up with an algorithm that combines all your individual scores into something that isn't broken in some special circumstances. That's why I think many just start capping the results.

This feels like all the labs have reached the peak of what is possible with the LLM architecture and just need an excuse to spend the next decade finding the next big jump in intelligence.