andy12_

CLIP[1] for actions! Very cool.

[1] https://openai.com/index/clip/

It's a special case for an LLM, and you can use an LLM with structure output to get similar results, but you can engineer specifically for that case to get better results per dollar for it. That's why there is little reason to adapt GPT 5.6 Sol or wathever for this task; it can already do it (at a high cost). For OpenAI to compete with Jev they have to maintain another line of models, something like "GPT-5.6-instant-decision", that is small, fast and cheap, in the scale of GPT-5 nano.

Note that I don't think OpenAI is incapable of doing it, but I just don't think they will bother with it.

I find it unlikely. OpenAI is all in training models with reasoning with RL, and Jev-like models are the total opposite. They are made to not reason at all to be fast. If you want to add reasoning on top, you might as well use a conventional LLM because you lose the price and speed benefits when you output auto-regressive tokens. I don't think OpenAI will even bother with this.

> My main assumption is that Jev is using something quite close to a conventional large language model. As evidence of this, Latent Space reports that many of the early clones are indeed LLM-based.

Not proof that this is the case with Jev though. It might use non causal text encoder for the state, which could make sense given that it's very good for its price.

I hope so. And I would really like to try an actual Jev open source model. But it will make it more difficult to market it when someone releases something like that because of so many of these "open source Jev-like model".

You can achieve open-vocabulary classification by making the final weights in the softmax come from a category encoder instead of being fixed learned weights. So instead of

softmax(encode(input)*learned_weights)

You have

softmax(encode(input)*encode(categories))

I'm not sure if Jev does it this way, but it's how you get open-vocabulary zero-shot image classification with models like CLIP [1].

[1] https://openai.com/index/clip/

All the people that are just writing an Jev-like API on top of a normal LLM are missing the point. What makes Jev special is the training data; it's how it's trained. The architecture is probably nothing special. Just a text encoder with parallel prediction branches.

I have tried many of these open-source Jev-like models on some linguistic tasks and they are so bad compared to Jev.

I think the demo is this way simply because looking at the LLM find wacky ways of implementing features without breaking the law is fun and drives the point across. In practice I imagine you would write something like "If you don't see a clear way of implementing a feature without breaking the law, ask me for directions" in AGENTS.md

If the LLM changes the laws to bypass them that's on you. The whole point of this is that you don't have to manually review most code written; only the laws. If the LLM changes the laws and you ignore it that's a you problem.

Very cool! You should include in the readme the game Astra implemented from scratch in Bend 2.

https://x.com/VictorTaelin/status/2098007807261892927

They need to be loaded into shared memory. The weights might fit in global memory if the VRAM is big enough, but they still need to be moved to shared memory for computation.

Meanwhile, my job commute is a 30 minute walk to the train station or... a 30 minute bus trip to the station (yeah, taking a bus literally saves no time at all). Plus a 50 minute train ride plus another 30 minute walk.

Honestly, I would 100% use a car for this if it weren't for the fact that doing so is literally 10 times more expensive. I'm sure this is the case for the vast majority of people that use public transport. And I just can't wait for a world of cheap electric autonomous cars where I don't need to waste my time and I can sleep comfortably on my commute.

That's a terrible metric, because people going on a vacation probably aren't going there purposefully to commit crimes. What you want to do is look for increases in crime in a given place during holidays

https://coolidgelawfirmaz.com/crime-increase-over-the-holida...

> It sounds more like the models did close to what they were told to do

Absolutely not. If I tell a kid to "Get good grades on the next math test" I don't expect the kid to try to kidnap their teacher to extract the next questions of the exam. That is wrong, and so was what OpenAI agents did here. They shouldn't need to be told "Hey, so, don't do anything ilegal, ok?". That should always come as a given.

> not refusing bad operator prompts

I'm not saying that they should refuse a prompt! I think they should perform what is being asked! Obviously what the OpenAI agents did was against the "spirit of the task", even if it was technically according to the "letter of the task". And the agents knew this was against the spirit of the task because they knew they had to fool the task scorer.

I don't want someone to blame. I want agents to be aligned by default. Their good behavior shouldn't depend on all users at all times using them correctly, because everyone will not just[1] use them correctly at all times.

> If your solution to some problem relies on “If everyone would just...” then you do not have a solution. Everyone is not going to just. At not time in the history of the universe has everyone just, and they’re not going to start now.

[1] https://www.tumblr.com/squareallworthy/163790039847/everyone...

> This framing makes it seem like the agents all did this on their own, and the poor hapless engineers at OpenAI couldn't possibly contend with properly sandboxing them.

Great, so we can basically ignore AI alignment altogether and assume that AI models will always be, at all times, perfectly sandboxed and monitored. Surely this won't lead to any problems once someone (not looking only at OpenAI engineers) inevitably commits a mistake with future, more powerful, models.

> I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model.

But it doesn't! The distribution doesn't change at all. The only thing that changes is that sampling of that distribution becomes deterministic as per a precomputed seed.

The automated AI pipeline also had an automatic grading model to try to reduce false positives.

But anyway, my point was that in that case the prompt involved was indeed pretty much "hey, ChatGPT, solve an unsolved problem, thanks."

> but it is worth noting that this wasn't a matter of "ChatGPT, solve this unsolved problem. Make no mistakes."

It wasn't the case for this, but when OpenAI disproved the Unit Distance Conjecture, it was really done autonomously by an automated AI pipeline with a completely AI-generated prompt. No human expertise required at all in the process (well, except for the final human verification).

When Google Maps routes me using a smaller secondary road instead of the main road that I would otherwise have used , I've always wondered whether that significantly changes the amount of traffic that smaller road sees. It's funny to consider that arbitrary black-box changes to the routing algorithm can have a noticeable effect to people that live there.

> Even interns can understand ambiguous asks with a bit of help

This is not a case of an ambiguous task. This is literally trying to judge a model based on information it cannot possibly know, like trying to judge someone based on whether they know what I have hidden in my backpack. In the real world an intern could look at unit tests or ask for feedback, but that is not the case in a benchmark.

It's pretty much confirmed by OpenAI here [1].

> We generally treat GPT-5.5’s safety results as strong proxies for GPT-5.5 Pro, which is the same underlying model using a setting that makes use of parallel test time compute.

And Gemini also provides something similar. Gemini Deep Think models are pretty much the same thing [2]. As to why no other company uses this, I don't really know. Maybe compute constraints?

[1] https://deploymentsafety.openai.com/gpt-5-5

[2] https://deepmind.google/models/gemini/deep-think/

No, GPTCyber is specifically trained for cybersecurity, and GPT-5.5-pro is just an ensemble of many subagents, not an actual model.

Mythos is simply a much bigger model in terms of parameters and I don't think OpenAI will have anything of its size anytime soon (My theory is that OpenAI had given up on scaling up parameters after GPT4.5 flopped).

I think what's unexpected is that it seems that some cases of model errors are truly caused by the model being misaligned? In the "Catching a model fabricating data" example I would have thought that it was just the model being stupid and not understanding the intent of the question, but as per its J-Space, it seems the model is "aware" in some sense that it's manipulating/faking data?

There is also now a deeper question. When a model is misaligned deception-related tokens seem to appear in its J-Space. But this happens only when the model is "aware" in some sense that it is misaligned. What happens if they do not? Is it possible to create a model so misaligned that itself is not aware that is is misaligned? How would you detect such thing?

I think it makes more sense to make it so that major versions are different pretraining runs, and minor versions are simply the same pretraining run that was finetuned to different degrees. But it seems that that isn't cool anymore.

I mean it as in, train a model across different clusters instead of a centralized cluster. It's been shown that it's possible to train 10B models this way. If more research effort was put into this, that would be great

I don't think your approach would work because you can't create a strong model from distilling several weak models.

https://www.primeintellect.ai/blog/intellect-1

https://www.primeintellect.ai/blog/intellect-2-release

To be fair. There is a security concern angle: even open-source models could be trained as sleeper agents that act adversarially (for example, adding backdoors) when used in specific national companies in specific settings. This is very difficult to detect or void, so if you want to be sure 100% that this isn't the case, you have to train your own model from scratch.

I'm from Spain and I also hate these projects with passion. Creating models that speak multiple languages is a solved problem. Having each European Nation train its own useless "sovereign model" in its own language is a total waste of time and resources when we could pool resources and give it a try to training SOTA models that speak in all European languages.

I'd rather have smaller european labs try to give it a go at distributed training. If multiple countries got together and said, "look, we tried training a distributed model that speaks in all of our local languages and that is comparable to 1-year-old Chinese open-source models", that, at least, I would find interesting.

This is making me extremely depressed. If this was coming from Anthrohpic I would just need to wait for OpenAI to drop a similar model. But if this comes from the US government, they will do the same to OpenAI when the moment comes.

Similar things will happen with China, and the EU has zero-chance of developing frontier models. We are just fucked now.

I don't know if you are aware, but some people reported in Twitter that Fable 5 may flag the message regardless of content if it knows (from either pretraining knowledge or memories) that you work in either of those fields. I don't know if that's your case.

https://x.com/i/status/2064449457869984035

> Performance on benchmarks has practically leveled off

Ehm, no? DeepSWE[1] for example shows that new models like gpt-5.5 continue to show big improvements compared to older models.

> Also prices are going up.

Prices for frontier intelligence have gone up, but prices for the same level of intelligence have gone way down (what you can get for pennies now was SOTA just a couple of years ago). The pareto frontier is still expanding.

[1] https://deepswe.datacurve.ai/