They're impossible to objectively track by design and intent. Intelligence is such a nebulous target that one can find any number of metrics to support any premise: that the model is smarter, and that the model is dumber. We use benchmarks to attempt to standardise comparisons but these are quickly ingested into the training data and then become effectively useless. You might have heard the term "bench-maxed." Meaning that a benchmark has an effective lifespan in months.

They've been using the Pelican test in the /r/Codex subreddit with some success. One major finding is that OpenAI has been silently degrading the model while charging Astra prices. Another finding is that even when the model has not been silently degraded, pelican quality is significantly lower. Sometimes comically so. The general consensus right now is that the new GPT-6 Sol model is an updated Terra model. Many intelligence metrics are roughly similar. Meaning the most recent model updates were an attempt to rebalance compute rather than improve intelligence.

Ultimately I've never seen users as upset about GPT-6 Sol/Luna than I have right now. Even Astra has been noticeably degraded for me and everyone else I have asked. This is compounded by the fact that Opus 5.5 is a generational improvement at an affordable price. There is currently no competition.

it's been known since the early days of GPT-4 that they alter the models whenever they feel like it.

https://arxiv.org/pdf/2307.09009

the accusations are quite a few because people notice.

In a Codex subreddit there is a bunch of stats.

I don’t understand how there isn’t a website out there tracking this stuff already.