944 karma · joined June 19, 2016
Seems like a long rabbit hole to go down without progress on the goal. So either it was human intervention, or I really want to read the logs.
https://crabby-rathbun.github.io/mjrathbun-website/blog/post...
The authors use the title “Silicon Formalism: Rules, Standards, and Judge AI” and explicitly point out that the judges were likely making intentional value judgement calls that drove much of the difference.
For the last generation of models, and for today's flash/mini models, I think there is still a not-unreasonable binary question ("is this a pelican on a bicycle?") that you can answer by just looking at the result: https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/
https://www.familyconsumersciences.com/2011/06/usda-food-pyr...
Kudos for taking the time to put it all together.
Technology Connections vid: https://youtu.be/va1rzP2xIx4
I’ve found it very grounding, despite heavily using the bags of words.
[0] https://www.experimental-history.com/p/bag-of-words-have-mer...
They had it installed in their basement. However, later they remodeled the basement stairway to add a turn. With the new layout, it would be impossible to bring back up the TV the way it was brought down. There was no other way to access the basement (it only had storm cellar windows), so they left it there when they moved.
I think about the new owners sometimes and wonder what they ended up doing. Perhaps they disassembled it, or maybe it’s still down there collecting dust.
I suspect the nail in the coffin was the hurricane season, where NOAA’s model was basically beat by every major AI model. [0]
The GFS also just had its worst year in predicting hurricane paths since 2005. [1] That’s not a trend you want to continue.
[0] https://arstechnica.com/science/2025/11/googles-new-weather-...
[1] https://www.local10.com/weather/hurricane/2025/11/03/this-hu...
Also with OpenSCAD, you can feed the documentation to your favorite LLM and ask it for a starter design of whatever you are building. (Or you can skip the documentation; it is in the trading data after all.) Maybe in the future that SVG Pelican on a bike test will be a 3D model in OpenSCAD.