- gold at IMO
- gold at IoI
- beat 9/10 humans in atcode heuristics
- longer context, better models, routing calls to cheaper models, 4-6x cheaper inference for 90% of the top models capabilities
- longer agentic sessions while being coherent/solving tasks (30-90min)
Yet every other post here and there are about "bubble this", "winter that", "plateauing this", "wall that"...
Are we in the denial stage, or bargaining stage? Can't quite tell...
> Are we in the denial stage, or bargaining stage? Can't quite tell...
I can tell quite clearly that even assuming that the models are not rather "fine-tuned" to win these competitions, these achievements neither transfer to the kind of coding that I do at work, nor to the one that I do privately at night.
At work, a lot of what needs to be done is
1. asking people who are knowledgeable about the business logic why things were implemented this way (there often exist good reasons, which nevertheless are often quite subtle).
2. If some new requirements comes up, think deeply into how these new requirements fit into the huge legacy codebase (I am allowed to change things here as necessary (which is an enormous concession that is uncommon in this industry), but my code changes should really never ever cause the software to produce wrong results or break business-critical workflows, because such failures can cost my employer quite some money, or increase the workload of already overworked colleagues (they will then legitimately hate me :-( ) who in specific months have to work under very tight deadlines. What is a "business-critical workflow" here that should better never be broken? The answer requires understanding the very demanding users over many, many years (believe me: it is really subtle)).
I cannot imagine how AIs could help with this.
Privately, I tend to write very experimental code for which one can very likely not find similar code on the internet. Think into the direction of turning some deep scientific results into more "mainstream" code or turning my avant-garde thoughts about some deep problems into code so that one can do experiments to see whether my ideas actually work.
Again something where AIs can barely help.
I'm still in the "wait and see" stage. Maybe throwing more compute at the problem will solve it, but maybe not. I would like to see benchmarks that take a more project-based approach, e.g. tell the LLM to go work on something complicated and ambiguous for a week and see what it comes up with.
Thermodynamics is the law of laws, unless they invent some kind of ultra-efficient, almost magical computers to run these systems, it's simply not economical yet.
The reasons could be that it's new (they did say they plan to release eventually but not soon), or that it's too heavily scaffolded for the task and not sufficiently general.
One day but not yet. Beyond pure capabilities the companies making AI don’t seem to have any sort of moat so it’s a $$$ incinerator for them so far.
Like the late 90s internet I suspect we’re in a bubble. But also like the late 90s internet I suspect there’s more in store here in the future.
> yet still the daily general models are still useful
Yup. I just had a ~30min session where gpt5-mini did everything I needed, almost 0shot. Nothing complicated, but production code. Wanted to refactor a small service, wrote a ~4 sentence goal, it went on and asked for read permissions on the repo, understood the API requirements perfectly, wrote itself a plan, did the refactoring, all the previous tests pass (confirmed manually just to be sure), all good. Would have taken me maybe 4 hours?