Ask HN: What was the last task where only a frontier model could do it? | Hacker News Reader

Ask HN: What was the last task where only a frontier model could do it?

24 pointsthedebuglife33 comments

ive been seeing a recurring claim that open (weight) models 6 months behind the frontier are good enough for the majority of ‘work’. if you've had a concrete task in the last month where GLM/DeepSeek/Kimi/Qwen failed and Opus/Fable/GPT succeeded (or vice versa!), please share

Ill provide a template as ive also frequently seen posters complain about a lack of context: - task (short and specific) - cheap model tried & how it failed - frontier model & did it actually succeed - would frontier-1 have been fine in hindsight?

View on HN

I don't bother to use open-weight models, but for all of the "security" or "security adjacent" work I have tried recently, GPT 5.5 x-high has been the first model that produced useful output.

Even very old models could spot the most glaring issues, but it's a different story if you scan a source repository where humans can't find security vulnerabilities even after hours of reading through the code. Feed something like that to, say, Gemini Pro 3.1 and you'll get a bunch of false positives back, nit-picking, or variants of "this could be insecure if the code around it changes in unreasonable ways in the future".

Feed the same thing into GPT 5.5 x-high and then tens of minutes later it'll find half a dozen unauthenticated remote code execution vulnerabilities, arbitrary file read/write vulnerabilities, or similar.

Until it got nerfed, Mythos was similarly a huge step up for a lot of people working on code security.

This latest crop of frontier models (Fable, Sol) are very good at broad exploratory directions like "audit for bugs" or "find places in this codebase that I custom-implemented functionality that has since become available in the libraries I am using" in ways that prior generations would fail or not return as high quality results.

I did a recent head-to-head comparison between Qwen 3.6 35B running locally and Fable on the above and while Qwen did an admirable job finding a good handful of things, Fable returned many more, and more serious, issues. This included many more issues that crossed system boundaries in a multi-service repo.

In terms of one-shot "follow these detailed instructions" I think we are at a point where frontier models are overkill, especially at the price. For broad fan-out explorations, I think we are just getting started.

I was worried about some messup with taxes (my tax advisor messed up here), I managed to get it sorted on time - this was super naive but in -germany when they send you the taxes they mention the "cents" place as well in a very weird way, I assumed that was the entire number and assumed the tax issue I had was 10x larger than the amount it really was (10x and not 100x as my brain is in a place of pressure due to other circumstances and I wasn't thinking clearly)

GPT 5.6 incorrectly stated that I had nothing to do, Fable got the issue correctly and I was able to see that that was indeed the cents place and that I was more worried than I realized, and I managed to get a temporary solution setup (that I verified and I am sure is correct).

Which is to say it managed to relieve me of quite a bit of stress haha

Fellow finanzamtpostempfaenger here. I've experienced the moment of shock reading their amounts and almost fainting more than i'd like to admit.

https://squishy.franzai.com/ - fable 5

even with opus 4.8 it could not come up with the necessary algos to reliable create harder and harder levels - in the now famous big fable 5 gap of 2026 opus even screwed up the algo a few times and needed to recover it from git.

Pudding Monsters (2012) [1][2], vibe-slop version, yes?

[1] https://en.wikipedia.org/wiki/Pudding_Monsters

[2] https://www.nintendo.com/store/products/pudding-monsters-swi...

slide game yes, different goal and mechanics though, and as stated developed / directed via frontier models

how is this slop? It's pretty good. Or do you classify ANYTHING made with ai "slop". So reddit is slop, x is slop, slack is slop, discord is slop, etc.

Definitely a yes to all those. It’s largely just middle aged dudes posting meme’s to each other.

do you truly think that every LOC on those services were generated by AI, lest not reviewed?

It all depends on how hands on you are willing to be.

If you are using the model to write to code faster with extensive human oversight you can develop a lot faster using the non-frontier models. I was doing that extensively last summer.

But now my thought process is I want to focus on architecture and product direction. I have not seen Sonnet level models be capable of performing autonomously enough to take a feature end to end reliably enough to be completely hands off. In fact there a many cases where Opus will fail as well where Fable will succeed.

Of course that is not to say that Fable will always do things correctly. It will happily take an under-specified problem statement and happily use up all of your usage to build the wrong thing, while Opus at least recently stops constantly to check in.

I have been experimenting with dissassembly and rewrite of 90s dos games in Rust.

It's not something I have experience in so I rely heavily on the models setting the direction and automating the work.

It's something where the level of intelligence has a huge impact. I still haven't succeeded, it's mostly an experiment for me to test the edges of model capabilities and see how they tackle such a hard problem.

Out of curiosity - is this a "couple hours" sort of thing, or a "tens of hours" or "hundreds of hours" or other?

hundreds.

likely my approach is often not optimal as I'm not experienced in this field

You have to reach set hard principles for disassembly. I've been experimenting with this for about 6 months now, starting with 4.6 in February where it would do a hacky half ass buggy job of an NES game.

I got a shitty PSX recompiler together in March. Been refining it since. This week along I stood up 5 different psx games and added widescreen to 4 of them and added an in memory injection English translation to another to avoid having to fight the assembly.

If you did a write up of that I would be very interested. I assume a lot of others would be as well.

I've been in heavy build mode for a while so my docs are a bit dated. But the repo is here: https://github.com/mstan/psxrecomp/ and at the bottom of it a Discord community I started for like minded individuals. Happy to chat there in the interim to get things rolling.

Oh cool thanks for the link. I look forward to reading through it. (I've noped out of discord after their ID demands though, never using them again.)

Would be curious to learn more from you if you have any other guidance or details.

I found the blog post: https://1379.tech/i-built-a-ps1-static-recompiler-with-no-pr...

That'd be it! A lot has matured since then on the project. Still in heavy build mode, though.

I feel like the converse question is more relevant: what projects have open source agents been good enough to implement on their own?

I don't agree with the people who say that frontier models are "overkill"; sure, they may be expensive (API usage especially), but the subscription plans are fair for the insane value they provide. I have a session where, if I had to pay for API usage, I would have paid close to $300.

= I had a lower‑tier model that I couldn't trust to make things happen for me; I somehow got the intuitive feel of what it could and couldn't do, so I use the "frontier" models for those tasks—things that include complex AI integration, weird responsiveness edge cases, or just when I find myself begging the lower‑tier models to just fix something again and again.

I wanted to see how Fable could do at getting Linux running natively on my M4 Mac mini. Turns out pretty well: it patched the Asahi installer and m1n1 bootloader to support M4, and created a workflow where it could build, push, and debug kernels with no interaction on my part after the initial setup. It's now happily running headless with ssh access, and hopefully will have video soon after a USB DisplayLink adapter arrives. (No HDMI/Thunderbolt yet).

I don't know for a fact that earlier models wouldn't be able to do that, but I figure if they could we would have heard about it by now.

My Fable today shipped a stable and app store compatible linux as an mac app.

Opus 4.8 started that project but even with rounds and rounds of feedback and gpt 5.5 quality assurance subagents it stayed buggy / unstable as hell.

the version now seems pretty stable. an ai within that app is humming along quite fine for hours without any crashes.

Is the networking a USB gadget, or were you able to get around the PCIe + DART crux?

USB2 only, which still makes for a surprisingly useful system with DisplayLink and wifi adapters. Supporting the rest of the hardware looks like it will require watching macOS under the hypervisor, which Opus doesn't think it can do and Fable isn't allowed to think about.

“USB2 only” still doesn’t narrow what I was asking about. Do you mean the existing m1n1 DWC3 device-mode/CDC-ACM proxy, or Linux actually running xHCI in host mode and enumerating a USB Wi-Fi dongle?

Asking/Curious because I have been working on something for a few months, building an OS that is tuned for inference on Mac hardware. The amount of tooling I have had to create to get things like native networking, working; let's just say it would be a heck of an ego hit if Fabel "just got it working" ha!

Coming up with good French grammar quizzes. GPT5.6 is noticeably better.

Older models usually pitch the difficulty too hard/easy and rely on me remembering the gender of words.

I’ve not noticed a big programming improvement with the newer models.

I haven't seriously used GLM/Qwen/etc for agentic work. I will say, that the GPT and Claude models from 6 months ago were very usable within their respective agentic harnesses (this matters a lot). But there is no way you could convince me to go back to GPT-5.0 from GPT-5.6 for a serious amount of agentic work.

I could probably accomplish most of the same tasks with GPT-5.0, but it would take a lot more involvement from me, more troubleshooting, and significantly more time.

it seems we should be storing chains of prompts instead of code, and then replay them when a new model is released.

6 months behind is probably a bit rich, I find the frontier models more capable in sustained agent and compiler tasks. Simple tool-chains and basic reasoning seems more or less on par across the board, at least for my use cases.

Reply on news.ycombinator.com