What is your evidence?
> "Once the weights of a model are public, they cannot be retrieved. If a model possesses dangerous capabilities, it is permanently out in the wild... We need to consider regulatory frameworks that account for the unique risks of open-source distribution of highly capable frontier models."
It definitely sounds like the kind of thing that ends the world in B sci-fi thrillers.
Open weights models are responsible for enabling reams of research on interpretability methods that do just that. And they have facilitated so much collaboration on architecture, inference optimizations, training and steering methods, and other topics that were completely out of reach with closed models like Anthropic's. It's really staggering to me.
“His warns that once powerful models are released openly, companies lose the ability to monitor misuse, revoke access, or update safety guardrails.”
Did fearmongers like Amodei say, "Oops, we were wrong! It wasn't that dangerous after all"? No. Of course they didn't.
1) the company has device-level control to the degree that they can not only restrict which API endpoints people can connect to but which accounts they use to do so (in which case this already isn't an issue); or
2) they don't, and all bets are off anyway, open weights or not.
Skill athrophy is a real thing though; we try to prevent this by have hackethons (for lack of a better word) without AI where I pick something extremely non trivial and we implement it for fun and profit without AI (with would not matter much as they are currently bad at these things); last one was flex paxos for our in house db with obvious metrics for the endresult: data integrity (duh) under failure and performance better or at least the same as our raft production version.
You’ll never guess what product your clients are looking to replace with their own next.
For now everyone is still sufficiently crap at using AI to need help. We had enough clients trying to build something themselves and then come crying to us.
I'm a skilled senior (I'm 54 and been coding since I was about 8; I've been 100% AI-generated code for at least 6 months now and have produced a combination of speed and quality that has astonished me; my velocity is apparent at https://github.com/pmarreck/) and this has been a massive net gain, so your claim is now officially in sheer defiance of reality.
In a skilled senior's hands, this is like an expert power tool. In the hands of someone less-skilled, it is likely also... less-skilled. It's a magnifier.
> and the hidden cost in terms of technical debt and skill atrophy is just being swept under the rug.
Nope, no it's not. It's being reviewed, measured, and controlled against. Because... you WILL need more controls to take full advantage. Look, I even invented a whole new control methodology around it called MFIC: https://gist.github.com/pmarreck/b30aa3ca69cb70a5526f8a63ab8...
I recently did a fleetwide upgrade to Zig 0.16. Do I remember every single change from 0.15? No. Do I have to? Also no. Both because I can look it up if I need to, but also because the LLM already does.
If I don't look at a codebase that I myself haven't looked at in a year, I will not recognize some things when I return to it. Is this sense of "atrophy" meaningful when this was a problem long before LLMs came on the scene?
On personal projects, where I am in charge of all the hats (product development, UI, UX, backend, security, server admin, etc) -- absolutely crazy force multiplier. You get a nice suite of backend and e2e tests running, with full business scenario layered on top of that, and constantly running agents to do the coding, another agent on a higher level of reasoning to review that work, and sometimes occasionally poping into another competitors model to review their work just for added comfort -- it feels like wizardry. I am not vibing it, but I wouldn't say I am carefully scrolling through every line. I review whats fundamentally important, especially when it comes to data, overall structure, and large, cross cutting concerns, but I would be lying if I say some code doesn't land that I don't read. But I have the security of the test suites and validations , so I pour more effort into that.
It's a nice self reinforceing loop.
All of this might sound like I agree with you, and to some extent I do, but I am realizing as the apps I have built out like a cannon shot out of hell with tremendous speed and polish right out of the gate are starting to slow down. Feature adds are getting more complex. My memory is not what it used to be. Each run and pass through the code consumes more of my tokens and limits. I am starting to do less in the same amount of time. Codex did a vertical slice of a feature for me (well defined and well planned). It contained functionality that has historically plagued us developers -- the dreaded time. I used xHigh GPT 5.5. It had obvious bugs, but I wanted the robots to catch it. I popped it in claude (on the new sonnet 5! heyo!) -- Claude caught the bugs. Even said they "immediately stood out" I wondered how this happened. Frontier model from company A was evaluated by workhorse model from company B. All of this again took massive amounts of usage. And time.
And this is -- best case scenario, perfect world, everything is in perfect alignment.
Now for the work reality.
Multiple product and experience owners. Multiple dev teams. Different enterprise teams support services you rely on. You don't have full unfettered access to frontier models. You have to use copilot, or some other enterprise harness, and you run out of credits for the month, you are SOL. It's not as good as your claude, you think to yourself, but hey, its familiar enough, and you have 5k credits left for the month for Opus 4.8, better make the best of it. But now you burned half of them working on that Transactional Bug that was mixing synchronous and asynchronous semantics that the other guy's model should have picked up on. What happened? Maybe he didn't use Opus, maybe he used Haiku, maybe his prompt was bad. Who knows. Gotta fix it. Oh, you gotta reach across the isle and put in a request to get the Enterprise team to look at this caching inconsistency on user data that you need and is really the source of your race conditions. Tick tick tick. Model limits approaching. You start wondering if you just did all this by hand like "in the old days" would you have got it done correctly faster? Or at least, cheaper. You'll never know.
the distinction between personal projects and Enterprise development is a big one. A severe bug in my personal projects, i fix it on the fly. A bug in our products rolled out, nightmare.
Scaling in this sense is not operational (“servers”), but conceptual (“features”).
I don’t want to be a downer but I find many devs are not great at this. Very clever folks, but they tend to not see these issues clearly. They’ll nod and recognize when you talk about separating content from form and the importance of various design principles like high cohesion and loose coupling but completely disregard them once in contact with reality.
Part of the problem, as you nicely showed, is that technology is only a single slice of this problematic pie. Organizations in general are systems as well and they tend to be either badly architected, badly maintained or often both. Some technological issues are downstream from organizational issues and IME those can be become rather dominant variables in the equation and no amount of AI - save full AGI taking control of the company - is going to save us from those factors.
Extraordinary thing to say about the fastest growing company in the history of capitalism. They will soon have access to public markets, essentially unlimited capital, and can build insanely large models that they don't have to make public... ever. They can just use those models to run their business, train better models, eat competitors, etc.
But maybe it's Anthropic that isn't thinking ahead enough - you clearly think you can see around corners with your proclamation. So why do you think they have "little to no chance" of surviving long term?
There is no such thing as unlimited capital. The faster they grow the faster they burn capital. Eventually it will run out.
But they take a lot longer to reach the same goal for complex tasks, so the difference is still very real, and the cost-savings are still very much a question of how well you manage to characterise the tasks they will do quickly and pick and choose what to use when.
I kind of agree that I think the cheap models will eat away at the moat very effectively, but if it doesn't seem more capable to you, you're not giving it complex enough tasks to see what they can do.
(FWIW, I've burned billions of tokens on each of Deepseek, Kimi, GLM5.2, GPT, Sonnet, Opus, Haiku using the same harness, and we've kept stats on cost per task)
Could you elaborate on this steep price that you have in mind? What does it consist of?
I hear what you're saying but I'm not sure I buy it in the context of this thread (a response to someone who is 54 and has been coding since they were eight).
I am in a similar boat, having been coding full-time for fourty years. The way I use the current tools is that I own all architectural and design decisions but let Claude Code fill in the blanks. I reckon the quality of the output is about 90% of what it would have been had I done everything myself, but I get a lot more done (easily 3-5X).
Will I forget how to write a "for" loop just because I haven't been writing many of them by hand lately? Those skills are so deeply ingrained that I seriously doubt it. I can ride a bike after a multi-year break, or converse in a language I haven't regularly spoken for several decades. Or write using pen & paper even though I hardly ever do it. I don't see why coding would be any different.
I also am not about to forget how to for(;;), that said, as a result of some years invested in aligning old pre WGS84 mapping with modern GPS and improving digital mapping, there are fewer people per capita with the skills to navigate via paper maps in the absence of GPS.
Old farts coding since age 8 (in which I include myself with a decade+ over a sprightly young 54) will retain coding skills for as long as they apply them - the fear is that fewer and fewer others will develop and exercise such skills due to AI.
It remains to be seen if that's a bad thing long term.
What I am worried about is us becoming dependent on tools that we as individuals neither own nor fully control, and gradually losing our ability to function without those tools. This, I think, is a huge societal risk.
Technical debt due to accumulated excessively verbose, badly architected, often redundant, feature-bloated code which always looks good, even upon earnest review, but actually sucks and becomes extremely difficult to maintain in ways which are not obvious in code review. The issue is this: your tooling can help, and can make you feel better, and you might think you wrote all the prompts and made all the tools to mitigate these issues, but you haven't. If you're not consistently seeing it generate code that is very very close to the way a skilled senior dev such as yourself would have done it (with similar line count, etc), that is a red flag even if the code looks great and works.
I can only judge from my own experience but with or without LLMs, these are the codebases that I have worked with during most of my career. To me, much of the question is whether LLMs produce worse code than the me and my colleagues have done in the past and I don't think that's the case. It is however very common that people hold LLMs to a higher standard than human colleagues and then it's not a useful comparison.
It came up with a correct LC-hard tier solution that involved dynamic programming, and was essentially an unreadable dense mess that was impossible to reason through as a human.
It worked, but it was so bad, that I sat down and realized after a bit that with maintaining a small cache, and being very particular about how the nodes are traversed, I reduced the solution to like a 10 line modified DFS, that I could understand.
I do the same with the LLM. I tell it that solution is convoluted and hard to understand, if I have a concrete suggestion I suggest one, otherwise I ask it for ideas. We get there just like I do with humans
This is in the interest of big AI companies: if they quasi-monopolize the skills entire sectors of the economy need in order to function, that will be great (for them).
Everyone keeps comparing this to compilers, but I don’t need a multiple-hundred dollar subscription to use LLVM. And people didn’t stop understanding how computers work either, just because they used C. And yeah, maybe local LLMs will become the norm, and I hope so. But market forces (hardware prices) certainly are working against that right now.
But we could build much better tooling around keeping the agents honest. The problems you are describing are absolutely real and I see them every they.
One friend of mine had almost a mental breakdown when he just went ahead and drilled a bug producing Claude to the point that it itself admitted it was “a piece of shit”. He knew that arguing with an LLM agent is more than useless, but it was cathartic for sure.
When I encounter a situation like this I always go down to - have I done everything I could to catch these errors in my automated validation, and update it as needed.
Agents are also more than happy to spend tokens refactoring, once you have such a test harness be good enough, producing successively better and more general abstractions is quite easy.
The old rule of thumb of “make it work, make it fast, make it pretty” still applies , just with much much faster iteration speed.
It seems with agents people have forgotten the last 2 steps since they produce a _working_ solution, and it might be hard to justify spending time “cleaning it up”, but this still remains essential.
If you don’t use a skill, it’s like a gene a species doesn’t need anymore, it will atrophy.
Is that bad and if yes, why? Skill atrophy is not intrinsically bad. I don’t know how to make tinted glas for church windows and I will never learn it because there are machines doing it now.
But I would for example think that critical thinking would be a catastrophic skill atrophy. As far as I know, there is no proven link though (and one would have to define what is “critical thinking” in the first place). Writing assembler without any autocomplete, I’m not so sure it’s such a problematic skill atrophy.
As far as I’m concerned, so long as we can be happy with AI we can run locally, AI is no different to the rise of scripting languages or the pocket calculator. It’s only problematic if the calculator is rented to you as a service.
I just did a big refactor with opus, it went ok, some bugs. The normal stuff. One of the bugs was in a part of the code no longer needed, which Opus had just filled with comments more or less. Asking it fix the bug worked, but then I really looked at the code and realized just that, this is pointless now.
I’ve only been coding for 20+ years so I might be more susceptible than the author, but I’m quite terrified about losing skills in writing code, but also designing good structure, coherency and system overview. These are the things people claim you need more of with LLMs, but is what you outsource the most, even if you think you are describing it in detail.
We are all collectively growing the skill of complacency and laziness though, and those are not great ”skills” to have. And I’m just as guilty as anyone.
Since automatic memory management became a thing memory management and pointers knowledge atrophied[1] across the workforce (although not nearly to the same degree).
I think the pattern here is that compilers almost always output better machine code than humans, automatic memory management doesn't output better machine code than skilled humans can very (especially with modern languages that give you a lot build-time safety checks).
And even then, there is still demand for assembly knowledge in the workforce, it is just very niche.
I don't think LLMs will ever be good enough to "almost always" output better code than humans. But, like automatic memory management, it will likely make some types of programming more niche.
The key thing here is that compilers are deterministic, deterministic tools have way less variance in output quality. Automatic memory management is not as deterministic as a compiler because it happens at runtime. LLMs output build-time code, but the can be drastically different if I sneeze too hard.
[1]: as in % of the workforce, not absolute numbers. Hard to get exact figures on this, but I think we have more experienced people actively using Assembly today than we had before compilers became the default (late 80s). We probably have more active C/C++ programmers today than before Java became popular (early 2000s).
At the end of the day there are goals achieved with coding. Coding is a tool to reach either your business needs or some personal aspiration.
When it comes to businesses I don't think a business cares if you used the best stack possible, or you've written it in assembly, as long as it works. Judging from the biggest coding drivers out there, most of the code produced globally and the biggest apps out there have had skilled engineers writing code but its not always perfect. As long as it works. Lets not forget that the web is build in php and js.
So again my argument is that, are you atrophying a skill that is going to exist in the next 1 to 2 years, or is everything going to shift towards LLM code writting.
Personally I think that LLM code writing is the winner, whether we like it or not, it accelerates business objectives, which at the end of the day its what is the deciding factor.
And yes I do miss the days I was writing code and I was solving complex problems myself.
This is your opinion and I even share it, but there are many people here for whom writing the code was/is the whole deal. You would not have languages and heck - even editors! - holy wars otherwise.
How many of them are driving goals and businesses though?
Like if we take for example an ex coder like Gabe, someone from his team comes up and says to him "we can launch this game in 1 year by using LLMs, codebase will be meh/okish" or "we can launch this game in 4 years we'll hire the top engineers and write the best software piece ever, a technical novelty".
I already know his answer... even if his answer was the 4 years which won't be, his board would disagree.
So yes there are people that love and enjoy writing software, but the truth is that business is leading software, not the other way around.
For me personally nowadays I don't need languages, I am so deep into coding using LLM's, not vibing, that I don't really mind on what is written. Also we are just at the start of this thing.
I usually start a task with an LLM and then do small refactors using the LLM and then do some manual refactors before I am done. But often for more complex tasks the manual refactors are quite large.
Maybe it is because they can read walls of text so easily, so they output walls of text that are hard to read for humans.
I feel quite sad because a lot of my fellow colleagues are not putting this extra effort in to make things easier to understand by humans. PR review is basically me just doing this extra effort for them and their LLM implementing my comments.
And that is when I can even pinpoint the bad taste in the code structure, sometimes it is not something you can easily describe in a PR comment besides "no human would structure the code like this".
Yes, some skills will atrophy, but the learning curve for LLMs is also steep and you will acquire new skills that will pay off the costs many times over.
We see this in discussions like these where you have people running the gamut from using them as glorified auto-complete or babysitting them (usually a net loss in speed, though it'll feel less draining) via people running multiple agents in several different tabs (a gain) to people prompting for harnesses rather than tasks, and putting the agent in the resulting harness (where the multipliers come in) and even people at the peak of experience with them today are only scratching the surface.
I'm very aware that just as my assembler skills are not what they once were, my skills in the languages I'm now writing less will not be what they once were a few years down this line.
But I produced far more before I started using LLMs through the force multiplier of modern languages and frameworks than I did in assembler in the 80's and 90's, and I produce far more now with LLMs, and I will produce even more in the future by learning how to take advantage of new capabilities.
I have Claude refining a system that wasn't tractable a few years ago in another terminal as I'm writing this. I don't care if it would take me a bit longer to get back up to speed on a C codebase again if I was stripped of all access to LLMs any more than I care if it'd take me a bit longer to get up to speed on programming assembler on a Commodore 64.
I have yet to see it, but OK
Either measure it or it sounds like a conspiracy theory
Seniors will be able to stay in the game much longer than before, mark my words.
When an LLM is making a bad design decision but the engineer doesn't have the experience to spot it AND the consequences don't become apparent until much later (which is often the case) -- it's kinda hard to learn.
Not disagreeing that LLM's are a force multiplier, but I highly doubt whatever value will end up finding multiplying in the next generation of seniors, at this rate. It's surreal to me that I have to point out that recognition AND recalling are both necessary components of skill acquisition, because humans largely knew this since the dawn of education.
If your "force" is above 1 then its ok to have AI power up your force. 2.3 to the power of 3 is 12.169.
But if you're a beginner and your "force" is bellow 1 so power upping this makes it worse. 0.2 to power of 3 is 0.008
Seniors should be paid to actively introduce juniors to the trade over couple of years. No more bootcamp entry.
And it would be significant $ for senior to agree to expend his time and energy on software engineering apprentices. There would be also very limited number of places with good seniors. Exactly like medicine for a long time now.
In fact it is already happening in some companies I know about - seniors geting their bonuses tied to juniors being under their wings.
Instead lets train the contractors of an IT sourcing company and then we don't need you.
This is all to say, we as a company are using AI a lot in all possible corners, but thankfully our leadership isn't schizophrenic and isn't mandating everyone hit token limits or whatever, it's more of a "Let's see what works and what doesn't" type of thing, and we measure a lot of statistics. Nobody here really cares whether LLMs are the next coming of Christ or not, as a company there are many people (even in SLT) that are indifferent to LLMs, and many who are reasonably hyped.
I wish I could link to the actual document we were all shown since it has a beautiful breakdown of the methodology and a fine-grained breakdown of the stats and the categories measured, but in the grand scheme of things, ALL the AI tooling we have implemented (at least on the engineering side of the equation) has contributed to a total of... drum roll please... 7 (seven) Percent overall productivity increase! The most productive teams saw a productivity increase of around 20%, while some teams actually saw drops in productivity into the negative percentage points. My team, none of us really give a shit about AI and we're somewhere in the 3-5% range on certain categories of tasks, which I'd say is a fairly good assessment.
Productivity here is measured in many ways, including but not limited to speed of MR review and merge times, feature/ticket/roadmap closure/delivery, rollback/revert incidence rate, how often people interact with the MR review bots and implement their suggestions/fixes, how many times people check back on AI transcriptions/meeting notes (hint: Nobody looks back on any of it, it's all just noise that gets generated and never actually referenced outside a few extremely rare cases) and many more things I'm forgetting. It is an imperfect number of course, because measuring productivity in engineering is a sisyphean task, but in my opinion it is accurate to the reality on the ground and outside of all the hype and marketing bullshit.
So, I remain thoroughly unconvinced of these personal anecdotes of people being "massively" more productive, especially once you factor in the fact that we now have a 2000EUR budget/month/dev for all the AI tooling, those productivity numbers start looking pathetic once you factor in the costs (which are only increasing as the AI companies need to start recouping the gazillions they've burned). Some teams have started begging to disable coderabbit and other similar tools in their MRs because they're producing nothing but walls of noise that makes reviewing any MR a nightmare of sludging through endless slop of useless bullshit, ours included.
I'm surprised no one tried fudging numbers though. That's usually what happens IME.
Oh some of the "AI Native" teams tried in the past, before the large scale statistics came out. The funniest one was the team responsible for rolling out AI to the whole org citing numbers like 500% more productivity, compared to what? Who knows, since that team didn't exist until relatively recently.
It probably helps that we're in a highly regulated field (banking), so by default we don't chase trends, especially not ones that are potential regulatory nightmares :p Things go a bit slower, which has its pros and cons, one of the pros being that you're forced to consider the long term a bit more and you have room to plan things out and conduct reasonable experiments.
Its like drug that will give you few years of great high, and ruin rest of your life afterwards. Use it by all means, I don't care about your output, nobody here does, you do you.
I do care about my long term skills, which aren't about piping some llm outputs. My employer ain't dumb fuck who is pushing for llms at all costs as much as possible. Anyway, most of my day work are processes, discussions, pushing things through - llms can't do a shit here, its personal conversations, connections, often psychical contact to get things done on time. Startup world would be different but I am as far from that unstable environment as I am from say gaming industry, just not worth my time outside SV area.
So I just use llms to verify my coding results, they are fine for that, but I do the creativity. Its by far the best part of my software dev work, why the heck would I be automating that away? Its like automating sex away so you can have more time... reading HN or some other way to just waste time, dumb approach from all angles.
Of course this changes if one is working on personal projects, self-employed, small startup etc but most folks here are not in that category.
Getting another agent to validate the first agent is a tower of sand.
> my velocity is apparent at https://github.com/pmarreck/)
Forgive me, but the active repos all look like reimplementations of existing good open source code (which of course is ideal training data) - rm_safe has rip for example. Or prototypes. Is there anything that actually has a user base > 1?
(And I say this as someone who agrees with you that it's garbage that these companies are trying to legislate their way into an oligopoly.)
Anthropic has gone past fearmongering and well into terrorism. I think people on Hacker News should not recommend working with terrorist orgs.
Or the largest ad company in the world (Google)?
I've had Claude Code running a /loop for the last week driving down complex crashing bugs in a prototype compiler entirely unilaterally. I occasionally glance over.
A few of those crashing test cases were ones I've spent more than a week trying to track down myself. I have 30 years of experience of doing this.
It's worked 24/7.
So far it has fixed over 500 of them.
Will there be technical debt? Yes. But nothing that remotely compares to the cost I'd have incurred of fixing all of those myself.
It is hard to reconcile those gains without thinking that if people are saying these are not a net gain, they haven't really tried learning how to get the full benefit. If you sit and watch a model work and keep intervening all the time, then sure, they're not going to be a net gain.
In practice, I tend to just use the default on Claude Code that works well enough. But I wonder to what degree other users really play around with these settings to optimize for their project.
I haven't used them in a while so my info may be out of date, but they tended to track whatever models were the best and auto-use them for each task (eg, one for planning, subagent for a code search, other frontier for implementing). Their CLI seemed very well thought out to make you do things "the correct way" -- for instance, `/handoff` instead of `/clear`.
They are often used for reading code though.
To expand on this, while the "big model to write a plan, small model to write the specific code" idea is quite common it trips up on edge cases.
In theory the flow works like this:
- small fast models read lots of code, and pass details to the large model to write a plan
- large model takes those details and writes a detailed plan
- medium models write the code
The issue happens when the medium model hits something that the plan didn't take into account (which happens a lot - the big model didn't actually read the code). Then it has to either guess, or pass back to the large model.
If it guesses, the plan usually starts to fall to bits.
If it passes back to the large model, inevitable the large model has to start reading lots of code. In that case you are paying the expensive tokens to read so you might as well have it write the code too (many less tokens are written than are read)
It might be possible to get this to work, but I haven't seen anyone who has tried agentic work with frontier models be satisfied with this hybrid setup.
I'd note that Amp (mentioned above) is probably the leader in using multiple providers in a coding agent but still uses frontier models to write code.
That's not something I understand very well. The less expensive models will quite happily chug away at tasks, if the codebase is well-structured (small files help a lot) and your instructions are clear. In contrast, I've never seen a large model turn bad instructions (instructions that would cause a human to think before starting) into a result I liked. You can run small models almost 10-100x as long for the same price in dollars, which covers a lot of correction and adjustment.
Why does everyone say the trade-offs are rarely worth it?
I think the distinction is here.
I expect my agent to build from product level descriptions. This might include specific special cases that I call out, but will rarely highlight existing special cases or edge cases - they already exist in the code, and I'd expect a programmer to make sure that behavior continues to work.
If a feature hits lots of these edge cases, the weaker model that is reading the code (aka Haiku) won't understand their significance, and will report back to the planning model incomplete or incorrect information.
The planning model (Opus - which hasn't actually seen the code remember!) will build a plan that is incorrect or incomplete and delegate coding to the mid level model (Sonnet) which will do it's best to make things work, without understanding the overall picture.
This is how you end up with slop - for example Sonnet reimplements things that already exist because it found one of the edge cases, but Opus had never known about it because Haiku didn't understand it.
It's possible that the new "agent teams" feature in Claude code can help with this. That keeps each agent alive with its context so they can ask each other things, but I haven't tried that enough to be sure - let alone with the specific model mix like this.
In your case, you are giving the Sonnet model specific instructions for what to implement mindlessly. I'd expect that to work well!
But that's not the same as the agentic workflow many other are using.
Understandable frankly.
I'm not going to play around with thinking level every request because the goal is to make me save time not spend it in a different setting menu.
Playing around with learning the differences is incredibly helpful to schedule on ones calendar weekly for an hour or two, while saving links throughout the week to try out.
I might just be having fun with models, but I have actually noticed their capabilities vary somewhat, and so my (perhaps vain) hope is that by using both, one can catch each the other's blindspots. It's still unclear to me if that's consistently happening, but I am making substantial progress in my personal and professional projects, so something seems to be working.
At the same time, I’ve invested in tooling that prints and lints architecture I want, so which model is less of an interesting decision, because the results tend to be very close.
I've done variants of this a number of times, but feel like it was a generally waste of my time to then have to compare them and write up which parts I liked or disliked: if the output is something substantial, each will have its pros and cons. Clear-cut wins aren't very common. Of course it could work well if we automated the whole thing with an orchestrator; you just need a model with actual good taste (according to your own preferences) ... so we'll have to compare all the models to find that one
For Opus 4.8 training with overblown internal dialogue and second opinions - Max effort burns just tokens and wastes time without much value. Spinning wheels.
Now that the ban is lifted, max effort Fable 5 is gonna solve this problem quite neatly. Fable to plan and review, Sonnet for the implementation.
Wait, never mind that. Subscribers will only have Fable for a week.
I am getting things done. I've made major progress on my projects, and even started new ones. My most requested tasks are: code review, brainstorming and research. The fixation on tiny details is exactly what I'm paying for.
Juggling between all different models/agents is quite simple with Zed.
A caution about OpenCode Go though, the entire company seems to be run by AI so there's lot of billing related issues with zero support. I subscribe new every month as I lost money due to double payment with automatic subscription.
For non coding related tasks I use local models.
P.S. If anyone is interested to read more about my setup, let me know I'll publish a blog post.
What sort of hardware are you using to run local models? And how do you use them?
The Z.AI is a bit wonky, so now I'm moving to Openrouter for Qwen+Kimi+Deepseek?GLM
My summer project is to figure out a proper agentic system where a "big" model does the planning, but automatically uses a cheaper one for the grunt work. Having Opus to config edits is just stupid :)
My company pays for the tokens so I don’t care. Biggest model and max everything. The slight risk of a smaller model making a mistake is more expensive than just running the bigger model all the time.
But I'm playing the long game you see. The tokens will get expensive and the monthly subscriptions will either go away or also get too expensive.
Then companies want efficient token use and cost control - and I'll already know how to do that =)
I trust neither for general knowledge and I still find Opus giving me answers that are completely BS. But the token spend for Q&A is nothing compared to coding, so I always use Opus + a lot of thinking. For coding, I find Opus to be better value/token but I haven't done any sort of rigorous test.
- For Claude.ai subscriptions I think Sonnet is much cheaper than Opus. This is why there was a "Sonnet only" usage bar for Max tier for the longest time.
- For some tasks the sheer amount of raw input tokens is the most important. For example multimodal computer use tasks. You can't make them any more efficient on Opus by turning down the reasoning, so a cheaper model like Sonnet is useful for them
it's still there. I still don't totally grok why I can't use all my tokens on Sonnet if I want to... maybe that signals something?
I don't really believe this however, because so much time is spent fixing up after models, that a slower but more intelligent model is a net time saver in my experience.
[0]: https://aibenchy.com/compare/anthropic-claude-sonnet-4-6-med...
It makes some sense, as models are trained more and more with reasoning, than without.
[0]: https://aibenchy.com/compare/anthropic-claude-sonnet-4-6-non...
However, I am also confused about market positioning. Too expensive to perform daily tasks - open souce models are much cheaper - and not frontier model to address complex real world problems.
Rarely used Sonnet btw.
The graph shows that Opus is cheaper than Sonnet for the same performance. Unless I am suffering a cognitive blindness thing right now.
Alternatively you can draw a horizontal "constant performance" line and see that Opus is cheaper for a given performance level.
There is a real advantage, especially for businesses, in using an off the shelf solution from a corporate provider.
Personally, the advantage of not having to set up multiple solutions from multiple sources outweighs the cost of a $20 a month subscription. Think about why a lot of consumers prefer Apple devices over Linux. There are a lot of advantages to Linux, but "never having to think about my tools" is its own advantage.
The graphs show parts of the cost/performance pareto frontier occupied by Opus 4.8 and others occupied by Sonnet 5.0. If Opus 4.8 was strictly better at cost per task like you say, by definition the entire frontier would be occupied by Opus.
So neither is pareto-dominant over the other. In contrast, Sonnet 5.0 is Pareto-dominent over Sonnet 4.6 on those graphs.
But the entire frontier is occupied by Opus under any reasonable interpolation scheme (piecewise linear which is what they've done, and most reasonable spline or polynomial fits would also lead to the same result) over the overlapping x values for which both are defined.
Under that interpolation scheme, for x > ($ cost of Opus low effort), Opus is Pareto-dominant over Sonnet 5. You can see this by picking any point on Opus's interpolation and realizing that you get strictly worse by switching to Sonnet for the same x value or the same y value. Meaning if you want to pay the same $x then you get a worse y, or if you want the same y you pay more $x.
If you mean extrapolate, at that point you're just making up data. The available effort levels are discrete and covered totally by the benchmarks. You can draw on the monitor with a sharpie to show a "ultra-low" effort level for Opus that scores better than Sonnet "low" at the same price, but it doesn't magic the ultra-low effort into actual existence.
(Anyway, the blog post now has an errata and a graph that shows substantially better relative performance for Sonnet 5.0 than the original graph.)
It was a claim that applies to a range of x-values where both curves are defined.
Of course if you go beyond those x-values where only one of the two are defined, then trivially the one that is defined constitutes the Pareto frontier in that region. Which is what I understand to be your point?
You could make it true by artificially dropping some of the data points, but, like, why?
(Again, this is moot given the updated graph.)
> Of course if you go beyond those x-values where only one of the two are defined, then trivially the one that is defined constitutes the Pareto frontier in that region.
Not so! It's only sound to do that at the low end of the cost axis (x) or the high end of the performance axis (y). You can't do it at the low end of the performance axis or the high end of the cost axis.
It would be great to see these charts with the promotional pricing just because it’s here for about two whole months.
I guess I could get Sonnet 5 to do it.
Does anyone else have any review token saving measures?
Assume it to get deprecated sooner rather than later.
I guess it's probably a lot cheaper for them to run, and it cuts costs for them. Seems disingenuous, though.
And what (avaiable) model do you trust to go off on its own?