I appreciate that the OP puts his finger on the problem. I think he's right. I wish there was more optimism or ideas about how to solve it, though. How do I fight against it as an engineer? Or, how do I become a "good" engineer? It feels like sometimes the only solution people have is effectively "don't use it."
You're right. But I used it specifically because of the open letter calling for open models — or at least for not banning them — which went around recently and represented a very large number of tech and AI companies. Anthropic seemed the only exception.
Isn't this the opposite of what everyone is saying should happen? That is, lead with open models -- or at least "openness" and don't leave the capabilities in the hands of an elite few? Did they learn nothing from the Hugging Face incident, where HF wasn't even able to use the models to defend itself from OAI's attack?
A compiler is not an LLM, and I do not want to equivocate, but there are aspects of similarity. We do not assume people read the bytes of machine code to ensure it’s correct — there could be mistakes. We also write and run automated tests to ensure the code outputted from a compiler and an LLM behaves correctly. At some point, we won’t have to literally read every byte of code that comes out because we have a reasonable assurance that it’s correct.
> Anthropic has never advocated for a ban on open-weights models.
This is not an unqualified never. The very next sentence makes a qualified statement: "Open-weights models that don’t have dangerous capabilities are a public good". That prompts the question, what about ones which do have "dangerous capabilities"? Are they not a public good? If not, then should they be banned? Who gets to decide on the definitions of these terms?
Self-hosting a language used to be standard practice. These days that's become a bit more controversial, but I definitely wouldn't lump that in as a sign of obsession.
I know everything is told from a perspective, but I find the Marxist “aristocrat / bourgeois” lens rather tired and severely anachronistic. This is told as if Luther and the Magisterial Reformers were just “siding with the aristocracy” and didn’t have any other reasons for opposing a violent uprising on anarchy. The Reformers were trying to effect spiritual and theological change. But when the magistrate, who was supporting the Reformation, encountered radicals who used those ideas for violence and their own ambition, the Reformers had a conundrum. Carl Truman writes in his book “Luther in the Christian Life” (Crossway, 2015), ‘This latter concept, often known as “the priesthood of all believers” or “the general priesthood of all believers” was explosive in Luther’s day because it struck at the very structure of medieval piety. … The problem became acutely significant for the Reformers, and especially Luther, in 1525, with the Peasants’ War. At that time, radicals used Luther’s language of freedom as part of the rhetoric of rebellion, for understandable reasons. Such language resonates powerfully with the oppressed and disenfranchised in every age. The problem, of course, was that Luther was using such words in a manner that assumed their refraction through the cross. The same applied to the language of universal priesthood: the obvious democratizing tendency of the language was popular with those seeking more social equality. Yet the anarchy of the war placed Luther in an impossible position: he was no democratizing radical; he needed to reassure his secular lords that reformation did not mean revolution; and he needed to make sure that his teaching was clearly understood.’ The simple Marxist interpretation is rubbish — the situation was far more complicated than oppressor / oppressed.
I tend to agree. But the devil's advocate in me says that then they'd be accused of not having empathy. That sort of thing might work in Europe — but for a US company, I think you have to pay some lip service to emotions.
But then you're back at the question from the OP of "why not?" Is it lack of good examples in the training data? Is there something of code "goodness" that goes beyond phenomenon that are measurable? Is it that LLMs lack in the ability to organize and think abstractly?
Isn't that how AI written software gets better too? By steering the model towards the goals of the user?
I don't know if it's a question of how many features to feed to the model, either. Of course, overwhelm the context with too many features and it will get confused. But that's where the memory management idea that G. Huntley talks about is helpful. You're trying to steer the model within its memory limits towards a certain goal.
The problem is getting it to produce "good" code. Formalizing what that means is the task of the programmer. How do you steer a model towards always, or more often, producing good code so that you don't have to do rework? That's the same problem as with a junior engineer, but the way you do it is different. Right now we're trying to do it with mountains of prompts — which sort of works but has diminishing returns — and with onerous code reviews. We've seen this get better over time, but I think some more mechanical methods will help as we figure out how best to steer the models.
> So, why can't models do software maintainability?
I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad design?" Of course the models aren't good at it ... they're not good at anything until you've tuned them and put them in a harness that rewards good edits and throws away (improves) bad edits. That doesn't explain why "models can't do software maintainability." The real question is why harnesses can't do software maintainability, and how to build a system that can do it. (I suppose that's the purpose of the ad at the bottom of the page.)
I mean that's how I personally would define ethical. But anytime someone makes a claim like that, you have to ask "by what standard." And it's usually just "me."
Isn’t that somewhat similar to everywhere else AI is being employed? More of the same — just faster? We aren’t getting novel software architecture out of AI, those things are still important, but it does help us rule out bad things (bugs, security vulnerabilities, etc) and help us focus on the important things. In that vein, AI could help mathematicians by ruling things out faster.
Just to be clear, this doesn't mean that anything on the die actually measures 0.7nm — it means that it's roughly double the density as the previous node generation. At some point the industry decided to keep talking about "nanometers" even though the actual transistor sizes have been decoupled from the node name for years.
If resold Anthropic tokens undercut even the at-cost open-weight model tokens, because they're reselling subsidized subscription tokens, then you'd have to start selling open-weight model tokens at a loss in order to match them.