Well I have the exact opposite experience. I don't know why people struggle to get good results with llms.
As a back end developer I am not familiar with the latest trends in JavaScript and CSS, and frankly I do not want to spend my time studying these. A LLM can generate an interactive web game based on my description. I review the code, it is usually okay, sometimes I suggest an improvement. I could have done all of that -- but it would take me a week, and the LLM does it in seconds. So it is a difference between a hobby project done or not done.
I also tried a LLM at work, not to code, but to explain some complex topics that were new to me. Once it provided a great high-level description that was very useful. And once it provided a great explanation... which was a total lie, as I found out when I tried to do a hello-world example. I still think the 50% success rate is great, as long as you can quickly verify it.
Shortly, we need to know the strengths and the weaknesses, and use the LLMs accordingly. Too much trust will get you burned. But properly used, they can save a lot of time.
Sonnet can absolutely get very confused and break things. And there were tasks where I had a really hard time getting it to do the right thing, or understand what I wanted. But I need you to understand: Sonnet made this thing for me in two and a half days of part-time prompting. That is probably ten times faster than it would have taken me on my own, especially as I have absolutely no design ability.
Now, is this a big project? No, it's like 2kloc. But I don't think you can call it "simple" exactly. It's potentially useful technology. This sort of "just make this small tool exist for me" is where I see most of the value for AI in the next year. And the definition of "small tool" can stretch surprisingly far.
Also your Google Drive API key is easily discoverable with about 15 seconds of looking at the JS source code -- this is something a professional software developer would (hopefully) have picked up without you asking, but an LLM isn't going to tell you that you shouldn't ship the `const API_KEY = ...` code as a file to the client, because you didn't ask.
I mean, it would have taken me a lot longer on my own. Sure it's not a huge project, I agree; I wouldn't call it entirely trivial.
> Also your Google Drive API key is easily discoverable with about 15 seconds of looking at the JS source code
No, I'm aware of that. That's deliberate. There's no way to avoid it for a serverless webapp. (Note that Guesspage is entirely hosted on Github Pages.) All the data stored is public anyways, the key is limited to only have permission to access the stored data, and you still have to log in and grab a token that is only stored in your browser and cannot be accessed from other sites. Literally the only unique thing you can do with it is trigger a login request on your own site that looks like it comes from Guesspage; and you can do that just as easily by creating a new API key and setting its name to "Guesspage".
The AI actually told me that was unsafe, and I corrected it. To the best of my understanding, the only thing that you can do with the API key is do Google Drive uploads to your own drive or that of someone who lets you that look to Google as if my app is triggering them. If there's a danger that can arise from that, and I don't think there is, then it's on me, not on Sonnet.
(It's also referer domain limited, but that's worthless. If only there was a way to cryptographically sign a referer...)
It just doesn't represent a realistic codebase. It is significantly smaller than a lot of college projects.
The current software system I'm working on now is ~2 million lines of code split across a dozen services.
AI has been pretty good for search across the codebases and absolutely hopeless for code gen.
LLMs just aren't that good yet for writing code on a decent sized system.
My point is more that LLMs can handle (some) projects that are useful. It's not just oneliners and hello worlds. There's a region in between "one-page demos" and "medium-sized codebases and up" where useful work can already happen.
[0] https://github.com/Guesspage/guesspage.github.io/blob/master...
The effort tradeoff is different for AIs than humans. Easy-to-understand-locally is more important than cheap-to-change, because it can do "read and check every line in the project" for like 20 cents. Making AIs code like humans is not playing to their strengths.
I don't think it's that bad anyways.
When I ask him to write a function that should do something much more complex, it usually do something so bad it takes me more time because it confuses me and now I have to back to my original reasoning (after trying to understand what it did).
What I found useful is to ask him to explain me what a function does in a new codebase I am exploring, although I have to be very careful because a lot of time invents or skips steps that are crucial.
It just does the job that cursor does there, but better.
Maybe us programmers should focus on making higher order programming tools instead of black box text generators for existing tools.
The real limiting factor is not so much task complexity as the level of abstraction and indirection. If you have code that requires following a long chain of references to understand, LLMs will struggle to work with it.
For similar reasons, they also struggle with:
- generic types
- inheritance hierarchies
- long function call chains
- dependency injection
- deeply nested structures
They're also bad at counting, which can be an issue when dealing with concurrency—i.e. you started 5 operations concurrently at different points in your program and now need to block while waiting for 5 corresponding success or failure messages. Unless your code explicitly uses the number 5 somewhere, an LLM is often going to fail at counting the operations.
All in all, the main question I think in determining how well an LLM can do a task is whether the limiting factor for your task is knowledge or abstraction. If it's knowledge (the intricacies of some arcane OS API, for example), an LLM can do very well with good prompting even on quite large and complex tasks. If it's abstraction, it's likely to fail in all kinds of seemingly obvious ways.
Only if that knowledge is sufficiently represented in the training data or on the web. If, on the other hand, it’s knowledge that isn’t well (or at all) represented, and instead requires experience or experimentation with the relevant system, LLMs don’t do very well. I regularly fail with applying LLMs to tasks that turn out to require such “hidden” knowledge.
If you grant that, the next question is how high the accuracy has to be before it's quicker than doing the research and writing the code yourself. If it's 100%, then it's clearly better, since doing the research and implementation oneself generally takes an hour or so in the best scenario (this can expand to multiple hours or days depending on the task). If it's 99%, it's still probably (much) better, since it will be faster to fix the minor issues than to implement from scratch. If it's 90%, 80%, 70% it becomes a more interesting question.
It's true enough that there are many tasks like this. But there are also many relatively arcane APIs/protocols/domains that LLMs do a surprisingly good job with. I tend to think it's worth checking which bucket a task falls into before spending hours or days hammering something out myself.
I think many devs are underestimating how arcane the knowledge needs to be before an LLM will be hopeless at a knowledge-based task. There's a lot of code on the internet.
i gave it a yaml and asked it to generate a json call to rest api . It missed a bunch of keys and made up a random new key. I threw out the whole thing and did it with awk/sed.
Every single example was completely useless. The code wouldn't compile, it would invent methods and variables and the instructions to go along with it were incoherent. All whilst gaslighting along with the way.
I have also previously tried using it with some Golang code and it would constantly add weird statements e.g. locking on non-concurrent operations.
LLMs are great when you are doing the same things as everyone else. Step outside of that and it's far more trouble than it's worth.
If you're doing something in a way it's not in the training data set, maybe your way of approaching the problem is wrong?
SuccessFactors is a popular HR platform and I was asking it any question and getting the wrong answer every time.
in my industry, the "training data set" won't get much farther from public code than the barebones, generated doxygen comments we call "documentation".
But in a way you're also right. The industry's approach is fundamentally wrong, making 20 solutions to a problem with plenty of room to standardize a proper approach (plenty of room where you need proprietary techniques, but that's getting less true by the month). But an LLM isn't going to fix that cultural issue and will suffer from it.
LLM-powered development may push the industry towards standardization. "Oh, CoPilot cannot generate proper code for your SDK/API/service? Sorry, all my developers use CoPilot, so we will not integrate with your SDK/API/service until you provide better, CoPilot-friendly docs and examples."
If you can't share your code with Anthropic then there is nothing to talk about. Expecting it to know non-public SDK's and docs isn't reasonable. But if you want it to help you with FOSS libs/docs, this is the way.
but i would never push llm generated code. never.
-
edit to add some substance:
if it’s someone who
* does a lot of manual local testing
* adds good unit / integration tests
* writes clear and well documented PRs
* knows the code style, and when to break it
* tests themselves in a staging environment, independent of any QA team or reviews
* monitors the changes after they’ve gone out
* has repeatedly found things in their own PRs and asked to hold off release to fix them
* is reviewing other people’s PRs and spotting things before they go out
yea, sure, i’ll release the changes. they’re doing the auditing work for me.
they clearly care about the software. and i’ve seen enough to trust them.
and if they got it wrong, well, shit, they did everything good enough. i’m sure they’ll be on the ball when it comes to rolling it back and/or fixing it.
an llm does not do those things. an llm *does not care about your software* and never will.
i’ll take people who give a shit any day of the week.
As a start, let me know when an AI can fail test cases, re-iterate on its code to correct the test case, and re-submit. But I suppose that starts to approach AGI territory.