> You tweak the last 20% by hand.
> How much of a productivity increase is that? Well jeepers, if you’re only doing 1/5th the work, then you are… punches buttons on calculator watch… five times as productive
Except, it’s probably four times as hard to verify the 80% right code as it is to just write the code yourself in the first place.
The error rate needs to come down significantly from where it's currently at, and we'd also need AIs that cover each others weaknesses - at the moment you can't just run another AI to fix the 20% of errors the first one generated because they tend to have overlapping blind spots.
Having said all that, I think AI is hugely useful as a programming tool. Here are the places I've got most value:
- Better IDE auto-complete.
- Repetetive re-factors that are beyond what's possible to do with a regex. For example, replaing one function with another whose usage is subtly different. LLMs can crack out those changes across a codebase and it's easy to review.
- Summarizing and extracting intent from dense technical content. I'm often implementing some part of a web standard, and being able to quickly ask an AI questions, and have it give references back to specific parts of the standard is super useful.
- Bootstrapping tests. The hardest part of writing tests in my experience is writing the first one - there's always a bit of boilerplate that's annoying to write. The AI is pretty good at generating a smoke test and then you can use that a basis to do proper test coverage.
I'm sure there are more. The point is that it's currently incredibly valuable as a tool, but it's not yet capable of being an agent.
It's a very invalid point. Perhaps in your life you've never encountered people who cheat and later change their ways, but it's fairly common for them to change their ways.
Would you also say it's ok for people to steal because they might one day not-steal? So an increase in robberies does not hurt? I am not sure I understood you correctly.
Several people have asked you, and you still haven't explained why it's OK to waste time with "non-cheaters" and not with "cheaters".
> Would you also say it's ok for people to steal because they might one day not-steal?
No. Unless stealing is permitted, which it usually isn't. LLMs, OTOH, are merely a tool and their use has no morality attached to it. It's you that seem to attach some morality to LLM use.
(Of course, if your company disallows LLM usage, that's another story).
> I am not sure I understood you correctly.
Your comments seem to make the assumption that LLM based code is shittier than what a typical poor developer would write, which I pointed out in another comment is often not true.
Just treat the code based on its merit. If it's utter crap, tell the developer it's utter crap. Just stop second guessing whether he wrote it or an LLM did. There is a threshold below which even I won't review code.
Aside: it’s confusing that parent commenter used “cheating” in a completely different way than the title of the article.
You don't need to. Focus on the work, not the person.
I recall a coworker screaming at a junior developer during a code review, and I set up a meeting later on to discuss his reaction. Why was he so upset? Because the junior developer was being lazy and not trying.
How did he know this?
He didn't. He used very dubious behavioral signals.
This was pre-LLM. As applicable now as it was then. If their code is shitty, let them know and leave it at that.
On the flip side, I often review code that I think has been written by an LLM. When the code is good, I don't really care if he wrote it or asked GPT to write it. As long as he's functioning well enough for the current job.
I work for a university. I have literally caught student workers putting the result of an LLM in the code without reading it, because they did it during screen sharing. You can't teach somebody who does not want to learn (because they just want to pass the semester and have no incentive to try hard). Now it's just way more easy to pretend to have put in effort.
It's also natural to ask why you think the growing fraction is because they cheated.
Look, I don't know how old you are, but the quality of the average developer has been dropping for decades. Prior to GPT, I routinely had to deal with developers who refused to learn anything (e.g. version control) and needed hand holding.[1] They didn't want to read man pages, etc. It's silly to suddenly get upset because LLMs are yet another reason for this phenomenon.
If there's a problem, it's hiring practices. Not LLMs.
> Especially because you can't know which fraction tried their best and deserve patience and mentoring, and which fraction are lying and wasting your time.
Given your comments, it's trivial to see why they would lie to you. They wouldn't to me, because I'm not going to discourage LLM usage.
> I work for a university.
Even before I read your comment, I was about to point out how your stance is valid in the domain of education, and not the workplace. And you just validated it :-)
The concept of cheating when it comes to education is vastly different compared to the work place. The goal of teaching is to learn, and even more relevant, for teachers to evaluate how well someone is learning. If they're using LLMs (or calculators, for that matter), then it becomes harder to evaluate. The goal is not to get something done, but to learn. That's why the environment in school is very constrained, and very different.
The worst jobs I've had are when people failed to realize that they're now in a different environment and they continue to apply those principles where they don't make sense. The main goal at work is to get something done, regardless of the tools.[2]
If someone tries to put good code in, stop caring if it comes from an LLM.
If someone tries to put bad code in, stop caring if it comes from an LLM.
[1] And yes, today's LLMs do a better job than many crappy developers I've had to put up with. Had LLMs existed back than and they used them, I'd be a happier man.
[2] As long as you're not violating laws, copyright, licenses, patents, etc.
>100X isn't typical, I'd guess typical in my case is >2X, but it's happened at large scale and saved me months of work in this single instance. (Years of work overall)
GPT-4o can only handle rudimentary coding tasks. So I only ask it rudimentary questions. And it's rarely wrong or bugged. o1-2024-12-17 is much better, it can handle "mid" complexity tasks, it's usually never wrong on those, but a step above and it will output buggy code.
This is subjective and not perfect, but when you use the models more you can get a sense for where it starts falling apart and stay below that threshold. This means chunking your problem into multiple prompts, where each chunk is nearing the maximum level of complexity that the model can handle.
https://github.com/golergka/hn-blog-recommender
Basically 95% of the code is written by Aider. I still had to do QA, make high-level decisions, review code it wrote, assign tasks and tell it when to refactor stuff.
I would never use this workflow to write a program where absolute correctness is top priority, like a radiation therapy controller. But from this point on, I'll never waste time building pet projects, small web app and utilities myself.
You'll probably get paid more and work less if you know what you're doing as opposed to being yet another disposable I'll handle the business side guy.
An LLM might be able to hammer out the unit tests, boilerplate react logic, and the express backend in 5 minutes, but if it collapses on the "20%" that you need actual assistance with, then the productivity boost is significantly less than suggested.
I dont disagree but here's another spin on what you observed.
Kernighan's law says you have to be twice as intelligent/skilled when debugging the code as you were when writing it.
A LLM centric corollary would be -don't have the LLM write code at the boundary of its capabilities in your domain.
It's a little hard these days to know where that line lies with the rate of improvements but perhaps is an intuition people are still new at with LLMs.
Why would that be true in general? You think it just so happens that a new tool gives an X speedup in ability but costs an exactly equivalent X slowdown in other places?
Switching the job of programming from designing and writing code to reading and verifying code isn’t a good trade-off in terms of making programming easier. And my first speei node with AI tools has done nothing to change my mind on this point.
To write code that does work, you generally have to fully understand the goal and what code to use.
So unless you just flat-out don't know what code to use (ie. libraries, architecture, approach etc.) it's going to be no faster and probably less satisfying to fix LLM code as it would be to just do it yourself.
Where LLMs do shine in coding (in my limited experience) though is discovery. If you need to write something in a mostly unfamiliar language and ecosystem they can be great for getting off the ground. I'm still not convinced that the mental gear shifting between 'student asking questions' and 'researcher/analyst/coder mode' is worth the slightly more targeted responses vs. just looking up tutorials, but it's more useful scenario.