Hacker News Reader

Comment by bigbuppo | Hacker News Reader

Also... if your workflow is "think hard, let LLM write it, read what AI wrote, think hard about what AI wrote, re-type what AI wrote, fix what AI wrote"...

Where in the heck are the efficiency gains? Couldn't you just drop the LLM part of it and save the company a trillion dollars in tokens?

View on HN

LLM speedups have always been predicated on not understanding the code they produce as well as if you wrote it yourself. Reviewing even well written code to understand it in depth, has always taken longer than simply writing it yourself - let alone sloppy LLM code that you have to fix

People are, in heavy LLM systems, realising that the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done. Any senior engineer knows this all too well

Its why I suspect we never tend to see any longer term LLM productivity stats being published, and why I extra suspect that LLMs have failed to achieve any kind of penetration into open source code. If you take away the pressure to produce bad code, writing by hand clearly wins massively in the long term. Its classic short term gains for a long term penalty

> the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done

Perfectly said. It was true before LLMs too.

As I've been reviewing more and more LLM code, I've noticed that while good developers can use LLMs to produce good code maybe 1x-5x faster (depending on circumstances), bad developers can use LLMs to produce bad code 1000x faster. And if you are collecting metrics that only capture that 1000x number, I bet you feel great about what you're doing.

> People are, in heavy LLM systems, realising that the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done. Any senior engineer knows this all too well

I've been pushing against this "maximum LLM driven speed" in our infrastructure as well and been working on a middle ground:

We can generate a change to Ansible or terraform code in half or a third of the time, sure. But we don't use this to make three times the changes in the same time frame. We rather use the freed up time to discuss the change, the context and the affected systems with 2-3 engineers maintaining them. And yes, at times this means us three are sitting around half a day discussing and drawing diagrams about our systems.

I'm just happy to work in a company understanding the value of speeding up less than we could in the feature direction, but investing this time in the direction of control and understanding of the infrastructure.

Yup, this is how I see teams who care about quality, design and architecture seems to use LLMs, not to "produce more and faster" but to retain same speed but with a lot more confidence and reliability. Hoping this will spread eventually, some companies seem to take a more... hazardous approach to the whole thing.

"""People are, in heavy LLM systems, realising that the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done."""

Very much this. It is also true for non-LLM code bases. I still remember my first job at a company with a bigger codebase than my typical "university project code base". It was a full fledged ERP system and just learning the lay of the land and understanding what lives where and why took quite a bit of time.

Interestingly, I think LLMs can be used for this "self-onboarding" and codebase-browsing/understanding. I've experimented a bit with treating an LLM as a guide of a bigger codebase it has "ingested" and having dialogues about it. Works reasonably well, I'm curious if there are companies out there that use this approach.

> LLM speedups have always been predicated on not understanding the code they produce as well as if you wrote it yourself. Reviewing even well written code to understand it in depth, has always taken longer than simply writing it yourself - let alone sloppy LLM code that you have to fix

I am at least 3x more productive by letting an LLM write my code and I review and understand every single line it writes.

Also, Fable 5 came up with some amazing solution and designs that I would have never been able to do myself.

I can say with 100% surety that my current project at work is MUCH better with an LLM compared to if I wrote it myself.

I don't think this personal opinion holds well when contrasted with reality. The truth of the matter is that in moderately large codebases, which is the norm in any production setting, at best you have a T-shape understanding of the system: in specific areas you might have somewhat deep understanding, but for the vast majority of the system you have at best a high level understanding of the software architecture.

This is why software engineering frames things like design patterns, principle of least surprise,single responsibility, etc as premium design and code quality traits: they allow developers to effectively extrapolate their high level understanding of the project onto components they know nothing about.

In any software development setting,you will find that the most successful and even senior engineer is that which successfully and skillfully manages this uncertainty, and is able to hit the ground running on environments they never touched. I had a colleague who once was a senior software development engineer at a FANG that mastered this art, and called it JIT onboarding.

So why are you pretending this is something new or novel?

I think this blend of criticism only serves to allow inexperienced developers to stand out by complaining that a very mundane aspect of working on projects as a member of a sizeable team is somehow a novel development caused by LLMs.

And don't get me started on the "slop" nonsense. Have you actually browsed through code written by people?

Thank you for writing exactly why people who complain about LLMs making it harder to understand what’s going on are so off. Nothing helps more than asking LLMs to tell you exactly what’s going on! They are insanely good at that. I wrote the plugin system in my product, but many people added to it over a decade and today I have only high level knowledge of it. But if I need to know something about a very specific part of it, I ask Claude and now I know exactly what I needed to know. I could even write the code by hand now, and sometimes I do for the fun of it, but Claude has just explained to me, so it also knows exactly how to do it so why not let it do it?? I can understand as much as I need to by reading the code , asking questions, carefully analyzing the tests, much more than I would if I had needed to focus on every single detail that is irrelevant to the task. Sometimes I even do some minor refactoring just to try and feel like I’m “inside” the code as I used to. But I know I will likely be out of the loop again in 3 or 4 years, as I just can’t remember everything I ever did. But now I have a LLM ready to help me get into it again whenever needed again.

>So why are you pretending this is something new or novel?

Because as the OP of this thread and LLM users are rediscovering, that understanding is the most critical part of software development

>I don't think this personal opinion holds well when contrasted with reality. The truth of the matter is that in moderately large codebases, which is the norm in any production setting, at best you have a T-shape understanding of the system: in specific areas you might have somewhat deep understanding, but for the vast majority of the system you have at best a high level understanding of the software architecture.

I suspect a lot of people haven't worked on a system with engineers who've been there for 20 years working on it. There's simply no substitute for that kind of deep institutional knowledge, and it'll take you 5-10 years to be as productive as them simply because of that level of extreme knowledge about a codebase. It isn't as sexy as swapping jobs every 2 years though, so you have to go outside of the big tech bubble

I will say that yes, sadly, I have browsed through code written by people, and sadly also have browsed through code written by LLMs. Who do you suppose wrote the code the LLM was trained on? Divinity?

I'm being tongue and cheek, but as someone who uses these things extensively in work and personal projects, I want to point out that it's new or novel that generating lots of plausible code is now vastly cheaper than it once was, so the scales are quite different.

And yes, slop is an apt moniker for something that requires lots of careful prompting and configuration and guardrails (which is extremely sensitive to small changes in these initial conditions in ways that can be fairly nonobvious) to produce something that will in most cases read at least a little bit worse than if a human had written it. And it will still go off the rails sometimes!

To be clear I don't really agree that typing out the code generated by LLMs is a reasonable counterweight, it largely defeats the purpose. I agree that at a certain point you have to treat parts of your code base like a black box and enforce modularity. But this is still nontrivial, especially in more specialized domains with lots of nuance

> I will say that yes, sadly, I have browsed through code written by people, and sadly also have browsed through code written by LLMs. Who do you suppose wrote the code the LLM was trained on? Divinity?

Training is a red herring. The root cause is constraints, or lack thereof. If you lock a junior dev in a basement and force him to deliver the same features that these LLMs output, you will see exactly the same type of slop. The root cause is that junior devs are inexperienced and oblivious to best practices and guidelines and even the team's internal standards. They output code unconstrained by these guidelines and thus output big balls of mud.

> I'm being tongue and cheek, but as someone who uses these things extensively in work and personal projects, I want to point out that it's new or novel that generating lots of plausible code is now vastly cheaper than it once was, so the scales are quite different.

I agree, but we need to be mindful of what is the actual root cause. I argue it's not AI coding agente but the diverse source of these code changes, which we also see in production settings in projects managed by large teams. In teams manned by junior devs you see very much the same slop building up to a big ball of mud in a few iterations. This is nothing new. What changed is that now ai coding assistants grant everyone access to what amounts to a large team of junior devs.

I've found I can get quite a bit of use out of it as a basically ever-present rubber duck.

Using a cheap model, I usually spend < $1-$2 per day (on API billing) using it a model like.

- here's the problem I'm trying to solve:

- here's my initial plan for the design:

- is there anything I'm not thinking about or my plan is missing?

It'll occasionally pop out a suggestion that I like more than my original plan, or it'll give me some new angles to think through the original plan.

I’m guilty of using this approach too and noticed that an (obvious?) side-effect is that I get fewer wonderful, human, “aha” moments with my colleagues.

I mentioned this at a stand-up and only then did others think about it and notice the same of themselves.

This, to me, is one of the major downsides of all this: less collaboration, more individualism

I think that's true. One caveat: If you're a solo coder like me, you almost never get to talk about implementation details with anyone who (a) understands or (b) gives a flying fuck. I'm more likely to have a meaningful convo about code structure with someone on a laptop sitting next to me at the coffeehouse. So sometimes a stand-in bot to bounce your ideas off of can illuminate corners you're missing, even if it's in some ways no operating just as a chat with yourself.

> This, to me, is one of the major downsides of all this: less collaboration, more individualism

Or put another way, it's replacing collaboration and human connection with your team with a subscription service that isolates you, makes you emotionally and cognitively dependent on the service while making your performance numbers look good. I really don't like the sound of that.

I worry about the finger-on-the-scale mean-reversion aspect of this .. the LLM corralling you ever so gently back towards the median.

otoh, LLMs really could become the great illustrated primer for educating kids in science / history / languages etc.

Isn't median code often a good thing in terms of ability for others to understand it?

Yes, but also not necessarily good code(according to some). A lot of people don't buy into the "code should be optimised for reading" stuff.

Well, the objetive right now seems to be other LLMs understand it.

> Where in the heck are the efficiency gains?

It has been the opposite for us. It is now being weaponized by PMs and leads to add more work for developers and not less in terms of documentation and sign-offs.

Now stories automatically have additional 10 subtasks for documentation and tracking. These are created automatically through "skills" and the task of detailed documentation and updating those extra tasks is a chore. When complained, they just say "use AI". Every JIRA task these days feels like submitting a petition to the government bureaucracy.

PMs are happy because they did something new, leaders are happy for more tracking, developers are expected to be happy with "productivity gains" due to not having to write much code.

I stopped writing code completely about a year ago, stopped reading code completely about 8 months ago, and I feel like I stopped thinking hard about anything at work about 6 months ago. To clarify - I still produce code for a living - all day, every day. Not sure about efficiency gains but I got promoted 3 weeks ago :)

The LLM is, in many ways, smarter than you and I: it can perform feats of advanced mathematics that have eluded mathematicians for decades. It's not at all unreasonable to think that an LLM could produce an idea better than you could have, but that reading it and typing it in gives you some better understanding of how it works.

knowledge, by doing that I retrieve the knowledge from LLM, I learn faster, write code faster, do things more careful... The efficiency is the better design, better code quality, higher test coverage, less bugs. Because LLM let me gain knowledge faster than previous tools.

Besides, many companies proactively punish doing anything not by LLM.

in code you do not care enough to read, in experimental code, one off things, tests, reading a codebase with no documentation, re jobs

> Where in the heck are the efficiency gains?

I think there's some confusion in your post. You're seeing people claiming that the rate their code changes is so fast that they find it hard to keep their mental model up to date and in sync with the actual codebase. They frame this as a LLM issue, but those of us who worked in large teams will understand how it feels to be faced with a dozen changes popping up in critical areas of a code base each time we prepare to submit our own code change.

Once you understand this, you'll easily understand where all the efficiency gains are manifesting. Today's lone cowboy developers are experiencing the same type of struggles to keep up with a project that in the recent past affected projects worked on by large teams.

I mean, think about it for a second: what do you think is behind this higher rate of change?

Reply on news.ycombinator.com