Hacker News Reader

Comment by Bukhmanizer | Hacker News Reader

I think a lot of people would disagree with this article on HN, but I’ve yet to see too many people say it’s made their coworkers more productive. That is, do people feel like they’re getting better, more reviewable PRs?

Personally, I’ve been seeing the number of changes for a PR starting to reach into the mid-hundreds now. And fundamentally the developers who make them don’t understand how they work. They often think they do, but then I’ll ask them something about the design and they’ll reply: “IDK Claude did that.”

By no means am I down on AI, but I think proper procedures need to be put into place unless we want a giant bomb in our code base.

View on HN

I don't think "IDK Claude did that" is a valid excuse. Immediate rejection.

AI may be multi-threaded, but there's still a human, global interpreter lock in place. :D

If you put the code up for review, regardless of the source, you should fundamentally understand how it works.

This raises a broader point about AI and productivity: while AI promises parallelism, there's still the human in the middle who is responsible for the code.

The promise of "parallelism" is overstated.

100's of PRs should not be trusted. Or at least not without the c-suite understanding such risks. Maybe you're a small startup looking to get out the door as quickly as possible, so.. YOLO.

But it's going to be a hot mess. A "clean up in aisle nine" level mess.

It's insane that any company would just be OK with "IDK Claude did that" any more than a 2010 version of that company would be OK with "IDK I copy pasted from StackOverflow." Have engineering managers actually drank this Kool-aid to the point where they're actually OK with their direct reports just chucking PRs over the wall that they don't even understand?

Depends on your incentives. People anecdotally seem far more impressed with buggy stuff shipped fast than good stuff shipped slowly.

Lots of companies just accept bugs as something that happens.

Depends on your problem space too.

Calendar app for local social clubs? Ship it and fix it later.

B2B payments software that triggers funds transfers? JFC I hope you PIP people for that.

It is even more funnier when you realize that because Claude and all AI models are trained on data including stackoverflow.

So I guess if you asked Claude why it did that, the truth of it might be "IDK I copy pasted from StackOverflow"

The same stuff pasted with a different sticker. Looks good to me.

Ha, I genuinely laughed at that. Thank you!

Of course they are okay with it. They changed the job function to be just that with forced(!) AI adoption.

This is exactly how I see it. Is not about the tool, is how it is used. In 1990 that would have been “IDK I got it from a BBS” and in 1980 “got if from a magazine“. It doesn’t matter how you get there, you have to understand it. BTW I had a similar problem as I was manager in HW development, where the value of a resistor had no documented calculation. I would ask: where does it came from? If the answer was “I tried and it worked”, or “tested in lab until I found it” or in the 2000 “I run many simulations and was the best value” I would reject and ask for proper calculations, with WCA.

As vibe coding becomes more commonplace you'll see these historical safeguards erode. That is the danger IMO.

You're right, saying you got something off SO would get you laughed out of programming circles back in the day. We should be applying the same shame to people who vibe code, not encourage it, if we want human-parseable and maintainable software.

> That is the danger IMO.

For whom is this a danger for?

If we're paid to dig ditches and fill them, who are we to question our supreme leaders? They control the purse strings, so of course they know best.

> If we're paid to dig ditches and fill them

This is a very cruel punishment sometimes used in forced labor camps. You are describing torture.

The same developers submitting Claude submissions can take 1-2 minutes asking for an explanation of what they're submitting and how it works. Might even learn.

Stack Overflow had enough provenance of copying and pasting. Models may not. Provenance remains a thing or it can add risk to the code.

I don't think it's common, but I've definitely seen it

I've also seen "Ask ChatGPT if you're doing X right?", and basically signing off whatever it recommends without checking

At this point I'm pretty confident I could trojan horse whatever decision I want from certain people by sending enough screenshots of ChatGPT agreeing with me

I don't think this is an AI specific thing. I work in the field, and so I'm around some of the most enthusiastic adopters of LLMs, and from what I see, engineering cultures surrounding LLM usage typically match the org's previous general engineering culture.

So, for example, by and large the orgs I've seen chucking Claude PRs over the wall with little review were previously chucking 100% human written PRs over the wall with little review.

Similarly, the teams I see effectively using test suites to guide their code generation are the same teams that effectively use test suites to guide their general software engineering workflows.

How long are you spending with a given team, and where per se in their "AI lifecycle?" I would expect (for example) a sales engineer to see this differently than a support engineer, if support engineers still existed.

If it pushes some nice velocity metric, most managers would be ok. Though you have to word it a bit differently of course.

"Look, the build is green and CI belongs to another team, how perfectionist do you need us to be about this?" is the sort of response I would generally expect, and also in the case where AI was used.

What about "the compiler did that" ?

As someone who doesn't use AI for writing code, why can't you just ask Claude to write up an explanation of each change for code review? Then at least you can look at whether the explanation seems sane.

It will fairly confidently state changes are "correct" for whatever reason it makes up. This becomes more of an issue with things that might be edge cases or vague requirements, in which case it's better to have AI write tests instead of the code.

Claude also doesn't know, because Claude dreamt up changes that didn't work, then "fixed" them, "fixed" them again and in the process left swathes of code that isn't reached.

This can be dangerous, because Claude doesn't truly understand why it did something. Whatever it writes a post-hoc justification which may or may not be accurate to the "intent". This is because these are still autoregressive models --- they have only the context to go on, not prior intent.

Indeed. Watching it (well, Anthropic, really) cheat at Baba Is You and then try to give a rationalization for how it came up with the solution (qv. https://news.ycombinator.com/item?id=44473615) is quite instructive.

Because the explanations will often not be sane; when they are sane, they will focus on irrelevant details and be maddeningly padded out unless you put inordinate effort into trying to control the AI's writing style.

Ask pretty much any FOSS developer who has received AI-generated (both code and explanations) PRs on GitHub (and when you complain about these, the author will almost always use the same AI to generate responses) about their experiences. It's a huge time sink if you don't cut them off. There are plenty of projects out there now that have explicit policy documentation against such submissions and even boilerplate messages for rejecting them.

AI is not a human. If it understands things it doesn't understand things like you or I. This means it can misunderstand things in ways we can't understand.

AI is not sentient, so it does not “understand” anything. I don’t expect the autocomplete of my messenger app to understand its output, so why should I expect Claude to understand its output?

I've been experimenting with Claude, and feel like it works quite well if I micromanage it. I will ask it: "Ok, but why this way and not the simpler way? And it will go "You are absolutely right" and implement the changes exactly how I want them. At least I think it does. Repeatedly, I've looked at a PR I created (and review myself, as I'm not using it "on production"), and found some pretty useless stuff mixed into otherwise solid PRs. These things are so easily missed.

That said, the models, or to be more precise, the tools surrounding it and the craft of interacting with it, are still improving at a pace where I now believe we will get to a point where "hand-crafted" code is the exception in a matter of years.

> I don't think "IDK Claude did that" is a valid excuse.

It's not, and yet I have seen that offered as an excuse several times.

Did you push back?

At least I would not accept from my team. Is borderline infuriating. And I would promptly insinuate, if that is the answer, next time I do not need you, I will ask directly Claude, you can stay home!

Maybe check what's their workload otherwise. Most engineers I've worked with want to do a good job and ship something useful. It's possible they're offloading work to the LLM because they're under a lot of pressure. (And in this case you can't make them stay home)

> The promise of "parallelism" is overstated.

100% my takeaway after trying to parallelize using worktrees. While Claude has no problem managing more than one context instance, I sure as hell do. It’s exhausting, to the point of slowing me down.

That's an intended effect. It doesn't matter to those in power who know what AI is really for. Once you get so exhausted that you can't work any more, there will be a hundred bright-eyed naïve programmers who will step into your place and who think they can do better. Until they burn out in a few years time.

I have been wondering when I would start to feel aged out of the tech industry… gosh is it here already?

I don't know if it is but I'm certainly glad I left tech a long time ago...

> If you put the code up for review, regardless of the source, you should fundamentally understand how it works.

Inb4 the chorus of whining from AI hypists accusing you of being an coastal elitist intellectual jerk for daring to ask that they might want to LEARN something.

I am so over this anti-intellectual garbage. It's gotten to such a ridiculous place in our society and is literally going to get tons of people killed.

I understand and agree with your frustration, but this is not what discourse here is supposed to look like.

> I don't think "IDK Claude did that" is a valid excuse. Immediate rejection.

I strongly agree, however manager^x do not and want see report the massive "productivity" gains.

You tell them clippy’s revengeance pr caused an outage worth millions of dollars because of push for productivity and they shouldn’t bother you for a couple of months.

Sane CTOs think "Claude did that" is invalid. I assure you: those leaders exist. Refuse to work for idiots who think bots can be held accountable. You must must understand every line of code yourself.

"Claude did that" is functionally equivalent to "idk I copied that from r/programming" and is totally unacceptable for a professional

> You must must understand every line of code yourself.

I have never seen this standard reached for any real codebase of any size.

Even in projects with a reputation for a strong review culture, people know who the "easy" reviewers are and target them for the dicey stuff (they are often the most overloaded... which only causes them to get more overloaded). I've seen people explicitly state they are just "rubber stamping" PRs. Literally no one reviews every line of third-party dependencies, and especially not when they are updated routinely. I've seen over a million lines of security-sensitive third-party code integrated and pushed out to hundreds of millions of users by a handful of developers in a matter of months. I've seen developers write their new green-field code as a third-party library to circumvent the review process that would have been applied if it had been developed as a series of first-party PRs. None of that had anything to do with AI. It all predated AI coding tools. That is how humans behave.

Does this create ticking time-bombs? It absolutely does. You do the best you can. You triage and deal with the most important things according to your best judgment, and circle back to the rest as time and attention allow. If your judgment is good, it's mostly okay. Some day it might not be. But I do not think that you can argue that the optimal level of risk is zero, outside of a few specialized contexts like space shuttles and nuclear reactors.

I know. It hurts my soul, too. But reality isn't pretty, and worse is better.

I think the "you" in the quote is referring to the programmer of the PR, not the reviewer. I agree that it's probably unrealistic to expect reviewers to understand every line of code in a PR. That's why it's crucial that the programmers of said PRs themselves understand every line of code. I'll go one step further:

If you submit a PR and you yourself can not personally vouch for every line of code as a professional…then you are not a professional. You are a hack.

That is why these code generation tools are so dangerous. Sure, it's theoretically possible that a programmer can rely on them for offering suggestions of new code and then "write" that code for a PR such that full human understanding is maintained and true craft is preserved. The reality is, that's not what's happening. At all. And it's a full-blown crisis.

This.

I know many who have it on from high that they must use AI. One place even has bonuses tied not to productivity, but how much they use AI.

Meanwhile managers ask if AI is writing so much code why aren't they seeing it on topline productivity numbers.

How does maximizing AI use prevents developers from reading their code? Especially if bonuses are not tied to productivity as you say. Just treat AI as a higher level IDE/editor.

There's more code to read as unskilled or sleepy developers push tons of sloppy changes. The code works, mostly, So either one loses more time chasing subtle issues or one yolos the approvals to have time for one's own coding workload.

I don't understand how your comment relates to what I've been responding to.

>> I know many who have it on from high that they must use AI. One place even has bonuses tied not to productivity, but how much they use AI.

> How does maximizing AI use prevents developers from reading their code?

In my mind developers are responsible for the code they push, no matter whether it was copy pasted or generated by AI. The comment I responded to specifically said "bonuses tied not to productivity, but how much they use AI". I don't see that using AI for everything automatically implies having no standards or not holding responsibility for code you push.

If managers force developers to purposefully lower standards just to increase PRs per unit of time, that's another story. And in my opinion that's a problem of engeneering & organisational culture, not necessarily a problem with maximizing AI usage. If an org is OK with pushing AI slop no one understands, it will be OK with pushing handwritten slop as well.

> If managers force developers to purposefully lower standards just to increase PRs per unit of time

That's basically what I'm referring to.

> I don't think "IDK Claude did that" is a valid excuse. Immediate rejection.

That will work, but only until the people filing these PRs go crying to their managers that you refuse to merge any of their code, at which point you'll be given a stern reprimand from your betters to stop being so picky. Have fun vibe-reviewing.

Yeah in a perfect world I absolutely agree. But the reality I'm observing is that everything continues to always be behind schedule (not a new phenomena) and if anything expectations from project leads / management are just getting less realistic leaving Jr devs in even more of a position of "no time to be curious or learn deeply just get it done" and teamleads reviewing PRs in a maybe even worse position of "no time to get into a deep review / mentorship session just figure out if it breaks anything or not". And ultimately clients are still just living in fantasy land in terms of expectations both in terms of build out time for basic features / patches and also how much AI razzle-dazzle they expect for new project proposals.

Nothing can move fast enough to keep up with these hype-fueled TED talk expectations all the way up the chain.

I don't know if there's any solution and I'm sure it's not like this everywhere but I'm also sure I'm not alone. At this point I'm just trying to keep my feet wet on "AI" related projects until the hype dust settles so I can reassess what this industry even is anymore. Maybe it's not too late to get a single subject credential and go teach math or finger painting or something.

> That is, do people feel like they’re getting better, more reviewable PRs?

No, like you I’m getting more PRs that are less reviewable.

It multiplies what you’re capable of. So you’ll get a LOT of low quality code from devs who aren’t much into quality.

I've seen the worst impact on mid/junior level devs. Where they would have to struggle through a problem before - AI is like a magic shortcut to something that looks like it works. And then they submit that crap and I can't trust them again - they will give me AI code without even understanding what it does fully. It robbed them of the learning process and made them even less useful, while making it seem to them they were achieving something. I'm seeing these kinds of people getting removed from workforce fast - you can probably prompt the AI better on your own, have one less layer of indirection and it will be faster.

Exactly this. In the right hands the outputs from LLMs are amazing. Not always right, but can be checked. In the wrong hands they are worse than useless because they cannot be checked.

A new coworker solved a problem that we'd basically been unable to fix for 2 years, and we were relying on workarounds. This was especially impressive because it was totally outside his expertise. When I asked how he did it, a big part of his process was using LLMs.

For the most part, I've noticed my LLM-using coworkers producing better PRs, but I haven't noticed anyone producing more.

That's the thing, there's a big difference between using AI to help you solve a problem and using it to write code that you don't understand (and not take the time to understand it). It would suck to be reviewing someone else's PR and find out you're putting more effort into understanding their code than they did. But these are powerful tools and they're not going away anytime soon

> It would suck to be reviewing someone else's PR and find out you're putting more effort into understanding their code than they did.

It does, but that experience was a real thing long before AI-generated code.

I've been at the same company both before and after the AI revolution. I've felt something similar. People seem to be more detached, more aloof in their work. I feel like we're discussing our code less and are less able to have coherent big-picture plans concerning the code at-large.

The ultimate purpose of AI is indeed to remove cognitive autonomy. Little pieces here and there may seem empowering, but added all up, it pretty much takes control away from people.

Bingo. This is a drive to go back to the good old days of leasing mainframes, before the peasants got their grubby hands on the means of computation.

> I’ll ask them something about the design and they’ll reply: “IDK Claude did that.”

"Then your job is to go ask Claude and get back to me. On that note, if that's what I'm paying you for now, I might be paying you too much..."

I'm really interested to see how the intersection of AI and accountability develops over the next few years. It seems like everyone's primary job will effectively be taking accountability for the AI they're driving, and the pay will be a function of how much and what kind of accountability you're taking and the overall stakes.

I hope so, in reality, people who review code are not always the superiors of the other party :)

Not meaning to put on a holier-than-thou hat here. But I hated this many times already. E.g. having to approve code that contains useless comments that were clearly generated by AI, but on the other hand, it's nitpicking to bring it up during review...

If my colleagues would do that more often and for more parts of the code, that's where I really start getting inclined to look for a new job.

This just sounds like a low quality bar or PRs that are too wide in large in scope for AI. In any case, it sounds like taking a single stab at it with an LLM and calling it good. I’m not really AI for everything type of dev, but there are some tasks that AI excels at. If you’re doing it for feature work on a high churn product without a tight grip on the reins, I fear for your products future.

I too have seen anecdotes that people "feel" more productive. But science says that they're 19% less productive. I'm going to go with science on this one.

Further, the problem with the word "productive" is that it's often measured in quantity and speed, not quality.

The study that said devs were 19% less productive was flawed in a number of ways: small sample size, poorly chosen tasks, no prior training on AI tools.

The science is by no means conclusive. A joint Harvard / MIT study found developers completed tasks up to 50% faster [0]!

[0] https://arxiv.org/abs/2302.06590

>I’ll ask them something about the design and they’ll reply: “IDK Claude did that.”

I would want someone entirely off of my team if they did that. Anyone who pushes code they don't understand at least well enough to answer "What does that do?" and "Why did you do it that way?" deserves for their PR to be flat out rejected in whole, not just altered.

> Personally, I’ve been seeing the number of changes for a PR starting to reach into the mid-hundreds now. And fundamentally the developers who make them don’t understand how they work.

Could this be fixed by adjusting how tickets are scoped?

AI is like a giant calculator you need a formula to make it work for your usecase

That's a great analogy. I recently read about integrating AI similar to the use of calculators in math class -- learn how to do the basic operations first +,-,/,* and then use calculator to scale so you get some theoretical grounding

Accurate. It's autocomplete on steroids.

Calculators are mostly deterministic and AI is explicitly not.

When computers give different answers to the same questions it's a fundamental shift in how we work with them.

A lot depends heavily on how you define productive.

At one of my jobs, the PRs are far less reviewable, but now the devs write tests when they didn’t used to bother. I’ve always viewed reviewing PRs as work without recognition, so I never did much of it anyway, but but now that there are passing tests, I often approve without more than a cursory skim.

So yes, it has made their work more productive for me to get off my plane.

But are the test actually useful? You can have a test suite that is actually harmful if it’s not ensuring business rules or domain correctness. Anything else makes for a brittle dev ecosystem.

Thanks for this take. It perfectly illustrates the productivity illusion.

Mostly blindly approving PRs mainly because they now contain some form of equally AI-generated tests is exactly the producticity fallacy that's the issue here.

It's extremely hard to measure productivity correctly and self-reports are worthless. I don't think AI tools are a net negative in the average case (people are definitely indexing too much on that goddamn METR article) but "i'm 10x more productive, source trust me bro" is equally nonsense.

Using AI tooling means, at least in part, betting on the future.

> Using AI tooling means, at least in part, betting on the future.

It means betting on a particular LLM centric vision of the future.

I’m still agnostic on that. I think LLMs allow for the creation of a lot of one off scripts and things for people that wouldn’t otherwise be coding, but I have yet to be convinced that more AI usage in a sufficiently senior software development team is more valuable than the traditional way of doing things.

I think there’s a fundamental necessity for a human to articulate what a given piece of software should do with a high level of specificity that can’t ever be avoided. The best you can do is piggy back off of higher level language and abstractions that guess what the specifics should be, but I don’t think it’s realistic to think all combinations of all business logic and ui can be boiled down to common patterns that an LLM could infer. And even if that were true, people get bored/like novelty enough that they’ll always want new human created stuff to shove into the training set.

An LLM is not a tool to allow you to add a layer of abstraction. It's a worker.

It works by translating language abstractions to code.

The probability of plain english being correctly translated to code depends on existing code and documented abstractions describing lower level functionality.

Not in any consistent or deterministic way, which is why it can't be used as a layer of abstraction.

Humans, for example, aren't a layer of abstraction. Having a team of engineers doesn't enable you to say "build X" and just forget about it.

> "i'm 10x more productive, source trust me bro"

Yet HN has what feels like one blog post or article to that effect every day. Some senior person describing their workflow, arguing that this makes them 10x of before.

It's refreshing to get the other perspective now and then.

Reply on news.ycombinator.com