Hacker News Reader

Comment by hliyan | Hacker News Reader

A good thing about the "you're the author regardless of the tool you used to produce the code" policy is that if a developer repeatedly submits poor quality AI generated code, one does not blame the model or the agent, one bans/terminates the developer.

View on HN

The issue that open source projects are facing at the moment is that it takes significantly less effort to submit a patch for review.

A lot of developers who are submitting these AI patches don't necessarily understand the patch, so the onus is on the reviewer/code-owner.

The reviewers are getting swamped (some reviewers are receiving 100s or patches per month). If feedback is provided at lot of the time the patch author will just copy paste from an LLM, so the reviewer is essentially just coding with an LLM with more steps.

Prior to LLMs reviewing code was a mentorship experience, the patch author would likely learn a bunch afterwards. Now less so.

As a result a lot of projects are closing to external contributors.

I'm not sure what the answer is, LLM are great at speeding up coding/understanding/etc, but the valuable/expensive piece of work has shifted to reviewing.

The question becomes, does it take more time to create or review in the Grand scheme of the software life cycle?

Because if it's still a time saver, even with the increased review load, then it's a win, correct?

I'm not a coder so I have zero idea. Thoughts?

It's a well-known trope, that it's harder to read code than to write it, and review is more complicated than to read it, so I do not understand what you want to imply?

Maybe it is for people who have written code all their lives, but as someone who started with reading code and has read far more code than I’ve ever written, reading code is WAY easier for me.

If you have little experience writing code, do you know that you actually understand what you’re reading though?

For example could you reproduce the code you read immediately after writing it?

It’s very easy to “read code” if you’re just reading for surface level understanding.

If it's easier for you to convert written code into mental models than the other way around, you are either a incredible smart and skilled person and a good programmer and should be able to convert that into a job and money easily, or you can't really write code at all.

Nope, most of the PRs are authored entirely by agents with people instructing them to “pr famous projects in my name to increase my GitHub profile value or “hire ability”. The original authors have no idea what their agents are writing, these are mostly badly quality models (people doing this are not only cutting corners, but also using the less amount of money/tokens to do so). If the agent creates 200, 400 or whatever PRs and get 5 in the agent is still “winning” for the person instructing it. The maintainers though have to filter these out on the other side. A common case is a fix to something that was already fixed simply because the agent worked on old code assumptions.

Just had a thought, what if you made new contributors write a human-authored essay on why they’re submitting a patch, and then check it against Pangram? Pangram is really accurate from what I’ve found…

> Just had a thought, what if you made new contributors write a human-authored essay on why they’re submitting a patch, and then check it against Pangram?

The problem is writing such an essay takes magnitudes more of an effort for people who are not native English (or whatever the language is in which discussions are done about the project) speakers.

Also, relatedly, essays written by non-native English speakers often get considered to be AI-written by common AI detection systems, even though no AI was involved when writing them; see for example

> I'm Kenyan. I Don't Write Like ChatGPT. ChatGPT Writes Like Me.

> https://marcusolang.substack.com/p/im-kenyan-i-dont-write-li...

I’m as skeptical of the AI writing detection as anyone else, but I’ve been trying to beat Pangram v4 for the last week and it’s essentially impossible.

Have you tried in both directions though? You need to test both sensitivity and specificity.

Yes. It’s scarily accurate.

They have a blog post on it if you look up “Pangram v4 System Card“

> Nope, most of the PRs are authored entirely by agents with people instructing them to “pr famous projects in my name to increase my GitHub profile value or “hire ability”.

I've heard this in a lot of places, don't have a way to verify this in action. Do you have any citation or just gut feel (still valid but less useful for me).

Anecdotally, as someone who's been starting to get more and more agent-only/driven PRs, a good amount of the time it's by an account with little to no activity until one day they suddenly start doing 50-100+ PRs a _day_ on entirely unrelated projects... you can kind of guess what they're doing.

As for motive, it probably varies, though some will loudly put that they're some agent manager or something with lots of contributions, so you can guess their motivations there.

In other cases, these are instrad brand-new accounts which is a related but I guess slightly different story... not gonna say these are spam bot accounts being juiced with fake activity but well...

These are all fair observations, I have not personally come across these yet but I am also not looking for them. I guess someone more involved in open source collaboration would have a better lens here.

There was this attempt (attack?) on Fedora in June: https://lwn.net/Articles/1077035/

It doesn't help with the additional work/chores, at least initially, but someone on HN had a really good idea IMO, just take the code, and close the PR.

https://news.ycombinator.com/item?id=49474708

I saw the argument that if OS maintainers "really care" about their projects, they would accept valid contributions regardless origin. Well, turn it around: if you really care so much about contributing, if you really just wanna HeLP, you don't need the credit. The warm fuzzy feeling that your contribution ended up being used should be enough.

Probably a time waster because of the intermediate patch contributor. It’s like a game of telephone at worst, at best the reviewer could just use their own LLM and get the same result.

Here’s my hot take: maybe code contributions are obsoleted by coding agents. No one will accept outside contributions because it’s faster to do it themselves.

It depends entirely on how much slop they are flooded with.

Well historically the consensus was that it was harder to read code than to write it, but that leads to uncomfortable conclusions these days, so that bit of common wisdom has mostly been ejected.

> I'm not a coder so I have zero idea. Thoughts?

I’m not trying to be mean, but this comment is basically “I have no experience with this topic, but it can just be boiled down to this one simple question right?”

I don't think that's being mean at all, and it's valid.

Yes that's literally why I asked it. Simple questions, here, regularly elicit long form explanations. I was interested in this topic and have no experience. So I thought I'd try a broad overview type question to see if I could learn something today.

I think I read it as a bit more flippant than you meant it because it was worded assuming the answer was yes.

> The question becomes [...]

"A question is...". To my mind not the most important question, if one plays-out to a logical conclusion the scenario you're proposing.

> [...] if it's still a time saver, even with the increased review load, then it's a win, correct?

No - and that's bordering-on (if not fully) rude disrespect of reviewers time and effort. One way to think of this is in terms of Brandolinis Law[0]. Pushing work back to submitters is going to have to happen. Low-effort "submissions" are first and foremost "low-effort" - that's going to have to be driven home.

[0] https://en.wikipedia.org/wiki/Brandolini's_law

Even before automation, most first time PRs were a negative time contribution to the project. The only value of them was that some contributors would become trusted project members.

Do all of these folks get the comped Pro Max subscriptions? If not O&A should be.

Or, at the very least, the community should be paying for them.

This is Amdahl's law in action. [1]

Until we figure out a good way to leverage humans in all of this ("Attention is all you need" applies equally to humans as it does to models) productivity gains for the system will always be limited by Amdahl's law. Gwern has an excellent post on this. [2]

[1] https://en.wikipedia.org/wiki/Amdahl's_law [2] https://gwern.net/guardian-angel

I'm an OSS maintainer and to me it's not just about the review itself. Being greeted by a wall of text for every little small thing is counter-productive. I hate going through 2 pages of text for each PR. It usually shouldn't take more than a couple of sentences if you understand the issue and the solution.

But most important for me: lots of time the PR just adds even more code, although other options do exist (ie sometimes REMOVING some code). You have to know the codebase well in order to find those objectively better solutions.

I've seen it. Walls of text with stereotypically worded non-summaries that just repeat all of the code in words, mutating values all over the place instead of the obvious canonical one place that touches related values...

Yeah you can use LLMs, but don't let me notice it from the quality of the output.

I've noticed that LLMs seem to be especially bad at things relating to space, position and movement. I guess they have to synthesize that part of human intelligence entirely, it's not in the words.

Give https://pyor.review a shot if you’re struggling with PR reviews on github.

Or, the SaaS-less approach, if a issue description is too messy/long, close it with "Please reopen with proper and concise description focusing on the issue" then lock it. Eventually people catch up and stop with the slop, just like in real life.

But you have to be able to say "No ...", rather than just slapping another subscription on top of an already broken workflow.

If you're maintaining OSS, that's understandable, and you're free to say no, but in the corporate world, that's not realistic, AI is here to stay, if they don't harness it they would just be left behind. even if the AI gets good and stops writing sloppy stuff, it's still gonna write a lot of stuff, and you're gonna review it anyway, and take responsibility and ownership, and it's still gonna take you more time, because the bottleneck is now reviewing and understanding the code.

I agree that the workflow is broken, but only on the reviewing side, AI is a tool we use to make products just like any other we used in the past, punch cards, machine code, assembly, ...etc. AI is just the new tool that sits on top of the code as the next level, no one codes with punch cards, no one writes machine code anymore, we used to write the compiled language and don't care about how it's compiled or turned into machine code, same with AI, although it's not there yet and still requires babysitting by engineers, but that's our new job now, and we need to learn how to use it and make our lifes easier.

> but in the corporate world

Sure, but that's irrelevant when someone says "I'm an OSS maintainer" and the context is explicitly about reviewing code submitted by others in a FOSS context, where you can say "No" and don't need any SaaS in order to do so.

That's not true at all of the corporate world. If your team is mass producing slop and you don't have processes in place to get it under control, you've got a big problem on your hand.

If any engineer sent me a 20,000 line refactor I'd immediately reject it and tell them to go back and start making changes incrementally at minimum. More likely I'd force them to have a whole design discussion with the team to make sure that what they are doing even makes sense.

What happens if they push out slop that significantly increases your infrastructure costs? What happens if they push out slop that significantly increases the number of bugs or outages? What happens if they push out slop that has no observational metrics, dashboards, or tooling?

In every case you push back on the team and make them fix their shit. I don't care if they are using LLMs or not. They are responsible for their work being sufficient quality. If they aren't meeting those standards, then they need to step it up.

I don't disagree with you on this, I worked my whole life in corporate and haven't worked as a OSS maintainer before, though I will _and already did_ reject PRs way less than that, I speak for myself and my team here and it's unrealistic to ship a single PR as complex as this, we usually plan features as tech designs with PRs of no more than 500 LOC, but that doesn't mean we could never have a 20k PR at all. in my experience, those huge PRs are usually the simple ones where most of it is just noise. I did one recently and moved the UI library in one of our old codebases three major versions up to the very latest, although it was 20k lines of changes, all of it is just mechanical chanes, renames, codemod stuff, test fixes, snapshot updates, ...etc. and it's not realistc to split this into multiple smaller PRs as this can bring other complications like having multiple versions of the same UI library in the codebase, which could cause more problems than it fixes, AI helps with that kinda work a lot and I would've never been able to do this migration is such a short time without it. despite that, reviewing it was a UI challenge, not code, with UI libraries, the breaking changes are usually in the UI so you can't see it from the code, and we did have a special process to review it, although on the code side, Github was a nightmare to deal with reviewing this PR, we noticed that github was the bottleneck here since it lays out the code changes without much context and is already hard to navigate and stuggles with huge PRs, the review surface and the developer experince on github was horrible, and that's why I suggested you look for better alternatives, there are a lot out there and all of them are free for OSS so why not try them?

How do you handle them? I'm not facing this problem as the team I work with is very senior and have good taste and discipline. But I can imagine it will be a problem at some point, and I frequently have to personally tell Claude to rewrite it's vomit in English. That's probably step one for people submitting poorly written PRs, reject them until they are written clearly and concisely. And if they are too big, also rejecting them and telling them to go back to the drawing board and submit smaller more focused change. But I'm not int his position so I'm taking an educated guess.

To me, that touches on the central issue. I think most people support the “I don’t care about the tool, only the outcome “

But, we have long had a stable social system for collaboration, and in particular dealing with low quality contributions. Mostly (or at least ideally) it involves taking some time to look at the contribution and provide some feedback. The time invested is not worth it on a single exchange basis, but it’s a repeat game, so you are investing in the future.

LLMs, and people willing to just spam crap they don’t really understand blows that all up.

Nah, that was already blown up way before LLMs existed.

Foss wasn't exactly not miserable before 2022. The dysfunction just got amplified so that we cannot fix it anymore by just throwing more human resources at the problem.

I tried to nod a bit to the inconsistency. Some projects clearly lost patience with imperfect contributions long ago. But many were still pretty supportive.

This dynamic also plays out in many companies. I see a lot of posts from mid to sr engineers frustrated with plummeting code quality and unsure how to manage the reviews. They are increasingly pulling away from the idea that they should mentor people to do better.

The correct response to irresponsible AI use in the workplace is maintaining standards and rejecting poor quality. Performance improvement plans as a fair warning followed by firing people works for incompetence or laziness, AI doesn’t change anything.

I’ll admit I’ve seen a spike in negative performance reviews but this is the new normal people will adapt.

It’s almost like late-stage capitalism isn’t built to support mentorship and education and enabling human improvement!

> They are increasingly pulling away from the idea that they should mentor people to do better.

Two considerations on this:

1. In the USA, it is common to switch jobs often (in opposite to, for example, Germany). So, in the US-American system it is not often worth the effort to invest a lot of mentoring ressources into some employee if he will leave soon anyway.

2. I personally observe that I often do try to give less experienced programmers good hints on how to improve, but my kind of thinking about programming is often so different that my advice is often of no use for them. Or I often get the arrogant reaction by less experienced programmers that I care far too much about obscure details. I thus seriously do believe that the only way to help junior programmers to become better is to completely crush their ego and arrogance (just like a doctoral study programme in, say, mathematics does to the doctoral students), which is of course not allowed at work. :-(

I'm not convinced it blows up. It might also end up placing a larger burden on contributors, and especially first-time contributors, to provide concise, high-quality documentation that justifies their contribution.

I'm thinking of something like: Explain, in at most 300 characters, why we should merge your change, or at least why we should invest the time to read a longer explanation (somethings things ARE complex). If you don't do that, or those 300 characters aren't convincing, or have nothing to do with your change or the longer explanation, then you get a canned response so the maintainers waste minimal time.

you know EXACTLY what would happen in such a system.

I don't, please enlighten me.

My prediction is that many new contributors would complain loudly. Some existing contributors would sympathize.

It’s I think a normal evolution of a social norm. There will be a period of competing values.

Explain, in at most 300 characters, why we should merge your change, or at least why we should invest the time to read a longer explanation (somethings things ARE complex) would simply become a prompt.

The solution, of course, is to use the best available LLMs as the first line of defense against slop contributions. But if the project maintainers have issued a sweeping fatwa prohibiting an entire class of useful tools, that option along with many others is off the table.

I wonder if some of these projects with ayatollah maintainers are looking at forks sooner rather than later?

people would use LLMS for those summaries, defeating the purpose. You can already see it in PRs in companies where LLM code is pushed.

That doesn't really solve the problem. I've gotten PRs that do do an adequate job of explaining why why something should be implemented. Then I look at the code and it doesn't fully solve the problem, or it does so in a way that will obviously break something else.

In that case, isn't a sufficiently reasoned "why" a contribution in itself?

Not trying to downplay the time wasted to reject the PR, though.

A colleague once setup a PR description template that asked for short sections like:

- Why is this change needed (big picture)

- Why are you doing it this way (small picture)

- What are the trade-offs you made for this approach

- How do you know this is correct, that it works the way you think it does?

- What other changes or approaches does committing to this way block in the future?

etc. If you had actually thought through what you were doing, it was a few min to fill it all out. People were just on auto-pilot really struggled.

> If you had actually thought through what you were doing, it was a few min to fill it all out.

... for people who are native speakers of English.

Software engineering is a highly collaborative process. To be successful you need to be pretty good at reading and writing whatever language the team is using. I've worked with plenty of non-native speakers who were well above the average I see for natives. In the last 10 years I found the level of just basic writing for new Jr engineers has really fallen.

If you can't write, you can't think

yeah, that's fair. Problem so far has been when poor developers send massive volume of slop for review. Takes time and effort to review or at least glance and test it, especially if developer hasn't bothered.

that seems like two separate problems.

too many patches from unknown and untrusted contributors is a triage problem - telling a known and trusted contributor they can't use AI to help work on something just because other people are sending slop for review is throwing the baby out with the bathwater.

More like throwing the bathwater out with the baby.

Terminates???

We need sufficiently severe penalties for vibe coding. First time out we break your legs, but if you come back with another vibed patch its around the back of the barn for you.

Nuke it from orbit.

What stops them from submitting PRs from another account?

I can't speak for all, but many mature open source communities have explicit or ad-hoc reputation schemes that are used to triage contributions. A fresh account means starting over with zero reputation, which in turn means your contributions wait at the back of the line.

ability to create low effort PR means that they can reset the reputation as much as they can..

There are no anonymous debian developers. All have been vetted by multiple people.

What stopped them before?

Before they were not able to spam large PRs

Why not? I'm pretty sure a developer could spam large PRs before and regardless of AI involvement is allowed or not.

How would you generate enough large PRs to count as "spam"? You could add gibberish or delete random lines of code but it was obviously not plausible looking.

this article is about the flood of junior developers who are producing enormous prs they’ve never seen full of untrustworthy code that a human hasn’t evaluated

arguing that it’s possible to make them by hand isn’t just missing the point; it’s missing the entire discussion

> arguing that it’s possible to make them by hand isn’t just missing the point; it’s missing the entire discussion

>> Before they were not able to spam large PRs

I was replying to someone insinuating that there was something I might have missed.

Interjecting about the topical article, is a derail of this thread for some reason that is personal to you. Good luck with whatever.

Obviously they could have. Obviously it's much easier now.

You can cut down a tree regardless if you use a knife or a chainsaw.

You can get to work each day regardless if you walk or drive.

Reputation, or lack thereof.

Proof of authority, essentially.

Nothing. And that's the problem -- the issue isn't "should a contributor be able to use AI" it's quickly becoming "is this entire contributor an AI agent". Every actual-human who gets a bad first experience hurts the project. The contributors having to be suspicious of new people hurts the project. The whole thing just sucks, and I don't know how to fix it.

I use claude-code, but the AI ecosystem and some of the autonomous tooling coming out of it -- which some people "helpfully" use irresponsibly and point at OSS projects they aren't personally invested in -- enables a LARGE amount of abusive behavior towards open source projects. This is even before we get into how many infrastructural resources crawlers eat up.

Me, and most other OSS contributors, signed up to be a mentor, to create software, to work together ... not to be the equivalent of a reddit moderator banning AI agents left and right.

What is a good project for a human to get human mentorship? Asking for a friend :)

https://stagex.tools

Always happy to train up new packagers and maintainers.

This is the second time I've seen this project mentioned on HN. Their white paper is very interesting and aligns closely with my interests but I don't see much on the main site about how to get involved.

Are they using their Matrix server to filter out drive-by/low-effort contributions?

I am the founder of the project. We accept every contribution that solves a problem, even drive-bys. Though they may often sit for a bit as we often merge in batches. If there is something missing, make a PR by all means.

Ask any questions on matrix.

Usually, mentorship is a quite well paid service.

Debian itself is actually great: https://mentors.debian.net/sponsors/

I work on OpenStack, which is a project folks work on primarily due to sponsorship -- not many people who want to do security/stability work for a cloud "for the fun of it" :D.

So mentees I've worked with have usually come as part of a program, such as MLH[1] or Outreachy[2]. Other projects I've helped with mentoring have come from college programs or professors[3].

I have no idea how hard it is to get through the filters to get these opportunities; I know generally we ask them to only send us the best and that's worked out really well.

[1] https://www.youtube.com/watch?v=fbsI8l-MaFA is an interview done on my podcast (by the other host, Tabatha) with CID, who works with me on OpenStack Ironic. A little over 2 years ago, he was an MLH fellow (https://fellowship.mlh.com/) working with me -- now he's a maintainer on OpenStack and working on my team. This is an exceptional outcome, but he's not the only person who has found success via MLH that I know.

[2] e.g. https://lists.openstack.org/archives/list/openstack-discuss@...

[3] https://lists.openstack.org/archives/list/openstack-discuss@...

"Nothing" is not accurate in the context of Debian as per comments above; it's not like Github in that sense.

In anonymous systems, where we don't use real names, your "handle" has an "age". That becomes your most prized possession.

I have a reddit account thats over 12 years old. I have left the posts/comments open. That means, my "post history" becomes a way to advertise my skills. I can reference comments from a year ago or 6 months ago about a certain topic to show how I handled something. That does have an effect on people.

As opposed to a random 1 day old account. No one takes them seriously, I guess the same is here as well

No one takes a 12 year old Reddit account seriously either. It's not a platform you can take anyone seriously. The biggest trolls have 12 year old accounts.

the age is not the measure of quality. the GP said they left the posts/comments open. anybody can read that and check if they are or not a serious contributor

Be extremely careful with that account then, because they ban accounts for the most minor things now. Stuff like posting evidence disproving something the moderator agrees with. And they ban your account from the whole site, not just from that subreddit any more. Automated algorithms also ban accounts for reasons that don't make any sense.

So if you're using your account to advertise yourself and look for jobs, I'd use it only for that. All off-topic discussion should be on a different account with a different physical device and IP address (reddit checks these things and cross-bans) that have never interacted with your main account.

What part of [0] or [1] makes you think that it is easy to have multiple accounts?

[0] https://wiki.debian.org/DebianMaintainer#Becoming_a_Debian_M...

[1] https://wiki.debian.org/DebianDeveloper/JoinTheProject/NewMe...

Well. This is Debian, known for how strict they are about vetting developers. I just remembered Sladen refusing to key sign because my passport circa 2005 didn't have a machine readable slug in it.

https://www.debian.org/devel/join/newmaint

Debian is notoriously very strict on accepting maintainers and they are not anonymous.

Presumably they are trying to farm karma / reputation, which (I think, mostly) doesn't work if they need to switch accounts all the time.

Reply on news.ycombinator.com