Hacker News Reader

Comment by rsyring | Hacker News Reader

> 26k code changes in 2 months..... rsync was 67k LOC as of 236417c (latest not obviously vibecoded commit it seems?).[1]

Wow.

1: https://github.com/RsyncProject/rsync/issues/929#issuecommen...

View on HN

When I look at the commits themselves, most of the ones generated by Claude are testsuite changes, or at least labelled as such.

https://github.com/RsyncProject/rsync/commits/master/

Is that suppose to make this better? IME the most valuable tests are those that test specific regressions. It's the scaffolding we build for ourselves to enable feature development. Remove that scaffolding and you get accidents. Pray to your god of choice these accidents don't cause harm or loss of life.

It should really be considered negligence at this point. Some of this software is extremely valuable, it's how we flourish as humans. Purposely fucking with that should bear some real world consequence. We do the same in every other industry, software is just as important too.

In my perspective, "Analyze code, come up with edge cases and gaps and create unit tests for them" is one of the use-cases where AI was starting to get really good at, so I can see why someone would want to extend their test-suite dramatically using it.

But yes, using AI to then generate code that still causes regressions doesn't quite square with that. Given the huge amount of test-changes I'd still assume good faith by the maintainer; possibly just a bit of overexcitement paired with a dash of too much confidence into the new tools that is now hitting reality.

I hear you, OTOH if this software was so valuable how come we aren’t funding it? A lot of the world runs on OSS with a coupe overwhelmed maintainers who get treated as if they owed everybody working software yet can’t make a living off it.

We should fund it. Go read the types of comments I make in my profile. I always advocate for explicitly taxing big tech to publicly fund open source development.

Also it's why we need to pass things like medicare for all and universal childcare to give workers some breathing room if they want to change jobs/industries without condemning them to death or poverty.

Taken at face value, most commit descriptions mention adding - not skipping - tests and assertions.

So basically, we're all in our high horses, not reviewing code, scalding the unpaid maintainer for … not reviewing code.

Time for - whoever actually cares - to do better.

> Is that suppose to make this better?

When I first saw the 26k changes statistic I was shocked. It made me think a large chunk of code running on people’s machines was AI-generated.

But the knowledge that a lot of the changes might be testsuite changes made me change my perspective. If for instance 25k of the changes were test changes and only 1k of the changes actually affected the .so and other artifacts used downstream, that would be a lot less dramatic.

I haven’t reviewed the code, only the messages, so I don’t know if these changes were removing or adding test cases. And there are a minority of Claude-assisted changes which are not listed as tests.

Aren’t LLMs notorious for just making tests pass and not actually testing functionality?

You have to keep an eye on them, but they don't just make tests pass.

Claude sonnet 4 (this time last year) did do this. It once made simulation if a test script passing. Literally a script that just echoed test names and then said pass.

Change happens fast, a year old model is pretty outdated.

I'm sure it can happen, hence why I said to keep an eye out. Its main mode of operation is not to cook the tests however.

Happened to me, 3 days ago - deleted some tests and flipped assertions after outlining that it wasn't to change any assertions.

Our team was doing a similar task to move between test frameworks, and I had to do a git diff of hundreds of thousands of lines to try and work out where a test had disappeared to.

> 3 days ago

Your fault. You should have used a model from 0.000005 seconds ago!

Reading is difficult.

> Change happens fast, a year old model is pretty outdated.

What change? That you should not fake the results of a test because that defeats the whole purpose of a test has been known before there were computers.

I don't know, the weather?

I’ve never seen Claude do that. It makes the new tests pass by fixing previously unknown bugs in my experience.

I love Claude but on several occasions I've had it do some really funky stuff just to get tests passing

I had it do it about a month ago. It changed test data which caused another test to fail and instead of isolating things it decided to flip an assert.

That's because Opus needed vacation and they routed your requests to its less sophisticated cousin, Claude Dynamite. ;)

Yeah, in 2024.

The correctness of tests is as important as the correctness of the main code. Changing test code isn’t somehow less critical than changing the main code.

Well to look at the last of that list. It added 134 - 3 lines to the project.

Of which, the actual change was

    -        __m256i mul_one;
    -            mul_one = _mm256_abs_epi8(_mm256_cmpeq_epi16(mul_one,mul_one)); // set all vector elements to 1
    +        __m256i mul_one = _mm256_set1_epi8(1);

and the rest was testing that fix.

Reply on news.ycombinator.com