Hacker News Reader

Comment by anymouse123456 | Hacker News Reader

The actual efficiency claim (which is also likely incorrect) is inverted from the original article, "We examined the conversion rates of approximately 2,300 individual test cases spread out within 338 files. Among these, approximately 500 test cases were successfully converted, executed, and passed. This highlights how effective AI can be, leading to a significant saving of 22% of developer time."

Reading that leads me to believe that 22% of the conversions succeeded and someone at Slack is making up numbers about developer time.

View on HN

> 500 test cases were successfully converted, executed, and passed.

Wonder what "successfully converted" means? A converted test executing and passing doesn't tell you whether it's still testing the same thing as before.

That suggests a test suite to test the test suite, which again would suggest another test suite to test the test suite testing test suite...

In the end, it is test suites all the way down.

You could evaluate the quality of your test suite with something other than a test suite.

such as?

A developer who knows the code and will own the consequences can review and merge it - easy. Just not sure why the LLM needed to get involved in the first place.

Presumably because they've got 500 test cases?

Well, you need someone to write the code, and someone (else) to review the code.

In this situation you would replace the author with the LLM, but leave the reviewer as human.

It's not as pointless as you make it out to be. You still save one human.

You can run mutation tests that intentionally seed errors into the codebase (flipping Boolean and to or, for example) and rerun the tests.

A good test suite will catch these errors. Something that is effectively a noop to get to green will not.

There actually is a way to test your test suite: mutation tests [1]. Basically, you change the covered codebase (invert an if statement, change a variable, etc.) and expect the tests to then fail. If the tests actually survive the mutations, they might not be good enough.

[1] https://en.wikipedia.org/wiki/Mutation_testing

Also given that there are huge differences in complexity between tests, how do we know that the successful 22% are not just trivial one-liner tests?

Thinking about the test suite in my current project there is a clear Pareto distribution with majority of tests being simple or almost trivial.

Yes, the 80% claim comes from comparing 9 tests converted by both the conversion tool and humans and comparing the quality - “80% of the content within these files was accurately converted, while the remaining 20% required manual intervention.” Not sure what to make of it since they claim only 16% of files get fully converted.

Was it the 20% of the code that requires 80% of the time to write?

I guess they believe the files that didn't get fully converted got like 76% converted on average?

Someone will still have to review and validate all the tests, which may take more time than rewriting the code.

Yep. Honestly, I don't know when I last read an InfoQ article all the way through. Too painful.

If you're reviewing the tests (and not just the results) after conversion by automation, wouldn't someone else also review the tests converted by a person rewriting them manually?

This is something people seemingly don't grasp about LLMs. If you want "Human alignment", then you will need humans in the loop either way.

For a test that is already passing, the validation is already there. The human is reviewing the PR, but reviewing a diff is much less time intensive when you can presume it basically works, and you are just assessing if the logic make makes sense relative to the prior code.

> approximately 500 test cases were successfully converted, executed, and passed.

How many of these passing tests are still actually testing anything? Do they test the tests?

Reply on news.ycombinator.com