Hacker News Reader

Comment by tareqak | Hacker News Reader

When I saw the test suite that Peter Norvig created for the program, I immediately thought to myself “what if there was a LLM program that knew how to generate test cases for arbitrary functions?”

I think a tool like that even in an early incomplete and imperfect form could help out a lot of people. The first version could take all available test cases as training data. The second one could instead have a curated list of test cases that pass some bar.

Update: I thought of a second idea also based on Peter Norvig’s observation: what about an LLM program that adds documentation / comments to the code without changing the code itself? I know that it is a lot easier for me to proofread writing that I have not seen before, so it would help me. Maybe a version would simply allow for selecting which blocks of code need commenting based on lines selected in an IDE?

View on HN

How about the other way. I define a few test cases and the AI writes code for a generalized solution. Not just code that regurgitates the test cases, but that generalizes well to unseen cases. You'll notice this is simply the machine learning problem restated.

The next step could be to have the AI write code that describes its own reasoning, balancing length of code and precision.

> I define a few test cases and the AI writes code for a generalized solution

How about the AI never writing any code, just training "mini AI" / network that implements the test cases, of course in a generalized way, the way our current AI systems work. We could continue adding test cases for corner cases until the "mini AI" is so good that we no longer can come up with a test case that trips it over.

In such future, the skill of being comprehensive tester would be everything, and the only code written by humans would be the test cases.

> How about the other way. I define a few test cases and the AI writes code for a generalized solution. Not just code that regurgitates the test cases, but that generalizes well to unseen cases. You'll notice this is simply the machine learning problem restated.

I think Barliman does something similar (http://minikanren.org/)

That's potentially more helpful than writing the code itself. Writing unit tests can take most of the development time.

And literally throwing random half junk unit tests at your code will better test it than you writing unit tests that are blind to the problems it might have because you wrote both and both bits of code have the same blind spots.

We should probably be developing systems that fuzz all code by default.

In the future, code beyond all human comprehension will be rated for reliability by the number of layers of garbage unit tests it's buried under. Tests for the code, then tests for the tests, then tests for those te-...

I warmly recommend QuickCheck-inspired libraries.

QuickCheck (not NN) generates test cases.

GPT can already comment on code.

Reply on news.ycombinator.com