Hacker News Reader

Comment by skissane | Hacker News Reader

Thought about automated discovery of laws in an existing codebase?

If you can find a law which the existing code obeys, and show it to a human, and if they agree, save it. And maybe the AI could make a decent guess as to what kind of laws would appeal to a human versus which wouldn't – a simple law identifying a fundamental constraint the system obeys is good, something really complicated or constraining something coincidentally true isn't

Or some kind of measure of coverage? you'd never want 100% – then your laws would become so complex you couldn't follow or maintain them – but if it is very low, that can be a signal to increase it

View on HN

Discover the laws in a current code base?

They are usually in the “tests” folder.

So anything capable of extracting unit tests is extracting “laws”.

One might consider the test name the text of a law. And the AI fills in the details.

I don’t think that’s right.

Unit tests check whether a specific code module is implemented correctly given its (implicit) specification.

Good “laws” are independent of the structure of the code.

Tests may be a useful source for inferring what the laws are, but much of the actual content of the tests aren’t “laws” at all.

I think the premise is more that if one is given a grouping/constellation of unit tests that one can derive generalized laws by looking at what behavior those tests were checking for or against.

if there are a dozen unit tests trying to determine if some regex can escape a sensitive area, then one can derive a generalized 'don't let the regex escape from here' type rule -- or at least you could theoretically. I'm sure in reality that'd be a big minefield much like harness self-skill-writing has been.

you might be interested in property-based testing, which somewhat enumerates tests based on invariance and induction

From the codebases I have seen the "laws" are scattered between unit tests and lots of little tests and assertions in the actual code, plus cultural norms and "do it like this" patterns.

But a good start could be found in the unit tests.

Reply on news.ycombinator.com