"Is it the same for this s y of inputs?" May be fine for a subset of things, but then that's a binary thing. If it's slightly wrong do you score by number of outputs that match? A purely binary thing gives little useful help for nudging a model in the right direction. How do you compare two that both work, which is more "idiomatic"?
[0] https://deepmind.google/discover/blog/alphadev-discovers-fas...
for testing, if you use something like quickcheck, you might find bugs that you wouldn't otherwise find.
when it comes to idiomatic, I'm not sure - but if we're at the point that gpt is writing code that works, do we really care? as long as this code is split into many small pieces, we can just replace the piece instead of trying to understand/fix it if we can't read it. in fact, maybe there's a better language that is human readable but better for transformers to write and maintain.
> when it comes to idiomatic, I'm not sure - but if we're at the point that gpt is writing code that works, do we really care?
Yes - it's certainly preferable. You may prefer working over neat, but working and neat over working but insane spaghetti code.
Remember this is about training the models, not about using them later. How do we tell, while training, which option was better to push it towards good results?
function add(a,b) {
return 4
}
The approach I was thinking of is that assuming we start with the Java program:
public class Addition {
public static int add(int a, int b) {
return a + b;
}
}
public class Addition {
public static int add(int a, int b) {
return a + b;
}
public static class AdditionAssert {
private int a;
private int b;
public AdditionAssert a(int a) {
this.a = a;
return this;
}
public AdditionAssert b(int b) {
this.b = b;
return this;
}
public void assertExpected(int expected) {
int result = add(a, b);
assert result == expected : "Expected " + expected + " but got " + result;
System.out.println("Assertion passed for " + a + " + " + b + " = " + result);
}
}
public static void main(String[] args) {
new AdditionAssert().a(5).b(3).assertExpected(8);
new AdditionAssert().a(-1).b(4).assertExpected(3);
new AdditionAssert().a(0).b(0).assertExpected(0);
System.out.println("All test cases passed.");
}
}
def add(a, b):
return 4
def addition_assert(a, b, expected):
result = add(a, b)
assert result == expected, f"Expected {expected} but got {result}"
addition_assert(a=5, b=3, expected=8)
addition_assert(a=-1, b=4, expected=3)
addition_assert(a=0, b=0, expected=0)