I've heard stories of LLMs changing tests to get all the tests to pass instead of actually fixing code. So I'm not sure I trust it just because tests pass. I also think that tests do not and cannot check everything.
I also don't trust it just because it compiles in Rust. The Rust compiler does not check everything.
So at what point do projects like this cross from "untrusted" to "trusted"?
That's very easy to prevent. Don't let them edit the tests! Run the test suite against a reserved copy.
But the article talks about how so much code was written so fast. Seems to me that to create that much code that fast you have to have AI produce both the code and the tests.
So I am not sure in this project if the AI can edit the tests or not. I assume that because this is Anthropic the AI is doing as much as possible, which would include editing tests.
You can go and check if the AI edited the tests yourself: look at the git history of those files in the public Bun repository.
In general I would agree though, and would take it one step further: the agent writing the source code should not even be able to see the tests, nor the agent writing the tests see the source code.