Interesting, I haven't tried tests outside of the code base the LLM is working on.

I could see other elements of isolation being useful, but this kind of feels like a lot of extra work and complexity which is part of the issue...