Its like rehiring an employee every few months then training them up. Its honestly tiring and cant stay like this.
Opus has the same problem too…
This is the csharp target for tsonic (a TypeScript to C#/Rust/Python/Triton transpiler). It has a bunch of tests here: https://github.com/tsoniclang/tsonic-csharp/tree/main/test
More comprehensive e2e proving grounds are at
1: https://github.com/tsoniclang/proof-is-in-the-pudding
2: https://github.com/tsoniclang/tsumo/
They were built specifically for testing the C# target. There are several other large projects we built specifically for e2e testing.
But more interesting would be the tooling built to support this. For example, our current TypeScript parser [1] is a file-by-file port of Microsoft's TypeScript V7 compiler written in golang. The challenge here is that every time Microsoft changes code, we'll have to fix our code and tests. It's doable, but a fair amount of work.
So we decided to write tooling to transpile Microsoft's v7 compiler from golang, and autogenerate our compiler. That tool is called gotots [2] - and it already produces a fully working TypeScript compiler. It's 3x slower than TypeScript v6 compiler, but we hope to get to rough performance parity in a week or so. Everytime Microsoft makes an update, we run gotots and our parser gets updated as well.
[1]: The old parser - https://github.com/tsoniclang/tsts-legacy
[2]: Golang to TypeScript transpiler - https://github.com/tsoniclang/gotots
My general point is that tests and tooling is tremendous value, and they are guardrails for LLMs to converge. I could have, for example, chosen not to write the go-to-ts transpiler, and live with porting Microsoft's parser line by line. But making such tools is something LLMs are good at, so it's a tradeoff well worth making. And the upside is that you don't have to use LLMs to port Microsoft's parser/compiler (a large and complex project) line by line.