How far down the stack of turtles do you go, though?
Should you include the libc that the compiler requires to run? Other parts of the operating system? The kernel? An emulator for hardware that can run that kernel?
Eventually, all of those things will stop being produced or easy to find. Even if you have the libraries and compiler in version control, can you build a game that ran on MS-DOS 5.x or CP/M or DEC VMS?
My point is that you may want to just designate a stable interface somewhere (a language standard, some libc ABI, etc.) as the part you expect to not change. Be aware of what it is, and account for it in your plans. If a language is the interface that you expect to be stable, then don't upgrade to a compiler version that breaks compatibility. Or do upgrade, but do it in an orderly manner in conjunction with porting your code over.
If you want your code to be runnable by our descendants 1000 years from now, you should probably have an incredibly simple VM that can be described in a picture and a binary that will run on that VM. (In other words, you go down so many turtles that you reach "anyone who knows how to code or build machines can implement this VM".)