The threat is not that the cache contains builds of untrusted code but that it contains builds that do not match the code that they are associated with.
As far as I'm aware (ICEs…) compilers aren't hardened against untrusted code, and a sufficiently capable exploit could be used to poison the cache.
ccache uses cryptographic hashing of file contents, in addition to matching compiler arguments, so you can be sure that the code matches.
It uses a cryptographic hash of the _inputs_ to the compiler, but there is no way to verify that the cached artifact matches the _output_ of the compiler without actually compiling it yourself.