The problem is actually really that GitHub’s caching offering is very limited for anything except the most basic of use cases and also they don’t offer a way to colo your own cache with them so that you aren’t paying cloud fees back and forth. You have to use their machines, their storage and their protocol which is only really viable if your definition of caching is literally just “upload files here” and “check if the uploaded built file already exists”.
Yes, I’m aware that buildkit offers “experimental” GHA caching support. But given how fat image layers are it’s basically unusable for anything beyond a toy project that builds a couple layers on top of an alpine image (as of the time of writing this post GHA limits cache size to 10gb per repo. Fine if you’re building npm or pypi packages or whatever, but hilariously inadequate for buildkit layer caching)