The default cache action also has issued with anything that isn't owned by the runner user, and caches are per-repository, so you can't just have one cache like you do for binary caches.