With SRI you can't store unique identifiers in the cache since the cached content has to match the hash in the URI. The only real privacy leak would be potentially determining whether certain data was already cached.
While you're correct, it's too little too late. You should assume that everyone effectively don't have a web cache. It's like the effect of Spectre and Meltdown: JS timers are no longer accurate to compensate for them.