For sensitive files with known names, you should be able to set that the cache is only valid for requests from the same host website (as the server might decide to send another response when the host is different). That would fix the sidechannel.