There's subtle, dangerous ways this can be exploited. (Short version: It'd make SRI usable as an oracle to confirm or deny guesses for the content of a cross-domain resource.)
You're just trading one problem (someone learning I previously requested a file) for another (leaking referrers to a CDN).
Also, if you're loading "highly sensitive/top secret" data with a <link integrity="" href=""> or <script integrity="" src=""> tag, you have bigger problems.
Obviously, this technique will only work if the contents of that resource are constrained enough that it's possible to guess them with brute force. Depending on how SRI interacts with the browser cache, though, it may be possible to make guesses very quickly -- it is likely that the browser will only fire one HTTP request for the initial attempt, and will load the resource from cache for all subsequent attempts.