It leaks private user info -- a malicious server could include a JS file confirmed to be highly sensitive/top secret, and measure whether the client already has that cached. If so then the user is confirmed a sensitive target.
Obviously, this technique will only work if the contents of that resource are constrained enough that it's possible to guess them with brute force. Depending on how SRI interacts with the browser cache, though, it may be possible to make guesses very quickly -- it is likely that the browser will only fire one HTTP request for the initial attempt, and will load the resource from cache for all subsequent attempts.
You're just trading one problem (someone learning I previously requested a file) for another (leaking referrers to a CDN).
Also, if you're loading "highly sensitive/top secret" data with a <link integrity="" href=""> or <script integrity="" src=""> tag, you have bigger problems.