There will be a time when computing is so cheap that ads will be injected to the stream in such a way that it is impossible to remove them without real-time AI detector that indentifies the parts where ads are.
It seems inevitable that this is the end game, and I don't really see viable ways around it for realtime playback. For offline playback, yeah, presumably that sidechannel includes enough information to cut out the ads.
Just look how Facebook does it, there is no "Sponsored post" anywhere in HTML, the literally place entire alphabet multiple times, each letter in separate span/paragraph tag, and then use CSS to actually style that into a message of their choice. All of that work just to prevent simple adblocking rules to work.