For one thing, this approach seems to inherently conflict with the fact that you can link directly to a particular timestamp in a YouTube video, either in an external link using the `&t=...` URL parameter, or by just including a timestamp in a YouTube comment.
So unless YouTube is willing to break all those links or drop timestamp linking entirely, then it seems to me that they'll have to give clients a way to convert "original" video timestamps into corresponding offsets into the delivered video stream. And if you have that, you have enough information to locate the ad segments, at least to the nearest second.
Or, another option: Presumably the server is just muxing together pre-encoded video segments corresponding to either the original video or the ads. So one could create a database of segment hashes, look for identical segments that appear in multiple videos, and add them to a crowdsourced database of known ad content.
It might be difficult to use these strategies to block ads interactively in a browser, by patching the behavior of YouTube's player, but they ought to still be usable in non-interactive tools like yt-dlp.