I played with this idea, custom extension rewriting ://.googlevideo.com/videoplayback?* webRequests, redirecting to my local proxy server, but in the end it was more trouble than its worth, instead I just have "download with yt-dl" button right under the videos I intend to watch multiple times/later/offline (https://github.com/raszpl/smplayer4YT).
Managing cache in my opinion is also complicated. For how long will should it last? Will you delete it on finish/page close? Or will ppl accept watching yt will take a big chunk of space, and disk performance? What if the browser crash? In mobile it would not be possible.
The way the video is generated is based on specific points in time.
While going back to that part of the video with it already downloaded would make sense, it might require going back decently far to get all of the pixels.
That's a far cry from having to pick out those segments from the memory of my browser as it runs.
Edit: I may have misread your comment, apologies if I misunderstood.