That works if you fix everything to a single quality and codec. YouTube videos tend to have something like 20+ variants that can get dynamically switched between while the video is playing (adaptive streaming).

The client can pick the optimal encoding and quality which makes that quite a bit more difficult. A user on a modern device that can hardware decode HDR av1 will have a file cached completely useless to a browser that can only do SDR h264.

I think there is a sort of network effect needed to make it become reasonable.