Couldn’t you do adaptive Bitrate and start streaming low-bitrate for a few seconds and then switch to higher quality once the video is already playing?
To support seeking you could encode a low bitrate stream, and a high quality stream, and then a number of ramps between these. So when you seek you start with the low bitrate stream and then after a few time units go on the ramp to the high quality stream.
… while yeah… a lower bitrate upfront might lower the required bandwidth and thus, latency, to get enough of a buffer to start playback … all the bloat on the page would be a better first port of call.
How much is loaded for a navigation from one video to the next?