Video startup times are kept low in modern clients by starting at low bitrates and then seamless adaptation up. This is very important for advertising companies, social media, live streams, etc. Every second of startup time cost massive user loss in live streaming (this is well researched and documented). Additionally streaming protocols (HLS/DASH) force files to be encoded such that a full download is not required for a decode to happen. This is not a requirement (and pretty rare unless you know what you are doing with an encoder) for file based workflows.

Fast forward for low power clients relies on an "iframe only" track as there decoders can't do many x realtime decode. This is not present in modern file based work flows.

Seeks use the same startup/segmentation requirements as video startup.

Thumbnail scrubbing requires a thumbnail track and is not supported by file based workflows at all.

Clip previews on sites load at low resolution/bitrates and can be poped up to "full resolution" via ABR. Doing a preview of 10x 4k streams without abr wouldn't work even on a modern gaming rig.

I'd love to hear how you'd do SSAI on top of a file based workflow.

You could of course build a "file based" implementation that had all of these features, but you would just be rebuilding DASH or HLS.