On use, the font files are cached for a year. Until relatively recently, that cache would’ve been shared across all sites, so it almost worked the way you described.
Now, the cache is per-site.[1] The arguments for partitioning the cache apply to fonts just as much as to other resources.
There have been thoughts[2] and proposals[3] around ways to make shared libraries without running into those issues, but nothing has really materialised AFAIK.
The broad issue is that 'just lazy load on first use' creates privacy reasons, and the idea of picking a list of popular resources to cache on launch never really goes anywhere.
[1]: https://developers.google.cn/web/updates/2020/10/http-cache-...
[2]: https://hillbrad.github.io/sri-addressable-caching/sri-addre...