Based on all this is does seem like there's a 'missing' option to only swap the fonts when they’re off-screen, which avoids:

1. blocking first-paint 2. 'flashes' of anything 3. Not showing the font at all.

A quick mockup: https://codepen.io/pjwdev/pen/ByoLJoR

(This is an intentionally bad fallback. If it were Georgia the effect would be a whole lot more subtle.)