Comment by mort96 | Hacker News Reader

Comment by mort96 | Hacker News Reader

Parent Full thread

Video playback timing is actually pretty difficult, with only trade-offs, no "right" solution.

The easiest solution is to use your computer's accurate clock to time the frames. This means, if your video is at 30 FPS, you will have gone through exactly 3000 frames after 100 seconds. But your monitor might not be refreshing at exactly 60 FPS, it might be targetting the 59.94 standard, or it might be targetting 60 FPS but with a clock that's a bit less accurate than your computer's, so every now and then, you get a stutter as a video frame has to shown for 1 refresh interval longer or shorter than it should.

What we could do instead is to calculate a fixed pattern to display frames. If your monitor is 30Hz and the video is 30 FPS, we show exactly one video frame for every monitor refresh. If your monitor is 60Hz and the video is 30 FPS, show each video frame for 2 monitor refreshes. If the monitor is 30Hz and the video is 60 FPS, skip every other video frame. This will already introduce some deviation from the "ideal" timing if your display clock isn't super accurate. And if the monitor is 59.94Hz, that's probably close enough to 60 that we want to show each frame of 30 FPS video exactly twice, but that means the video is playing at 99.9% of its intended speed.

10 seconds over 20 minutes is a surprisingly big gap though. Playing every frame of 30 FPS video twice on a 59.94Hz monitor would net a 1.2 second deviation over 20 minutes.

EDIT: Why the downvotes? Everything here is correct according to my understanding of the situation after implementing video playback software and working with other people's. If anything is wrong, please correct me, don't just silently downvote.

Usually playback will be locked to a hardware clock, typically the audio clock. This does mean that the audio clock and frame clock will drift over time, as they're not locked to each other; it doesn't matter how accurate the clocks are, they will drift. This is one source of dropped/doubled frames. It's possible to sync to the video framerate instead, of course, though audio desync artifacts tend to be more noticeable and objectionable than video ones. You can handle this by dynamically measuring the clock drift and variably resampling the audio, but this is relatively complicated and computationally expensive. Generally this type of drift isn't too noticeable, these clocks are usually well within 100ppm, which would result in 1 frame slip every few minutes.

The issue with framerate mismatch is a fundamentally different one, in that it's not a synchronization problem. It's the result of a non-integral ratio between display frames and playback frames, and exists regardless of clock source. So it's the harder of the two to deal with, and is worse in magnitude too at several slipped frames per minute with e.g. 29.976fps vs. 30fps. In the classical approach, this simply results in frames getting doubled or skipped, since it's not tracking the video rate at all, it just asks for the display to be updated at its frame-time, and the display system will either get a chance to show it or not. The slightly more refined approach is to play back at the closest integral ratio (e.g. play 29.976fps video at 30fps on a 60fps display), and resample the audio to match this new rate, but again this is computationally expensive and complicated.

mpv team has a good writeup: https://github.com/mpv-player/mpv/wiki/Display-synchronizati...

Excuse my ignorance, but don't monitors take their clock signal from the host? I would think that synchronizing a stream of frames to the display refresh rate is a mismatch to handle on the display end, not the host end where frames might get doubled or skipped (and eventually introduce long-term timing errors). This is assuming an 60Hz rate advertised by the monitor.

I don't know whether the clock is technically in the host machine or in the monitor, I imagine it depends on the type of connection. But the idea of a different "display clock" and "playback clock" still applies, since a program's own timekeeping based on usleep + gettimeofday will be different from the monitor clock, wherever that happens to reside.

And unless you're using some kind of variable refresh rate, the monitor always, always refreshes on a fixed interval. It doesn't do any "synchronizing a stream of frames to the display refresh rate". But with VSync, you can synchronize your software with the display refreshes, which is how you can decide to draw every frame twice and that sort of stuff.

Pixel clock is generated on the host. All the VSYNC/HSYNC pulse timings are also generated by the host. Monitor has little choice but to respect it, or refuse to operate.

On some embedded platforms, if you can't get a precise clock you need from clock subystem, or massage the display mode timing to fit whatever pixel clock you can generate, you may be running your display at 34 FPS instead of 60 FPS if you're not careful eneough. And you'll barely notice unless you measure it, or try to play video or do anything else sensitive to timing.

Don't forget that while you might not notice a 20 minute video taking 1.2 seconds more than 20 minutes - but you will notice if the audio plays at the correct speed and ends after exactly 20 minutes - and you will probably notice before the video ends.

Yeah, you obviously have to sync audio and video with each other. It's just usually not that important to sync the audio and video with some "true" time as measured by an atomic clock.

Reply on news.ycombinator.com