Your blocky errors in video playback aren't caused by a pointer going wild due to bad pointer math in ffmpeg. They're caused by bad or corrupted input or an improper seek to a non IDR point in the file. ffmpeg is rock solid for playback of standard corruption-free h.264 streams. This idea that any time you see corrupted video it's a (potentially exploitable) bug in ffmpeg is just not true. Nobody is getting "signal drops" due to ffmpeg problems.
Now if you start fuzzing ffmpeg and giving it wildly noncompliant and broken input, yes, you're going to run into memory errors at some point - which is why there are companies doing just that these days, to find and fix those security flaws. Most of those bugs aren't even in popular codecs (since those are extremely well tested), but rather in the very long tail of obscure formats that ffmpeg supports (try counting how many obscure video game formats it supports some time!), and therefore in codepaths that nobody watching YouTube or Twitch is ever going to hit.