It's not my field of expertise but would AFL help here, or fuzzing in general (or in before anyone says Rust)
Re Rust: the problem here is hardware-acceleration, as far as I can tell. Even if we had a pure Rust H.264 decoder, you'd probably still want to use whatever your hardware has to use overall fewer resources. The drivers might be the place to look, and there's some progress on that front in Android for example, but as things stand fuzzing like that is extremely valuable.
It will take ages for AFL to generate a valid H.264 NALU that isn't rejected outright.