Also, given that these videos are going to be reencoded, which is tremendously expensive, I feel that any optimization in this step is basically premature. Naively launching ffprobe 10,000 times is probably still less heavyweight than 1 reencode.
Right click on a YouTube video and select "Stats for Nerds" to see which format it's using in your browser. AV1 will be something like "av01.0.09M.08".
You've probably watched a lot of AV1 video without realising it.
I'm not sure it's fair to call reencodes expensive. Sure, its relatively expensive to using ffprobe, but any 4 series nvidia gpu with 2 nvenc engines can handle five? simultaneous realtime encodes, or will get up to near 180fps if it isn't being streamed. Our "we have aja at home" box with four of them churned through something like 20,000 hours of video in just under two weeks.
The PSNR/bitrate is much lower for HW encode, but the encode rate is typically realtime or better. That's a great tradeoff if you are transcoding so that a device with limited bandwidth can receive the video while streaming, or so that you can encode a raw livestream from a video capture or camera. It's not so great if you are saving to disk and planning to watch multiple times.