But the point of ffmpeg is that it can consume pretty much any format. If you let end users upload videos, that is exactly what you need, it’s not desirable to have a very restrictive list of formats you will accept.
Not having massive exposed and untested surface area is very much desireable. Every platform has a highly restrictive list of formats and containers for this exact reason.