Unfortunately this won't be supported by the proprietary NVIDIA drivers. It requires them supporting DMA-BUF, which they can't due to those symbols being marked as GPL. Even then, they don't support VA-API.

There is a library that maps VA-API to VDPAU (their other video acceleration library), but that doesn't support DMA-BUF either.

I wouldn't hold your breath that this would ever be supported. It'll probably be easier to modify Firefox to use NVDEC directly, given that FFMpeg already supports that. But I don't think that'll be easy given their reliance on DMA-BUF for composition.