The biggest problem with that idea is that it wouldn’t be possible to have hw decoders as efficient as the current ones.

But there is definitely a trend towards more customizable decoders, with a larger set of primitive operations (DCT, motion estimation/compensation, convolution filters) and parameters. The codec space is slowly moving towards a model where the decoders are VM highly optimized for specific tasks.