"In future we’ll have to do something, but for Pi 5 we feel the hardware encode is a mm^2 too far."
Sounds reasonable, given a fast cpu & less-than optimal hw-accelerated encoding options. As for that "something", maybe:
1) Drop hw-accelerated encoding and decoding entirely, and use the freed up silicon for much beefier cpus (like ones including -bigger- vector units, more cores etc. Cortex X?). That would be useful for any cpu heavy applications.
2) Include hw encoder for a common (1), relatively 'heavy' codec. And hw decoder for same + maybe others.
3) Only include decoder(s?), like they seem to have done for RPi5.
4) Include some kind of flexible compute fabric that can be configured to do the heavy lifting for popular video codecs.
Combined with:
5) Move to newer silicon node to obtain higher efficiency or transistor budget.
Whatever route a future RPi would go, imho hw-accelerated decoding is much more useful than encoding.