Are they, despite not having most of the attack surface other browsers have (due to not supporting a lot of features)?
I typically send content through rdrview[0] before piping through w3m-sandbox[1], which should be pretty safe. I also only browse one site per w3m instance.
[0]: https://github.com/eafer/rdrview
[1]: https://git.sr.ht/~seirdy/bwrap-scripts/tree/trunk/item/w3m-...
Parsing HTML is difficult.
I'd say, this is a security feature - even if the code is not the most secure.
Out of interest, did you report your findings from fuzzing? Did things got fixed?