OP here. Sadly, SVG file size is not as small as I'd want it to be. The main problem is that a few attributes (x, y and lengthAdjust) have to be repeated on each text tag to position each character at the right place. I can't use CSS for this since those attributes are not styling attributes.
Using SVG definitions efficiently to avoid duplication is also not as easy as I thought it would be.
I have a few ideas on how to reduce file size, this is the next improvement I'm planning to work on.
For the examples shown at [1], file size ranges from 56kB to 252kB, the worst case scenario being lots of single characters with different colors.
[1] https://github.com/nbedos/termtosvg/blob/master/examples/exa...