I happen to have worked on many document generation systems in my career, I have written systems targetting odt, docx, postscript, and pdf, either from scratch (for odt and docx) or xsl/fo (ps), docbook (pdf) or TeX/LaTeX (pdf).
The most satisfying, easy-to-build, but not the easiest-to-understand, the one that gave the best result in terms of time-to-develop/quality/satisfaction was TeX/LaTeX. So sure, the math can suck in TeX, maybe, I don't know (didn't feel that way at uni when I was using it for that), but as a typesettings low-level language to target, it does a very good job.
Also I fail to see how TeX is an issue, or even how it prevents competition to happen, when all it takes is writing a better front-end language for it.