HTML with MathML may make a decent system as well; possibly with an XML source and XSLT for templating, which is apparently how OpenStax textbooks are composed (via CNXML, though that also has just a couple of rain frog pictures in its documentation repository -- seems to be a common pattern around typesetting systems).
Then there is troff with eqn(1), which looks simpler, but not sure if there is an actual specification for it around, either.
And then there are Texinfo, org-mode with LaTeX embedding, other TeX-adjacent options, perhaps Markdown with HTML and MathML embedding.