In my experience, LaTeX has the advantage of being able to handle even very complex documents and books. LaTeX can be challenging to master, but in return it can produce amazing results. See some of the references below. In particular take a look at [4] (it's a 1200 page LaTeX document) and consider if it could be written in Markdown and Asciimath. The hundreds of graphics appearing in [4] are also produced by LaTeX--the document is the manual for pgf/TikZ a popular graphic package that is itself a set of LaTeX macros.
[1] https://uva-fnwi.github.io/LaTeX/week2/maths2/
[2] https://en.wikibooks.org/wiki/LaTeX/Advanced_Mathematics
[3] https://tex.stackexchange.com/questions/397453/what-are-all-...