One thing that's immediate: the only math typesetting language serious math writers ever use is LaTeX, and I think in terms of "math mode" support it would be enough to fully support whatever subset of tex syntax is supported on sites frequented heavily by mathematicians, specifically mathoverflow.com (I believe it's MathJax with some extensions for special scripts and diagrams like [1]). It seems like the technology to get TeX->HTML is already there at mathoverflow, so I imagine it wouldn't be too hard to avoid reinventing the wheel.
The other things are less math specific and more organizationally specific. If I were writing a serious academic math text, it would be full of theorems, corollaries, and exercises which I would refer to in later chapters by a label. For example you might put some simple but tedious computation as an exercise in an earlier chapter and then when using the result in a later proof you can say "The claim that BLAH is Exercise II.4.3." Leanpub appears to currently only have support for figure labels, but in math we often want labels for these other things too, which we can then reference across the entire book. In TeX, anything that is assigned a number can also be assigned a label.
Serious math texts also have a need for bibliographies at the end of each chapter and at the end of the book, and if there's anything a writer doesn't want to do it's alphabetize and manage labels for a hundred references. We have BibTeX for this, but Leanpub appears to have nothing. One could write a tool that translates bibtex output to markdown, but again it's a lot of work.
[1]: http://mathoverflow.net/questions/159655/commutative-diagram...