If you are doing anything with equations, LaTeX is definitely the way to go. Unfortunately, it's a steep learning curve for regular ebooks.
The downside is it's not super HTML-friendly (there are converters, but you can't display equations easily in HTML) so converting to Mobi, ePub, etc. is not straightforward.