Modal styling (ala WordStar and all its descendants—tools requiring you to break flow to find the Bold button), and markup languages like LaTeX or SGML, are crutches to deal with the limited interaction model the TTY + keyboard setup gave us.
There's another, very different interaction model for entering text, though—the IME, where the OS keeps a buffer of keys, presents menus, retargets modifier keypresses to itself, and delivers its own synthesized objects (which don't have to be characters, but usually are) to the application in place of your typing. The TTY brain damage has caused us to think of the IME as merely a domain-specific tool for entering international characters, and sometimes for accessibility. (Or, in a maddeningly singular out-of-domain use, for entering identifiers into IDEs.)
But a "math IME"[1]—or a "rich-text IME"[2], or an "AST IME"[3]—would be much simpler to wrangle than a markup language, or a styling mode. You could stay in flow, in about the same way Emacs' Paredit and Outline modes keep you in flow.
---
[1] As an aside, the program Mathematica acts like a "math IME"—but it's an illusion, and only works for the "text" that stays within its confines. It isn't using your input to directly manipulate a sequence of real Unicode characters (or MathML, or whatever) that happen to render through the OS as a mathematical expression, which you could copy and paste into and out of the program as you please and so forth. It's just using your input to shape an internal tree structure, and then rendering that tree structure itself. It's more like a very clever WYSIWYG markup editor, like LyX.
[2] I've noticed lately that Tumblr's post editor (of all things) is half-way to a rich-text IME given that it can do inline menu-driven text replacement. But it doesn't do the key thing: buffer the user's direct text input at the "menu-driven" layer. Yes, if they did this, and the user was using an OS IME, this would result in a sort of "IME stack", where the user's IME resolved into text in the application's IME buffer, which you would then tell the application to resolve. It works, but it'd be much better if the application could just hand off a descriptor file or plug-in of some sort to extend the OS's own IME with the application's formatting options. (Where browsers could then expose that capability as an HTML5 API, etc etc.)
[3] Seems crazy that it's editors, and not IMEs, that automatically balance bracket pairs, no? Picture a Unicode codepoint for "enclosure of list of length N" for all reasonable N, and a set of combining characters to tell it what kind of list it is (S-expression, vector, set, dictionary literal, etc.) If your IME can intelligently insert those, and your text-rendering system understands circumfix/interfix combining characters, then you get your brackets (and probably commas or whatever else) for free, and it becomes literally impossible to have unpaired brackets or invalid lists, because at the character level, what you have is a prefix-length serialized tree. And modern OSes would handle that fine! (And, most fascinating of all, the difference in such a system between rendering a function call as "(a b)" and "a(b)", or a list as "[a b]" vs "[a, b]" comes down to your font choice.)
[0] http://www.amazon.com/Where-Mathematics-Come-From-Embodied/d...