The Fortress programming language was a research project at Sun that incorporated mathematical notation.

See page 24 of http://www.oracle.com/technetwork/systems/ts-5206-159453.pdf

They also had an ASCII-only representation for everything, so you could use a regular text editor if you wanted, kind of like Markdown.