http://www.graphviz.org/Documentation/TSE93.pdf
This work is absolutely amazing. I've tried implementing it a few years ago (as an excervise when preparing for Google interview). So naive. I've found out that I have read each and every sentence extremely carefully to get things right. I got quite far but never finished the implementation. (And I did not get an offer from Google either.)
There are research groups (I'm thinking of Tim Dwyer's specifically) that are specifically trying to improve the situation. When you say "it's just a pile of incompatible heuristics", it sounds like you haven't taken the time to try and implement a better solution.
Graph drawing is the kind of thing that seems trivial, right until you try to work on it.
(disclaimer: I'm clearly biased here since I worked with the people who wrote graphviz.)
I hardly think they'd turn away algorithmic contributions, if you've got some.
Do you have an example of your hideous bow ties? I don't usually use graphviz to 'draw' anything, so I don't really have the context to understand your complaint. I've never seen a result I think of as a hideous bow tie.
Page 6 Figure 5.