This torque issue was really bothering me, and I finally realized why: I was pretty sure I'd already seen torque represented by a letter other than tau. Sure enough, I checked my copy of Introduction to Electrodynamics by David Griffiths, which is a popular (and truly great) intermediate E&M text, and there it was on p. 162: torque written as N = r X F. The same usage appears in several other places in that book, and Introduction to Electrodynamics is a standard text, so I'm confident that other sources use it as well. So we see that the idea of using a letter other than tau for torque is not some theoretical possibility—it has already happened, and in a standard textbook to boot. I've updated the manifesto with a note to this effect at http://tauday.com/#sec:four_arguments.
I realize that this may not convince you, but I urge you to reserve the right to change your mind about tau. After all, I changed my mind about pi. ;-)