What you're missing is that a 59 character password (with secret length) is extremely secure,
despite the even larger speed boost.
If you worry about any speedup in password cracking that is less than an order of magnitude, your password was too close to failing to start with. Make your password 5% longer, which will make it at least 20x slower to crack, and then you won't have to care if "20x" gets reduced to "15x".
You may say "It's not harmless to give up 25%. What if I give up 25% several times? That could make even a good password become insecure." but there's a limit to how much speedup someone can get from knowing the structure of your password. And the best way to evaluate the strength of a password is to assume that all the structure is public. So I can say that my typical passwords, being 20 mixed-case letters and numbers, all have a security of 2^119. It's possible that an attacker that uses the wrong algorithm would have to guess even more, but I'm not just worried about a clumsy attacker, I'm also worried about a moderately-high-quality attacker. It's a bad idea to depend on that extra .1 bit I could get with this character set, or that extra .4 bits I could get with a smaller character set. Just assume the length is known.