E.g. an attacker can go "Oh! this password will take FIVE SECONDS to test, so I know it must be a simple password." or "Hey, check this out; this password can be tested in 0.1 seconds. It must be pretty complex."
In general, I'd guess that these kinds of information leaks are pretty bad because if an attacker can see how hard a password is to test, he now knows something about the password.
It may be better if, given a single unchanging hash, if it takes a variable amount of time to test a given password against this hash, though that might have its own can of worms.