You can eliminate the problem entirely by using a random full-allowed-character-set password. The whole point of KDFs is that the password/passphrase is predictable (usually: by being generated from a seed dictionary).
50x is not the decider between allowing a good password and allowing a bad password. It's a tiny little nudge.
Edit: To put "little nudge" another way, if you have a 1-10 scale of password quality, most steps in that scale are going to be more than 5.6 bits apart.