Adversaries will simply use common pass-phrase words as though they were part of a larger alphabet. Depending on the distribution/definition of "common", 2-4 word passphrases are then much worse than an 8 character password, for exactly the reason you state.