Suppose Alice uses a 14 character password, each character chosen at random from the range [U+0021, U+007E] (e.g., the 94 printable ASCII characters above space). There are 4.21x10^27 or 2^91.8 possible passwords for Alice.
Bob, on the other hand, uses a 20 character password, also chosen at random, but Bob used a much smaller character set. He just used the 10 ASCII digits. There are 1x10^20 or 2^66.4 possible passwords for Bob. (Bob would need 28 digits for his password space to be as large as Alice's).
Bob's passwords come from a much smaller set, and so could be brute forced much faster--if the attacker knew that they only had to search that much smaller set. In most cases, though, the attacker will not know that.
But, a lot of people do use reduced character sets, so I'd expect brute force attackers to give some preference for searching those first--but how much? Would they be likely to find Bob's 20 character all numeric password ahead of Alice's 14 character all-94 password?