Once hackers realise people are using ~4 random words for a password the entropy will decrease hugely.
In practice the attacker must cast a wider net because he doesn't know exactly which word list you use, or if you are using a completely different password scheme. This increases the difficulty.