So, you're wrong about the entropy in the pass phrase. XKCD goes into the math, and for a sufficiently long phrase, there's plenty of entropy. And yes he takes into consideration there's only so many English words.
Remembering 10 random words in order isn't that much easier than 10 random alpha numeric chars and symbols. The words might have some mnemonic value, but the point is, if you need entropy, don't mess about, use a real private key.
95^10 = 5.9873694e+19
7776^10 = 8.0828128e+38
How much computing power is needed to attack a 7 word Diceware passphrase?