That he has it or can get it, yes. Even if he doesn't have it, there are a few dictionaries (/usr/share/dict/words on various popular distros would be a good start) he could try and only guess the probable options in both of them.

Furthermore, it's possible he could figure out which dictionary you were using by another means. If a site that stored your password in plaintext was compromised, that would give him 4,5,whatever words in your dictionary, and using a similar attack on this flawed prng as discussed above, he could narrow down the possible positions of those words, thereby figuring what dictionary is most probable.

The assumption that the attacker can get your word list in this sort of password scheme is actually implicit in how the entropy is calculated; it's calculated assuming the attacker is stringing together words from the same known source in the same general format.