Let's assume a word list of 7776 words. All words are lower case alpha. The attacker has pur wordlist. And the attacker knows there are seven words in the passphrase.
That's still 7776^7.
Even if we prune the wordlist ("I'm not using 'zerg' in my phrase", "I rolled 11111, that's not random so I'll roll again") it's still not an attackable space.
So, there are weaknesses if humans roll the 5 dice and get 1, 1, 1, 1, 1 and then say "That's not random, I'll roll again"[1] or they roll a number, look it up and say "I'll never remember 'zerg', I'll roll again".
My buddy Tom always generates UUIDs by hitting the button several times. It gives him pleasure to 'waste' all those random numbers.
When someone rolls 1,1,1,1,1 and rejects that word they've cut down the word list to 7775 words. That's tiny, but it is a change.
An attacker can under certain circumstances exploit this behavior. For example, if they notice you are picking shorter words or words that consist of certain characters, they will have an easier time to crack the password. Whole categories or patterns can be rejected because of that, reducing the search space.
An extreme version of your method, is picking only the patterns that relate exclusively to you, like picking your favorite music or hobbies. Then the whole scheme becomes useless.