It's not the icon, it's the hash value. If it doesn't match a known hash, you have an imposter. Full stop.
- the site is a careless impostor
- the site is the real deal
- a hash collision
Case 3 must exist by the pigeonhole principle given that the hashes are smaller than most favicons. Otoh, if it does show up, you can exclude it by doing a full comparison.