It's an overgeneralization, it's like conflating straight males with people that watch a male raping a female.
Regardless, though, the term "pedophile" by definition is someone who is sexually attracted to children. There is no distinction about whether or not they actually committed a sexual act with a child. A person who commits a sexual act with a child is by definition a pedophile and a rapist, since children cannot legally give sexual consent.