Something like one mule (horse-donkey hybrid) in tens of thousands is fertile, having by chance received just the right combination of chromosomes to be able to reproduce. But the rest are sterile. The rare successful interbreeding doesn't make horses and donkeys the same species either, despite being closely related. (In their case, IIRC, the infertility of hybrids has to do with differing chromosome counts between horses and donkeys.)
So we can have some Neanderthal genes (whether from prior to y-chromosome differentiation or from the rare successful interbreeding or from successful and fertile but rare female offspring) but still be separate species.
Besides, it's oft cited that we share 98% of our DNA with chimps, and nobody's claiming that we're the same species.