"A species is often defined as the largest group of organisms in which two individuals can produce fertile offspring, typically by sexual reproduction."[1]
As far as I know there is no evidence that neanderthal females could produce fertile offspring with a male homo sapiens -- but only the other way round.