The taxonomy of archaic humans is debatable to say the least so "we are only the ones left" may just be an artifact of 19th century anthropology and prestige chasing. Realistically the genetic variation between the average human and Neanderthals is only about twice the variation within the modern human gene pool. Compared to our nearest primate at 10-50x the difference depending on how you measure.

Given the number of genetic bottlenecks we have evidence of in our past, we may all be the same species and the modern human is just the last surviving group of inbred Homos.