Mitochondrial DNA is one way they could tell neanderthals were a distinct species. Teeth shape as well. It's not just looking at the DNA itself, but the behavior as well, such as isolated reproduction, different adaptations, and ancestors as well. Modern humans can be traced back to a particular ancestor, and Neanderthals can be traced back to the same ancestor, where they branched off from. Neanderthals diverged from homo heidelbergensis around half a million years ago, and modern humans did as well, though later. It's not just about looks.

https://en.wikipedia.org/wiki/Human_evolution#/media/File:Ho...

I think this image explains it pretty well. You can see they branch off, and then kind of get absorbed.