This is what happened, logically speaking. Though not averaged out in a balanced manner: instead, to the degree that Neanderthal genetics comprise anyone's genome.
Not to say that there weren't interspecies genocides, but we interbred with them and then outcompeted, for resources, the pure species into extinction generally speaking. Which was inevitable, parallel to how homo sapien lineages also die out from simply being outcompeted.
It is likely that the Neanderthal genome was outcompeted down to a percentage at which it was and is competitive. 50-50% Neanderthal-Modern Human? No dice, Darwin. It had to be reduced to a few percentage points, at which point individuals started being competitive. Was the last pure Neanderthal pair murdered, did they die from starvation, or did they just refuse to have kids? It could have been any of those, but the general procedure was likely defined by resource competition.
And Neanderthal isn't the only alien-homo genome mixed with that of modern humans.