Can we all agree that 'human' should be defined as all members of the genus Homo?
(I would definitely call Neanderthals "human" in almost any context.)
So the shared genes alone can't be used as a reason to call Neanderthals human.
Some human had Neanderthal ancestors, some did not, as you rightly suggest. However, all humans and Neanderthals share a common ancestor that's much more recent than our last common ancestor with chimps.
Ie we only have 'chimp DNA' in the sense that we share a lot of DNA with them, and that we haven't changed too much since our last common ancestor. Exactly the same is true of us and Neanderthals.