I agree. And on top of that, neither of us have actually met John Brown face to face. Our _entire_ knowledge of him is based on indirect interactions. We've seen things he's (ostensibly) done, artifacts of his work, we've deduced a set of characteristics about his nature based on his behavior. In a sense, John Brown is nothing more than the label we give to the entity having characteristics A,B,C,D.

So like you say, if the overlap between your A,B,C,D and my A,B,C,D is sufficiently small we should probably acknowledge that it's not the same entity. See also: Ship of Theseus.