SSH chat does not ensure that "I know who you claim to be". Only that "I know you're someone who has access to the same SSH key as that someone with whom I was talking yesterday".
In terms of authentication, that is about on par with an IRC "nickbot".