People want to believe that their medical condition(s) are a special, one-in-a-billion mix requiring years or decades of rapport-building and conversations. That's not the case for the vast majority of people, even with very complicated medical histories and diagnoses.
"Has seen me in the past" and "has my charts" are not indicative of medical efficacy. There are doctors I have rapport with, absolutely. I still have lunch occasionally with the physician who was my family doctor as a child and still sees my parents. But that doesn't mean his medical treatment is going to be any better than a doctor seeing me for the second or third time.