Yeah, the core issue is really that you can't cryptographically prove which human being is holding the cell phone you're talking to. You can't tell if you're talking to Bob, or someone named Charlie who is pretending to be Bob. Charlie could then hire an accomplice to call Bob and pretend to be
you so you'd both think you were talking to each other.
The threat model here assumes Charlie controls whatever mechanism you used to learn about Bob's phone number / public key / whatever, and that he can convincingly alter anything you say to Bob that involves checking whether you have the correct phone number / public key / whatever.
Certificate Transparency type solutions make it easier to compare notes about whose phone number / public key / whatever is whose, but Charlie can still (with enough effort and resources) defeat them by impersonating everyone you try to talk to about Certificate Transparency.