That person may be in your address book, which you've never shared with LinkedIn, but you are in their address book, which they clearly have shared.
No idea how they've gotten access to that, but I've never knowingly authorized a connection to my address books.
The only other mechanism I can think of is that maybe they're getting them via Android or iPhone apps (not necessarily linkedin's app) with less-than-upfront disclosure of what the app's doing. Either way, it's slimy behavior.
Rapportive captures the contact info of every person whose email you open (or who you write to).
This is a serious question, I don't know and am curious.
Harvesting of contact information even without permitted access is less clear cut but there's widespread concern due to the large number of questionable coincidences.