This has never happened to me. Did you ever share your e-mail contacts with LinkedIn? (Under your LinkedIn "Account preferences", select "Syncing options", then "Sync contacts" to see if your contact lists are being synced with LinkedIn.)