One of the bigger issues of the voicemail system is that they typically use the phone number of the subscriber as an authentication token.
This means that if the voicemail system sees an incoming call from a specific phone number it assumes that the phone number is authentic and allows access to the voicemail system.
Simply by googling for "spoof phone number" you will find a host of services that still provide this ability to people for a few bucks a month.
You should get in the habit of setting a pin number on your voicemail, and most of them do support this (it is the way to pick up your messages when you don't have your cell phone) - it is a little less convenient but good security usually requires a change of habits.