I actually implemented this idea using Twilio almost a year ago and have been using it actively ever since. I give my close friends 4-digit PIN numbers to my condo. When someone dials my number in the directory, it first prompts them for a PIN. If they enter a valid PIN, it welcomes them by name (my friends love this) and unlocks the door. If they enter an invalid PIN or no PIN, Twilio rings my and both my roommates' cell phones and connects to whomever answers first.
I also can create one-time-use PINs and one-day-only PINs for parties or whatnot.
As an extra plus, I used to have a home phone because my intercom requires local phone numbers. Since I built my "doorman," I don't need it.