Whitelisting sounds appealing. I was going to say that it wouldn't be effective due to the ease of faking caller ID. However, I must admit that wouldn't be a big deal unless the scammer already knew one of the numbers on your whitelist.
A bigger problem, though, is that I'd like to accept some calls from strange numbers, like couriers trying to deliver something, or emergency calls from loved ones, or customer service reps calling me back about something.
For whitelisting to be effective, we'd have to change other things about the way the phone network and its participants work.
BTW if all your friends also turned their phones off except when they wanted to call someone, you'd seldom be able to reach them, unless you happened to call just before/after they had made a call.