Part of being a grownup is doing whatever the hell you want. If you don't want to pick up that phone (and have a potentially awkward conversation with some sleazebag recruiter that's trying to sell you on a shit job you don't want), don't.
This reminds me of the people that think women (or men) that don't want to go out with them owe them some kind of explanation. No, dude. Get some social awareness and move the fuck on.