I think it's more than the money. Many engineers view FB and Google as kind of a rite of passage in tech. Ex-Googlers and ex-FBers are highly regarded. Your startup is safe hiring them because they have been vetted already.
I turned down an opportunity to interview at FB in 2008 when they were just starting to build their native mobile team. I was even offered a free flight from Chicago and was told I'd get to meet Zuck. Some of us have refused to work for companies that exploit user data and have always put ethics first. But "person who refused to interview at Google" doesn't sound as nice as "ex-Googler" to most.