who said there is a social penalty? There are a number of reasons which I explained in a reply below.
This seems to clearly state that you have a negative opinion of you for turning down extra work that is clearly undesirable. I assume that you're not the only one on your team that feels this way either, that's what I meant by social penalty.
Depending on who you ask it may not be a penalty but that would mean everyone on the team has to think this way. I personally don't think people should be considered a bad mate if they don't want to do an optional thing -- what if they value their time more than the pay+extra that's being offered?
I don't know if you've noticed, but time is the one thing you can never get back. Once you make a certain amount of money, you can live very comfortably -- priorities shift to things that you can't just buy, time is basically the most lucrative and rare yet abundant resource there is.
There are other companies to work at, and we make our expectations clear before the person is hired in relation to on call.