If all of your friends were jumping off of a bridge, why wouldn't you? If you don't trust them in what they do, why are they your friends?
The minority of people who litter make the experience of walking/living/sitting in that area less pleasant for everyone else, and they increase the likelihood that others will litter ("this park is already full of trash; should I really go chasing after that wrapper that fell?"). Alternatively, you could not make it worse, or you could even make it better by picking up a few things and throwing them out as you pass through.
The minority who don't vaccinate encourage others to avoid vaccinations, and it doesn't take very many of them to undercut or even nullify the benefits of population vaccination (like protecting newborns who can't yet be vaccinated).
Etc. etc..
Do you want to be worse, meaner, less moral, etc. than the average? Or do you want to strive for a little more than that in life?
Realistically, most of us aren't going to have a huge positive or negative impact on the world; but thinking locally, if you are going to seek a mate, friendships, etc., you'll probably do better (and be happier) in life if you at least try to do better than the average.
You may even get an opportunity to make a seriously good decision somewhere along the way, if you have good moral habits and keep your wits about you.