A common argument I've seen relating to fairness is that "government/society should ensure equality of opportunity, not equality of outcome" but we don't even do that:
"Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination": http://www.nber.org/papers/w9873
"Black people twice as likely to be charged with drugs possession": https://www.theguardian.com/world/2013/aug/21/ethnic-minorit...
"Seeing the poor cuts support for a tax on the rich - Visual reminders of poverty do not make make people more sympathetic to the poor." http://arstechnica.com/science/2017/01/encounters-with-poor-...
Instead of developing empathy, we cultivate and celebrate ruthlessness. In my opinion, the widespread lack of empathy is the root of most societal ills.
And these examples and opinions are just about the developed world (obviously, similar forms of them exist in all societies). Imagine the "luck" and "opportunities" you would have if you happened to be born into an impoverished nation or conflict area.