> In light of these features, our public decisions will be fairer and more moral once we put empathy aside.

I think the problem is less that we relate more to the few than the many, and more that we don't really relate to the many at all. Junking caring as much about individuals will likely just result in us not caring enough about anyone.