This is optional, though. If you opt in to it, that's somewhat on you. Granted that our social systems could be improved so that natural human tendencies push people toward right actions. However, I think the injustice is far more severe in cases where there's no out that's within a person's power to reasonably grasp.
And "the girls at the next table in the bar," don't know about your friend's car or house, unless your friend is an insufferable jerk who carries pictures of his ride in his wallet. And they likely don't care that much either way.