It's always been about socioeconomic classes, but the best proxy for it is race (or how you look...). If there were something more obvious, such as giant face tattoos or being obese and wearing mismatched/sloppy clothing, then we could use that too to infer which socioeconomic classes they are in and how they are likely to act.
So what you're really saying is that it's about culture. And that class is very highly correlated to culture
Perhaps, I'm sure it's not a one size fits all type thing, but I would bet that humans (and all animals) are wired to create heuristics for things that could impact their safety, and for more advanced species, heuristics that could indicate something that could help them. And probably throw in some power or ego issues in there too, which might not have any other reason other than to pump up one's self worth.
People tend to be quite capable of judging someone else's social class at a glance, though, or at least after a few words have been spoken. Racial cues are a pretty small part of it.