You're missing the cultural insight needed to understand caste. For example, two people from New Jersey may tolerate each other even if one of their ancestors are from other places. Castes on the other hand, carries on from generation to generation. It's extremely hard to get rid of the caste label you're born with. Another factor is the level of discrimination involved. Caste is widely used to judge academic and professional proficiencies. It's extremely hard for lower caste people to rise up even if they work hard. They get denied and often demotivated in public. It gets much worse within India. Lower caste people are kept away from good residential areas, many commercial establishments and even village wells.