Are they the only ones who tell you, a fellow white man, or are they the only ones who are offended? I strongly suspect (this is not meant to be insulting) that you may not have a relationship with the people who this language impacts. It takes an emotional toll to confide with someone that something is upsetting to hear, even with coworkers or casual friends, especially if you don't think they'd understand.
Also, language has the opportunity to shape and influence how we structure society, even unintentionally. Thats why there's a lot of emphasis on teaching children about what words are appropriate. Even if naming your git branch "master" won't realistically impact racism and the history of slavery, describing some bad event in your day a word that has an origin in describing people can lead to negative associations.