Bullies instinctively realize that there's an unusually smart person in their tribe who is wasting their gift on fruitless endeavors, thus reducing the survival chances of the tribe.
Of course, our modern society can afford to have smart people play chess. However, there is no evidence that chess skill transfers to other areas. Therefore, until proven otherwise, developing chess skill should be perceived as frivolous, just like developing skill in any other less prestigious game.