So you have replaced a debate about values and shared definitions to one predicated on groupthink. Basing your moral decisions on shared experiences and cultural knowledge leads to the exact kind of in-group/out-group thinking that the idea of rights was designed to combat. It isn't a perfect moral framework, but it is one that has achieved a great deal of good in the last few centuries.