It's not about it being religious, but about it being dangerous, or in some ways a hinderance/harmful to the self.

Kind of like how self harm is forbidden. Vamamarga is psychological self harm, in a sense. Aghoris then take that same principle to the next level.

The problem is people fetishize, or in other ways make arguments and ideas about Indian origin things within the context of religion. That has in my opinion done far more harm in understanding the philosophical approaches of Dharma than any other misconception, including translation errors.