The difference is that white males are rarely a minority in their schools and workplaces. So no, it's not just small expressions or aggressions, there's an overarching social-structural context.
Your logic is so broken I don't even know where to start.