I think you're saying that people who harass should be able to be removed absent a code of conduct. I agree. However, harassment isn't always the overt act you may be picturing, it can be more subtle. Code of Conducts give a single canonical state for what's considered acceptable, but more importantly, give organizers something tangible to point to when there is a violation. In addition to that, they inform people (even the harassed, who may not have considered what's happening to them harassment!) of what I'd acceptable.

As a white, male engineer, I haven't ever found a need for codes of conduct because I've had the privilege of never having been harassed. This isn't the case for all groups, so it's best to have one.