Handling it personally makes a lot of sense. I have children, I don't teach them to immediately run to authority the minute they are offended. You should be able to confront/request somebody behave appropriately without them "giving back a mouthful of abuse". And therein lies the key, if they give you back a mouthful of abuse, then take it higher up. There are absolutely situations where going directly to conf organizers and staff is appropriate, but this doesn't seem like one to me.