One key question is if the teams are being effective referees or just people titled "referee".

If it's the latter, then getting rid of them does not seem like a loss.