The key here is that the professionals claim it fundamentally changes the game. Not only does it allegedly drastically change the dynamics of how the rock travels, it also allegedly takes most of the skill requirement out of shooting and puts it into sweeping. Not to mention it also supposedly causes significant harm to the playability of the ice surface, which is paramount in curling.
Somebody posted a video on youtube (https://www.youtube.com/watch?v=ScelTWytLFQ&feature=youtu.be) of a draw shot that the sweeper was able to manipulate the rock into traveling a straight line, while a rock a few feet over curled five feet. This sort of manipulation completely changes how the game is played, both physically as well as strategy.
Now, I'm still not sure I believe all the claims, and the video leaves a lot of unanswered questions, and some claim it's just a smear campaign by BalancePlus against competitor Hardline. I don't know what to believe, and am reserving judgement until the evidence is more clear. BUT, if these brooms can in fact do everything people claim they can, I would be in favor of disallowing them.