If you're meeting their expectations it's none of their business in my opinion.
But hey -- you're quite welcome to believe otherwise, of course -- and to throw yourself against the wheels of the legal system on the basis of your "opinion" of how they ought to work, if you like. It's a free country, after all.
I'm sure they can fire you but it seems to be "at will" employment in most of the US anyway.
In Ireland and Canada such a clause would likely fall under being "unreasonable" for low paid employees. Similar to how a "non-compete" is not worth the paper it's written on (excluding when direct financial compensation is paid. Eg gardening leave).