Generally that's not actually breaking the law. Avoiding accidents, following reasonable directions from road crews and the like explicitly override other laws.
Legally.
The law where I am is that human traffic directors (police, sheriffs, construction crews, etc) take precedent over what's in the book. If they didn't, nobody could ever get around a fire truck at a fire.
I just ignored him and went around when it seemed safe.
And that's where you get a ticket. Plenty of cops around here sitting at the exit of construction zones waiting for people just like you.