Rightly so. Children can't (as reliable as you) decide how safe it is in a given situation to cross.
For the most part, this is not a problem, people just explain their kids that these are things that they cannot do now but will be able to do later (hopefully explaining why).
But people don't put crossing the road at a light in those terms, they tell them nobody can safely cross the road safely when the light is red. So an adult doing just that and being safe directly contradicts your message and your authority. It makes you look dumb. And that's why they get riled up.
We should just treat the red light as an aid for the young and infirm and explain it to kids as just that.