What you did was perfectly legal, and I assume that your state, like mine, has a law about stopping for pedestrians in the crosswalk. I feel the need to ask though why the race of the man who yelled at you bears mentioning?
There is certainly a strong relationship among perceived entitlement/privilege, apparent race and ethnicity, and apparent gender. I shouldn't need to explain this.