For better or worse, disadvantaged minorities always have to consider this. It's probably one of the biggest privileges that white males have in day to day life -- they rarely need to consider if race/gender played a role in any decision they encounter.
There was an interesting experimental study that showed that a white male just released from prison was as likely to get a job callback as a black male with no criminal record [1].
There are typically multiple factors at play for any employment decision, but race and gender loom very large -- large enough that ignoring it is just ignorance.
And note, that the domain matters. Jeremy Lin faced discrimination as a basketball player being Asian. But for most domains, being a white male is preferred.
[1] Discrimination in a Low-Wage Labor Market: A Field Experiment, Am Sociol Rev. Oct 1, 2009; 74(5): 777–799, http://www.princeton.edu/~pager/ASR_pager_etal09.pdf