Being "blue collar" and "from a small town" probably set you back in at least 3 or 4 different kinds of priviledges (as in economic, well learned, and urban/cosmopolitan). So, if you compare yourself to black women that had all advantages, yes, your life more often than not will turn to have been harder.
If, on the other hand, you genuinely want to understand what "white male privilege" is all about, you should look at how women of racial minorities fared in your home town, or other similar towns. And even greater insight can be gained by observing how people treats other white men from the same town who are otherwise disadvantaged: guys with physical or cognitive disabilities, gay men, men with "foreign" accents (specially if their speech ressembles that from a not-so-far rival town), or for the matter old folks. Even if they are treated "fairly", the small indignities they have to endure on a daily basis may build up to something significant.