It's not about handouts. And you can have privilege in one domain and lack of privilege in others.

Are you a white male? The fact that you had a hard life doesn't negate the fact that you derive certains advantages from being white, and male.

You say you've had it hard. That's an absolute measure. Privilege is a relative measure. If you had had the same circumstances, except you were black, your life would be hard-ER. A relative measure.

You should take a look at the linked privilege checklists. For each item that applies, ask yourself: would my life be hard-ER, in a relative sense, if this weren't true for me?