If you don't include that as "welfare and benefits of the poor" I don't know what you would include. I guess you're right in the sense that only a small portions of ones taxes go to pay people who never worked and never plan on working.
I double checked and you are right that most of SS and medicare go to retired people. They probably aren't considered "poor". Do you know of any unbiased (or low biased) place that counts what percentage of the budget goes to poor people (food stamps, housing projects, extended unemployment benefits, etc.) ?