The public debt, obviously.
> If I stop working and don't pay tax, I didn't rob you.
Depends if you have income from capital.
> Me paying less tax doesn't rob you.
Depends on the tax code.
> Stupid spending of your taxes on things you don't need or want robs you.
Fair.
The argument comes down to the "fairness" of society. What level of taxation is "fair"? There's probably not a single right answer.
However, there are answers that result somewhat deterministically in rising inequality. So for those of us who don't want to live in a highly inequitable society, there is a "more right" answer.
Now, I'm sure other people (presumably people who are doing well under high inequality) would think the "right answer" is to increase inequality as much as possible.
Personally, I don't think that kind of society would be very nice to live in.