If we can agree that the VAT is levied because the government has added some value to each product sold (value added tax..) then I don't see why spending money on VAT-exempt items makes VAT a bad tax.
If you spend it on labor, housing, there are taxes associated with that other than VAT. Lobbying and charities might be tax-deductible or exempt, that is another discussion.
I think your point is that regressive taxes are bad, is that a fair analysis? I don't agree with that.