Here's a thought experiment that might explain it.
Imagine you are at some form of social gathering and someone starts going round with a jar collecting for a charity. Now, if you witness someone not putting any money into the jar you're not necessarily going to think bad of them. There could be lots of reasons why they are not donating in this instance and it's not mandatory. However, if you saw someone put their hand in and take out a fistful of money you would be outraged, because this person has stolen money that other people have donated to the charity - and most normal people will find such an act distasteful at the very least.
This is the same. It is the difference between someone not paying a tax and someone stealing tax that other people have paid.