If no-one uses them, great.
If everyone uses them, you have a lower effective tax rate and different policy outcomes to the ones expected, so there might be distortions.
If others use them but you don't, you're at a competitive disadvantage.
An analogy for the everyman - imagine that there's a loophole me and everyone else in my city could use as individuals to completely avoid tax. If I choose not to do so, I'm receiving, say, 20-30% less income than I would annually, putting me at a huge disadvantage when performing normal activities such as buying a home, say.
I don't think that tax avoidance is immoral, I do think it should be fixed because of the distortion (favouring large businesses, which already generally have advantages due to scale).
It's a bit like how one can be a marxist whilst also choosing to own private property. You can't exist without mimicing society to some extent.