I've always felt your definition of "most unfair" to be a great fallacy. Yes poor people pay a higher percentage of their income. They also receive more in benefits than they give and the flat tax ensures the rich pay their share. The more complicated you make the system the more it will benefit the rich.
VAT is a net benefit for the poor. Making the whole system work that way, particularly in the face of globalization, is the best course of action. To declare it unfair is to cut off your nose to spite your face. In my opinion.