And I think that's part of the problem. I've done my taxes (in the U.S.) manually for the past couple of years (i.e. filling out a 1040 directly, no TurboTax etc.), and the amount of times I've read through some absurd item that is clearly only in the tax code to get the vote of some special interest is depressing. Like most 'benefits', there would be uproar if you tried to take them away now by the people impacted, but maybe they could be moved out into separate programs.
Seems like getting to a point where ~75% of the population didn't need to file would be doable if there was enough motivation for it. If your only income is on a W-2, and your only mortgage / banking / trading is with U.S. companies like Chase or Morgan Stanley, no reason that couldn't all be done for you.