It's annoying, but the trouble is overstated. The common version of this statement talks about jail! As long as you're not actively evading taxes, the trouble consists of some penalties and interest. Better not to have it, but also not a very big deal.