The "penalties" are just what you owed plus interest. I think the IRS will only charge an actual penalty if they think you were deliberately trying to avoid taxes.
Experiences like that aren't unique and are enough to put lots of people off trusting the government to be on their side for resolving disputes like this. Most people don't separate out their state or local authorities from the feds. It's all just "government" to them. And even the IRS can make life extremely stressful by auditing people who aren't all THAT rich to begin with (because people with real money have lawyers who make it expensive to go after them).
I, personally, think the solution is to make the federal (and state) bureaucracy more service oriented rather than treating everything like a punitive law enforcement role. But apparently not many people agree.
Sure, interest after 6 years can accumulate into a nice penalty sum on top of what is owed. If IRS ends up owing you, and they discover it 6 years later, would they send you the refund with an interest as well? I am guessing they won't.
They also sent me some form the year after that so I could declare the couple dollars interest I got from the IRS as income.
This is probably just a practical matter, because accounting for interest during the tax year would be complex/impossible because you would need to know when the various amounts of withholding exceeded the amount owed. I'm not even sure there is a single consistent way of doing this.