I'm with you. The system, as it stands, is suboptimal, but it's pretty good. If I want to call my brother in Hawaii, it's easy to subtract 5 or 6 hours (HI does not observe DST) and know immediately whether it's a good time to call.
As far as the suboptimal aspects go, DST isn't very nice, and AM/PM is a bit superfluous in the age of digital clocks (compared to 24-hour time).