it drives me nuts that I still have to write IE specific CSS.
Or give up on the idea that everybody has to see you page in exactly the same way - as long as it works and isn't totally broken, who cares?
1. works perfectly as intended in Firefox and Chrome at reasonable window sizes (this usually means it should in other reasonably compliant browsers too)
2. degrades gracefully (i.e. some display discrepancies but no behavioural problems or illogical on-screen layout that inhibit viewing/use of the content) in other browsers (including IE8), and/or when pressed into a very small viewport (such as on a mobile device, though obviously some content is never going to be particularly usable at QVGA resolutions)
3. try not to do anything that will break the design on IE6, but no care overly if that does happen (and give users a warning that it might at the top of each page: the one IE specific CSS hack I use is one to make that message display to IE6 users)
If not, why not?
There are still lots of browsers out there, and they all work in slightly different ways. It's been this way since Netscape came out and was different from Mosaic. It'll continue to be this way until one browser dominates to the point where others go away.
Personally, I'm rooting for Chrome. Like you, I get tired of having to cross-tweak my code. But realistically I find I spend more of my time these days working around FF issues than IE ones.
On a serious note, IE has always been a challenge for front end dev, but its at least up front about its lack of support. Chrome, FF and Safari will sometimes have a minor margin bug that will drive me more insane then the expected behavior of IE.