I like the idea of choice that's available when you need it, but hidden when you don't. Take Firefox for example. It does a pretty good job of providing features that you need most of the time either in the interface or in the menus.

More advanced options are available in the logical place. Even those are carefully organized in a logical manner. When you select an option in the tools panel, you have a pretty good idea of what will happen.

If you really want to get crazy, though, you can open up about:config and be presented with a huge variety of options, many of which make no sense unless you know what you're doing. It kind of reminds me of the advanced options in Internet Exporer (with Carat Browsing and all), except that it's well-hidden from the average user. Those that need it know where to find it. Those that don't will never stumble on it. I think that setup avoids a lot of what Joel is talking about.