You can set up default private browsing/incognito mode if you want to. None of the major browsers have support for doing it through the normal options GUI as far as I know, but getting it working isn't at all hard.

In Firefox, set the "browser.privatebrowsing.autostart" about:config flag to true.

In Chrome/Chromium, append " --incognito" when starting to open an incognito window. Alias it or add it to your GUI shortcuts to start it that way by default.

In IE8+, append " -private".