The one problem nobody talks about is that browsers won't cache content served over ssl, which means your website will be a lot slower to your users. That will directly costs you money.
Firefox does cache content - I just ran Firebug to confirm. I can only assume from how fast my site works in other browsers that the same applies to them. What browsers are you referring to?