There's also integrity of your site, and many more future proof reasons. These days, there's no reason _not_ to use HTTPS, really.
https://developers.google.com/web/fundamentals/security/encr...
http://www.earth.org.uk/note-on-carbon-cost-of-CDN.html
Edit: I note that I got downvoted, though no reply with reasons. I accept that it is currently widespread dogma/fashion that everything should be HTTPS, and there are some good reasons for it, but as someone that has worked in what might be called 'real' security in on-line finance for example, I fear that HTTPS-everywhere is as much security theatre or cargo cult as taking your belt off at the airport scanners...
That's not worth a slow-to-load or even failing-to-load page.
It's important to make sure that inoffensive sites are behind the same barrier, to make that a poor signal.
I don't believe that any penalty is currently significant, nor likely to be soon. There's plenty of good historic sites which SEs don't want to exclude just because someone doesn't have the resources to rework them and keep them maintained with the latest TLS. (Note issues like embedded http resources that would all have to transitively converted to avoid warnings.)
https://webmasters.googleblog.com/2014/08/https-as-ranking-s...