I thought SSL pages/resources aren't cached like they would be normally under regular HTTP, so there would still be extra latency (for repeat visits) if you ever visited a website that required SSL everywhere (like say Github).
Or is my assumption wrong here?