"Any modern CPU does AES much faster than the internet connection most people have"
But the server side, which is serving thousands or tens of thousands of these 10Mb/s connections, is a bit more challenging.
For high-bandwidth CDN applications (like Netflix, Youtube, etc), the drawback to SSL encryption is indeed due to the bulk data encryption. The penalty is twofold. First, the need to encrypt the data requires that the CPU have access to it. Even discounting the costs of computation, the need to touch the data removes a lot of benefits of sendfile / splice / etc. There is an interesting Netflix paper about this: https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf
Due to the nature of SSL, applying hardware offload solutions is problematic. The most efficient hardware offload would be a PCIe card that does encryption AND TCP offload. But then you're dealing with TCP offload, which is seldom a good idea. If you use an encyption-only hardware solution like the Intel QuickAssist cards, then you wind up in danger of running out of PCIe lanes. Eg, to serve at 100Gb/s per CPU socket, you need 16 PCIe Gen3 lanes for network, 16 PCIe Gen3 lanes for storage, and 16 PCIe Gen3 lanes for Quick Assist. So you're up to 48 lanes, but a Xeon e5-xxx v3 CPU has a max of 40 lanes.