Right, so I've currently got CloudFront in front of it, but doesn't that move the problem? Now the connection between CloudFront and S3 is unencrypted.

(I'm probably understanding this wrong, but I'd like to understand why.)