AWS has access to the AWS-managed SSE-KMS keys, since AWS manages those keys on behalf of customers.

I am wondering if AWS has access to data encrypted with the SSE-CMK keys as well?

They state that the SSE-CMK keys reside in tamper-proof FIPS-validated HSMs and not even Amazon has access to them. But it’s a bit misleading because they use envelope encryption. Furthermore, the encryption and decryption are still server side.

They could have better phrased it that, the only advantage of the SSE-CMK keys over SSE-KMS keys is that, with former, the users can define key policies (providing another access control layer similar to IAM) and monitor usage.