I cancelled Amazon prime a couple years ago and ran into the same issue. It was absurd frankly.

I went through the cancellation process got a cancellation confirmation email and then still got billed at the end of the month. I called and they did reverse the charges and told me it had happened because a behind the scenes box was checked for automatic payments - I scoured the page to see if that was a user facing option and it was nowhere. Cancelling but continuing payments simply makes no sense.