Hacker News Reader

Comment by pwarner | Hacker News Reader

I assume AWS will start using graviton for their S3, EBS, ELB etc services aggressively?

I believe they are already using them for ELB’s. It is mentioned in their announcement that they will, but I read somewhere else it was already happening: https://aws.amazon.com/blogs/aws/coming-soon-graviton2-power...

They announced it is their intention to have all their services and SaaS running on top of Graviton instances. i.e They are eating their own dog food. They said this will be a long time, multi year method. ( And whenever Amazon is saying this that means they are doing as fast as possible to optimise for cost )

The S3 protocol needs MD5 hashes, which these benchmarks don't cover. A Xeon core can do that with a couple of hundred MB/s. As it's a continuous hash of the whole object, this limits S3 performance.

I guess AWS does not care that much about single stream S3 performance but will optimize for costs.

I don't think it uses md5 any more, the newer protocol versions support sha256 hashes and thats only if the user checks, may not be what is used internally.

Link? The eTag of an object is still the MD5 AFAIK.

The request signature (assuming version 4) is SHA256, the object eTag is MD5.

Yes, S3 uses the content-md5 header to verify data integrity for puts.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjec...

For the ETag, yes and no that it's an MD5 of the object depending on encryption and multipart upload: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonRe...

For the object upload, AWSv4 request signature uses SHA256 on the payload/object, but I don't know if S3 also computes & compares the digest or just uses the x-amz-content-sha256 header value.

AWS support via Twitter said that the SHA256 digest of the payload is calculated and compared for S3 when using AWSv4 signature requests.

Do you happen to know if AWSv4 with S3 also computes & compares the SHA256 of the payload/object or does it just use the specified x-amz-content-sha256 header's value?

FYI S3 Object Lock requires Content-MD5 header.

AWS support via Twitter said that the SHA256 digest of the payload is calculated and compared for S3 when using AWSv4 signature requests.

Offloading a hash function to a custom chip seems appropriate. I suspect they are deploying an entire specialized box, not just a custom processor.

Edit: Nevermind, I may have been thinking about the wrong end of the pipe.

The MinIO code is highly optimized for the md5 hashes using AVX512 https://blog.min.io/accelerating-aggregate-md5-hashing-up-to...

Reply on news.ycombinator.com