Benchmarking AWS, Azure, & GCP in the 2021 Cloud Report

Hey yuncancode - Regarding placement of machines, we only controlled the placement of machines in the same availability zone in the us-east region. We did not use more granular controls, such as AWS's Placement groups, Azure's Proximity placement groups, or GCP's placement policies. We did this to ensure fairness in the test between the cloud providers and to align with a common pattern of distributing machines within an availability zone to maximize availability.

Fairness: Each of the cloud provider's have varying placement systems. According to AWS's documentation, AWS's Placement groups allows one to choose among three different placement strategies: Cluster, Partition, and Spread. On the other end of the spectrum, Azure's Proximity placement groups are focused on a single strategy of minimizing network latencies. In the end, we decided to go with the default placement of machines for each cloud by not using these granular controls.

Maximizing Availability: As discussed in more detail in the Network Benchmark section of the report, there is a tradeoff between maximizing availability and minimizing network latency when placing machines. For this report, we intentionally did not use the Cluster strategy. We oriented our tests with the common pattern among CockroachDB use cases of distributing one’s machines across an availability zone to maximize availability. Depending on one’s use case, it also may make sense to focus more on minimizing latency.

Hey all — one of the authors of the report here. We ran 1,000 benchmark tests across 54 machines from AWS, Azure, and Google. The team will be available to answer any questions you have.

Hi John, on the page it says "We do note that when it comes to placement policies, it is important to keep in mind possible randomness in the physical distance between instances and the correlated effects on network latency." So are any of the placement group / placement policies features used in the test? e.g. AWS Placement Group, GCP Placement Policies?

Thanks!

The report says that AWS's ARM-based Graviton2 cores were tested. Where can we find the download link for the CockroachDB release that was used to test on ARM?