Google Cloud Is Having IO Issues in US-EAST1

However, it has been my personal experience that Cassandra handles instances going down much better than it handles instances getting very slow but staying online. In that case Cassandra will work extra hard to just spin its wheels. If you know that is what is happening you can be better off manually bringing those nodes offline.

You need something that provides the block store abstraction on top of the primitives exposed by Colossus/D. Think of something like what modern SSD do in order to work efficiently with the underlying flash memory.

Then you have to hook that adapter in your virtualization stack (e.g. kvm) so you can boot from the volume and mount it from inside the VM. You could implement a kernel module or do it internally in kvm/qemu somehow, but iSCSI provides a straight-forward way to implement this in user-space: you have a process on your physical machine that speaks iSCSI upstream, and speaks Colossus/D RPC downstream.

(I don't know if they still do this but I have a vague memory of somebody describing the stack of an early version of GCP while I was working there long time ago)

But that’s just a distraction. We hear about those outages once in a blue moon, because many rely on them. What we don’t hear about is that any given colo, managed service, or CSP customer’s apps go down on their own, all the time, not because of the colo or CSP.

Such outages are banal, so we forget how much more likely they are, and fail to risk-weight our engineering efforts accordingly.

Many distributed systems try to be reliable by retrying failed nodes' work, but designs aren't always sound.

Rackspace announced that it has completed the acquisition of Onica, an Amazon Web Services (AWS) Partner Network (APN) Premier Consulting Partner and AWS Managed Service Provider.