Comment by iambvk | Hacker News Reader

Comment by iambvk | Hacker News Reader

Parent Full thread

> But you don't need an atomic clock to get Spanner's guarantees.

I always hear this from CockroachDB folks and fans, but no details. What are the downsides?

As per my understanding, if a server/replica could not catch up wrt time (i.e. goes out of sync), it can be identified and usually be marked as a temporary-replica-failure -- which is the maximum extent that Hybrid Logical Clocks can help us with. Rest of the system has to deal with consequences: A new replica must take it's place to keep-up the fault-tolerance-level and start making-a-new-copy -- which is in the order-of the amount of data stored in the out-of-sync replica. I assume this puts lot different design requirements on the storage-engine; also, making-a-copy and cancel-copying operations would eat network traffic, thus throughput.

IIRC Google Spanner uses atomic clocks only on few severs per data-center because cross data-center latencies are much higher and erratic (over internet). So CockroadDB has much higher rate of temporary-failures due to out of time-sync and associated downsides. It would be helpful if CockroachDB guys could shed some light on this.

(Cockroach Labs CTO here)

> > But you don't need an atomic clock to get Spanner's guarantees.

This comment continues "...but GPS clock-sources will do just fine for Spanner's 6ms quantum". Providing Spanner's guarantees with reasonable performance requires specialized hardware, but there are more options for that specialized hardware than just atomic clocks.

Note that Spanner itself uses both atomic and GPS time sources according to Google's publications; when we talk about "atomic clocks" we're usually talking about the entire category of specialized time-keeping hardware instead of distinguishing atomic clocks from GPS clocks.

> I always hear this from CockroachDB folks and fans, but no details. What are the downsides?

As we describe in our blog post (https://www.cockroachlabs.com/blog/living-without-atomic-clo...), CockroachDB on commodity hardware provides a slightly weaker consistency model than Spanner (serializable instead of linearizable), and latency is sometimes higher as we need to account for the larger clock offsets in certain situations.

If you do have a high-quality time source available, we have an experimental option to use a Spanner-like linearizable mode.

The CockroachDB team did put up a blog post that talks about the tradeoffs.

The pertinent part was: "Spanner always waits on writes for a short interval, whereas CockroachDB sometimes waits on reads for a longer interval. How long is that interval? Well it depends on how clocks on CockroachDB nodes are being synchronized. Using NTP, it’s likely to be up to 250ms. Not great, but the kind of transaction that would restart for the full interval would have to read constantly updated values across many nodes. In practice, these kinds of use cases exist but are the exception."

https://www.cockroachlabs.com/blog/living-without-atomic-clo...

The clock thing is highly dependant on workload. If your load is append only, clocks within a few seconds would be fine. If you truly have competing writes for the same key within ms, then clocks need to be pretty good. Most big data stuff I see is closer to the former.

Reply on news.ycombinator.com