Comment by derefr | Hacker News Reader

Comment by derefr | Hacker News Reader

Parent Full thread

Specialized, but not proprietary. Spanner presumably uses off-the-shelf Chip-Scale Atomic Clock (CSAC) modules with custom system-level integration. But you can buy CSACs yourself—mounted on PCIe cards—for your own data center. (Every site I can find is of the "request a quote" variety where prices are concerned. They were apparently $1500 apiece in 2011, and have probably come down since then.)

But you don't need an atomic clock to get Spanner's guarantees. CSACs are more convenient in terms of setup and requirements—but GPS clock-sources will do just fine for Spanner's 6ms quantum. You can buy a commercial-off-the-shelf GPS NTP appliance (https://www.microsemi.com/products/timing-synchronization-sy...) and run signal lines from it to all your machines; or cobble a similar solution together yourself, in true HAM fashion, using a GPS antenna + a Linux box + a UHF Software-Defined Radio card (i.e. a TV-tuner card) + GPSd + NTPd. (Or or you could buy cheap USB GPS receivers and hook them up to each and every server in your DC—but you'd need a very thin ceiling for that to work.)

Amusingly, the practicality of that last approach also means that you could run Spanner just fine on a cluster of Android phones. :)

> But you don't need an atomic clock to get Spanner's guarantees.

I always hear this from CockroachDB folks and fans, but no details. What are the downsides?

As per my understanding, if a server/replica could not catch up wrt time (i.e. goes out of sync), it can be identified and usually be marked as a temporary-replica-failure -- which is the maximum extent that Hybrid Logical Clocks can help us with. Rest of the system has to deal with consequences: A new replica must take it's place to keep-up the fault-tolerance-level and start making-a-new-copy -- which is in the order-of the amount of data stored in the out-of-sync replica. I assume this puts lot different design requirements on the storage-engine; also, making-a-copy and cancel-copying operations would eat network traffic, thus throughput.

IIRC Google Spanner uses atomic clocks only on few severs per data-center because cross data-center latencies are much higher and erratic (over internet). So CockroadDB has much higher rate of temporary-failures due to out of time-sync and associated downsides. It would be helpful if CockroachDB guys could shed some light on this.

(Cockroach Labs CTO here)

> > But you don't need an atomic clock to get Spanner's guarantees.

This comment continues "...but GPS clock-sources will do just fine for Spanner's 6ms quantum". Providing Spanner's guarantees with reasonable performance requires specialized hardware, but there are more options for that specialized hardware than just atomic clocks.

Note that Spanner itself uses both atomic and GPS time sources according to Google's publications; when we talk about "atomic clocks" we're usually talking about the entire category of specialized time-keeping hardware instead of distinguishing atomic clocks from GPS clocks.

> I always hear this from CockroachDB folks and fans, but no details. What are the downsides?

As we describe in our blog post (https://www.cockroachlabs.com/blog/living-without-atomic-clo...), CockroachDB on commodity hardware provides a slightly weaker consistency model than Spanner (serializable instead of linearizable), and latency is sometimes higher as we need to account for the larger clock offsets in certain situations.

If you do have a high-quality time source available, we have an experimental option to use a Spanner-like linearizable mode.

The CockroachDB team did put up a blog post that talks about the tradeoffs.

The pertinent part was: "Spanner always waits on writes for a short interval, whereas CockroachDB sometimes waits on reads for a longer interval. How long is that interval? Well it depends on how clocks on CockroachDB nodes are being synchronized. Using NTP, it’s likely to be up to 250ms. Not great, but the kind of transaction that would restart for the full interval would have to read constantly updated values across many nodes. In practice, these kinds of use cases exist but are the exception."

https://www.cockroachlabs.com/blog/living-without-atomic-clo...

The clock thing is highly dependant on workload. If your load is append only, clocks within a few seconds would be fine. If you truly have competing writes for the same key within ms, then clocks need to be pretty good. Most big data stuff I see is closer to the former.

CSACs seem unnecessary. Within a few racks, PTP (IEEE 1588) can do a pretty decent job of getting things synchronized to tens of microseconds. Requires that your ToRs and NICs support it, but that's not a very onerous requirement, particularly if you're GoogleBookSoft.

This means you only need a few time receivers for a datacenter, along with careful monitoring and implementation of your time distribution, but that happens through your Ethernet switches.

More: http://www.ni.com/newsletter/50130/en/

But you need your datacenters to stay synced even if you lose GPS. You can use an atomic clock for this (Cs or Rb). But, for the rest of us, a good GPS-disciplined double-oven crystal oscillator (XO) can get you within the range needed for Spanner, IIRC spanner's time sync requirements. For example, this little one: https://www.microsemi.com/document-portal/doc_download/13341...

will do +- 7 microseconds / 24h holdover. ("holdover" == operating when it's lost GPS).

Sounds like spanner can't tolerate much drift: "The most serious problem would be if a local clock’s drift were greater than 200us/sec: that would break assumptions made by TrueTime."[1]

[1] http://www.bluetreble.com/2015/10/time-travel/

Reply on news.ycombinator.com