Three Clocks Are Better Than One
tigerbeetle.com
tigerbeetle.com
"When you have a clock you always know what time it is. When you have two you are never quite certain." -Mark Twain
An ancient adage warns, "Never go to sea with two chronometers; take one or three." — Fred BrooksBut if you have a atomic clock don't trust those 5 ntp's at all. Maybe buy a 2nd clock or even better 5 of them ;)
> Priority 1 – the user can assign a specific static-designed priority to each clock, preemptively defining a priority among them. Smaller numeric values indicate higher priority.
> Class – each clock is a member of a given class, each class getting its own priority. Accuracy – precision between clock and UTC, in nanoseconds (ns)
> Variance – variability of the clock
> Priority 2 – final-defined priority, defining backup order in case the other criteria were not sufficient. Smaller numeric values indicate higher priority.
> Unique identifier – MAC address-based selection is used as a tiebreaker when all other properties are equal.
Only useful if you trust the data: Class, Accuracy, and Variance (which seems to be a self-measured estimate).
Currently it's just set as the time server via DHCP for my computers, but I'm making a nixie clock which will use an ESP8266 running SNTP.
I've got some IN-14's, they can't handle more than 2.5mA. Operating voltage is ~150V, so that's at most 0.38W per tube.
So no, they won't get hot.
This is unlike vacuum tubes, used for amplifiers and similar, which are hot-cathode and needs a heating filament. They can get quite hot indeed.
Its timekeeping will suck though if it won't receive the messages periodically, but in practice it's quite fine. Except once when the ethernet board somehow broke down; fixed it by replacing it.
Here's a short writeup I did for hackaday: https://www.modeemi.fi/~flux/projects/TopfieldClock/
Pi with GPS
$ chronyc sourcestats
210 Number of sources = 7
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
==============================================================================
PPS1 8 6 112 -0.018 0.020 -105ns 336ns
GPS1 7 6 95 +55.685 111.018 +126ms 1058us
GPS2 8 6 112 -0.018 0.021 -89ns 341ns
2a01:79a:408::250 18 10 310m -0.057 0.080 -1364us 522us
time.cloudflare.com 19 13 344m -0.068 0.038 -1274us 258us
ntp.coreless.net 6 3 85m -0.343 0.706 -2355us 354us
anduin.net 21 8 379m -0.020 0.051 -835us 388us
$ chronyc sourcestats
210 Number of sources = 5
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
==============================================================================
timepi.localdomain 13 7 64m +0.001 0.031 +328ns 28us
79.161.39.253.static.lys> 8 6 120m -0.002 0.184 -2122us 189us
125.51-174-177.customer.> 22 13 464m +0.287 0.117 -839us 1201us
time.cloudflare.com 6 5 85m +0.009 0.472 -1615us 277us
c1EE4653E.static.as2116.> 10 7 155m -0.146 0.603 -3315us 1256us
The GMW-B5000 series is a reasonable successor, but it's thicker, heavier, and 4X the price: https://www.google.com/search?q=casio+gmw-b5000&tbm=isch
One of the three was left as a todo :D
https://github.com/trottier/original-bitcoin/blob/92ee8d9a99...
My Garmin watch syncs to GPS satellites every time I start an activity. My cell phone (which many of my friends and family use as a replacement for a watch) syncs to the cell towers. Even appliances now are operated by microcontrollers with crystal oscillators having a frequency tolerance on the order of 20ppm: their clocks have little drift. Mine are not 'smart appliances', but those are becoming pervasive, and they, too, sync via NTP.
Ultimately, though, all those NTP and GPS clocks have their master at the NIST in Boulder, Colorado - the same clock that WWV, broadcasting from Fort Collins, 50 miles north, synchronizes against.
But on second thought, it can also interpreted as such: in a large synchronous system, it's often important to have a standard reference clock and to lock all the other clocks onto main clock, otherwise the system can be inconsistent due to timing problems, e.g. distributing a frequency reference in a lab, distributing the clock signal on a circuit board, or distributing standard wall clock time via NTP (the Stadium 1/2/3 hierarchy). So the quote makes perfect sense again.
MOE: How long has that been in the soup, froghead?
CURLY: About uh--- [looks at the three watches on his wrists]
MOE: Hey! What’s the idea of the three watches?
CURLY: That’s the way I tell the time.
MOE: How do you tell the time?
CURLY: This one runs ten minutes slow every two hours. This runs twenty minutes fast every four hours. The one in the middle is broken and stopped at two o’clock.
MOE: Well, how do you tell the time?
CURLY: I take the ten minutes on this one and subtract it by the twenty minutes on that one. Then I divide it by the two in the middle.
MOE: Well, what time is it now?
[Curly grabs a clock from the inside of his jacket pocket]
CURLY: Uh, ten minutes to four.
MOE: Hmm so it is!
[Moe breaks the watch on Curly’s head.]
I love the pursuit of performance, but who would actually need 1 million journal entries a second (from their home page)? Is that the bottleneck somewhere?
Apologies for the stupid questions, I just... have no idea where this thing (which seems pretty technically neat) is actually intended to be used.
It's a database with purpose-built primitives for recording financial transactions between accounts, particularly two-phase commit transactions between different payment systems run by different operators.
Over the past decade or so, some of our team saw that these primitives kept getting re-implemented over and over again, often with duct tape ad-hoc solutions that are not always safe, or scalable.
At the same time, the payments world is rapidly changing. Things are moving to high volume, low value digital transactions, which means that operational cost becomes important.
We extracted the design for TigerBeetle out of a real-world payment switch, so that TigerBeetle can be transplanted into these kinds of systems very easily, and in a way that's very hard to get wrong because the interface maps one-to-one to the domain.
We also solve a storage fault model that's relatively new in the research literature, something that most traditional databases were never designed for, and we do this because this makes for a safer system of record, not to mention a better operating experience.
If anything, we place the pursuit of safety much higher than the pursuit of performance. We work much harder at the safety, the performance came pretty easily with the design right from the beginning. So the safety of TigerBeetle is what we're most excited about. The performance is a nice-to-have to go with it.
These are hopefully "the things that don't change" when it comes to databases. Nobody wanted a less convenient, or more dangerous, or slower database.
Or add a temperature Sensor and start modeling a drift compensation: https://blog.dan.drown.org/rtc-comparison/
Or why reinvent a homebrew version of what looks like PTP IEEE 1588:
https://blog.meinbergglobal.com/2013/09/14/ieee-1588-accurat...
It's complementary, a clock fault failure detector. We're not trying to synchronize clocks, merely to detect when too many clocks in the cluster no longer agree.
TigerBeetle needs to detect when the network is partitioned in a way such that the clock sync service (PTP or NTP etc.) is not functioning while the TigerBeetle cluster still has a functioning majority on the other side of the partition.
No matter the clock sync service you use, you still need some way to verify that the clock sync is in fact happening, which is what this is.
Hope that clarifies it for you!
We simply need a mechanism to know when PTP or NTP is broken so we can shut down as an extra safety mechanism, that’s all. Detecting when the clock sync service is broken (specifically, an unaligned partition so that the hierarchical clock sync doesn’t work, but the TigerBeetle cluster is still up and running) is in no way required by TigerBeetle for strict serializability, it’s pure defense-in-depth for the financial domain to avoid running into bad financial timestamps.
We tried to make clear in the talk itself that leader leases are “dangerous, something we would never recommend”. And on the home page, right up front (because it’s important to us) we have: “No stale reads”.
We also want a fault-tolerant clock with upper and lower bounds for timing out financial two-phase commit payments that need to be rolled back if another bank’s payments system fails. We can’t lock people’s liquidity for years on end because of a clock fault that we could have detected otherwise.
You can imagine TigerBeetle’s state machine (not the consensus protocol itself) as processing two-phase payments that look alot like a credit card transaction that has a two-phase auth/accept flow, so you want the leader to timestamp roughly close enough to true time, so that the financial transaction either goes through or ultimately gets rolled back after roughly N seconds.
Hope that clarifies the post (and gets you excited about TigerBeetle’s safety)!
P.S. We’re launching a $20K consensus challenge in early September. All the early invite details are over here [1]. Hope to see you at the live event, where we’ll have back-to-back interviews with Brian Oki and James Cowling, who authored and revised the pioneering Viewstamped Replication consensus protocol respectively.
This talk by Jon Moore is another great talk on time, causality and synchronization, especially where cluster membership is large and dynamic: https://youtu.be/YqNGbvFHoKM