Hacker News Reader

Comment by pclmulqdq | Hacker News Reader

I worked on Colossus while I was at G. It is an incredible piece of technology, and the current versions are a lot better than GFS. It makes me a little sad that no papers get written about it. Scale-out ZFS and Ceph from the outside world may be the closest competitors, but they don't hold a candle to Colossus in terms of scalability and reliability.

View on HN

Google publishes a lot of papers in AI, networking, and kernel-type work (isolation, scheduling, etc.). This seems to attract a lot of talent and helps build relationships with academia.

For whatever reason though, the storage and analytics groups don't publish nearly as much as they could.

Bigtable folks wrote a follow-up paper to the original, but it was turned down by the conference's reviewers for not being novel enough or something like that, so the team just went ”ok, whatever..." and moved on. Years later, there were a couple of talks by Carter and Phaneendhar at the HBase conference that might have been hosted at Google NY.

Ahh yea, I was an SRE back when Colossus was still bootstrapping on GFS. Bigtables all the way down.

Ceph is still largely based on the original GFS last I looked at the architecture.

does it use custom hardware or soes it work om commodity PCs? If I recall correctly, spanner used GPS.or atomic clocks to have precise time on every machine, and that enables it to perform much better.

It uses commodity hardware. Also, Spanner doesn't really need the atomic clocks, they just help performance. CockroachDB, which is similar, runs fine on commodity hardware.

I seem to have misunderstood Spanner by and large. I thought their achilles heel was the atomic clock since it is this clock which makes consensus at scale possible and there by transactions. Cockroach and Yugabyte both work on similar principles but they use these hybrid logical clocks. The hybrid logical clock is a function of NTP and a monotonic local clock so I guess this works ok for Raft consensus.

https://www.yugabyte.com/wp-content/uploads/2021/05/Distribu...

From Jepsen - "yugabyte seems mostly resilient to clock skews' - this is a very safe but at the same time a vague statement. What does mostly mean ? Good for 99% of practical use cases ?

I think it means "we can't prove that it works, but we have tested it and we haven't found a case where it doesn't." There are conceivable cases where the hybrid logical clock won't work, but most of these are cases where other systems will fail first (eg TLS/SSL).

Edit: To address the first part of your comment, the atomic clocks narrow the bounds on how bad your NTP clock can be. Spanner essentially waits out the bounds on clock variance on each write transaction to get consistency. There still would be a measurable bound on your clock precision without the atomic clock, but it would be a lot larger.

Ceph is incredibly reliable. I run a ~ 250 PB Ceph install with the equivalent of 1.5 medium-paid full time engineers[0], and I sleep very well at night.

Failure after hardware failure, it just keeps running and running. We currently fail at a rack level, but no reason we can't do that at a building level with enough hardware.

0: We have more people on our team, but we also deal with other things that take a lot more of our time.

Reply on news.ycombinator.com