Hacker News Reader

Comment by jsnell | Hacker News Reader

Obviously not, since you could not deliver it. It seems that you maybe don't realize what CFS is in this context, and are thinking of something else that you could just "set up"?

What jeffbee is talking about is Google's proprietary Colossus File System, and all its transitive dependencies.

View on HN

I meant it sarcastically, but for "serious money" you can have any software system you can dream of. You have to dream of it, though - that's one of the hard parts.

It looks like every other clustered file system. What's special about Google's Colossus?

There are some semantic differences compared to POSIX filesystems. A couple big ones:

  - You can only append to an object, and each object can only have one writer at the time. This is useful for distributed systems - you could have one process adding records to the end of a log, and readers pulling new records from the end.
  - It's also possible to "finalize" an object, meaning that it can't be appended to any more.

(I work on Rapid storage.)

Why would you wish for a system with constraints like that, which other systems don't have?

Other systems don't offer the performance that Colossus offers, is why. POSIX has all kinds of silly features that aren't really necessary for every use case. Throwing away things like atomic writes by multiple writers allows the whole system to just go faster.

It sounds like you have to find a design that meets your performance target and usage patterns - just like anything else. It also sounds like Google's CFS is a grass is greener situation - you heard Google had something that solved the problem you have, so you want it. But the reason it sounds good, compared to the other designs, is that you haven't had to actually use it and run into its quirks yet.

Google's internal systems have been written against the Colossus semantics for many, many years and thus benefit from it's upsides (performance, cost efficiency, reliability, strong isolation for a multi tenant system, ability to scale byte and IO usage fairly independently, tremendously good abstraction against and automation of underlying physical maintenance, etc) while not really having too much of an issue with any of the conscious trade-offs (like no random writes).

On the other hand, if you've been building your applications against expectations of different semantics (like POSIX), retrofitting this into your existing application is really hard, and potentially awkward. This is (IMO) why there hasn't been an overtly Colossus based Google Cloud offering previously. (Though it's well publicized that both Persistent Disk and GCS use Colossus in their implementation.)

One of the reasons why it would be extremely hard to just set up or build CFS elsewhere or on a different abstraction level is that while it may look quite achievable to implement the high level architecture, there is vast complexity in the practical implementation side. The tremendous user isolation it affords for an MT system, the resilience it has against various types of failures and high throughput planned maintenance, the specialization it and its dependencies have to use specific hardware optimally.

(I work on Google storage part time, I am not a Colossus developer.)

Reply on news.ycombinator.com