Torus: A distributed storage system by CoreOS

While I strongly agree with bcantrill, I have to disagree with your HPC origins model in the most strenuous fashion.

The HPC world can be thought of as an alternative universe, one where dinosaurs came to some sort of lazy sentience instead us mammals. HPC RAS (reliability, availability, serviceability) requirements and demands are profoundly different and profoundly lower from what enterprise or even SMB (small & medium sized businesses) would find acceptable. There are or were interesting ideas in GPFS but that was long ago, far away, and sterile besides - not that it stops IBM squeezing revenue from locked-in customers.

Some people in HPC do realize their insular state, but mostly there are hopes for things to be done better that were done better earlier (e.g. "declustered RAID") in the "real" world. Their ongoing hype is for yet-another-meta-filesystem built with feet of straw upon a pile of manure.

The likes of Google infrastructure, are built instead on academic storage R&D, which has been a great source of ideas and interesting test implementations - log structured file systems, distributed hash tables, etc. There is some small overlap, for example in the work of Garth Gibson, but it's not a joint evolution at all.

Yes, that's absolutely been my experience -- and even then, when it comes to the data path, you will likely find new failure modes in "tried and true" as you push it harder and longer and with the bar being set at absolute perfection. I have learned this painful lesson twice: first, with Fishworks at Sun when we turned ZFS into a storage appliance -- and we learned the painful difference between something that seems to work all of the time and something that actually works all of the time. (2009 was a really tough year.[1]) And ZFS was fundamentally sound (and certainly had been running for years in production) before we pushed it into the broad enterprise storage substrate: the bugs that we found weren't ones of durability, but rather of deeply pathological performance. (As I was fond of saying at the time, we never lost anyone's data -- but we has some data take some very, very long vacations.) I shudder to think about building a data path on much less proven components than ZFS circa 2008, let alone building a data path seemingly in total ignorance of the mechanics -- let alone challenges -- of writing to persistent storage.

The second time I learned the painful lessons of storage was with Manta.[2] Here again, we built on ZFS and reliable, proven "tried and true" technologies like PostgreSQL and Zookeeper. And here again, we learned about really nasty, surprising failure modes at the margins.[3] These failure modes haven't led to data loss -- but when someone's data is unavailable, that is of little solace. In this regard, the data path -- the world of persistent state -- is a different world in terms of expectations for quality. That most of our domain thinks in terms of stateless apps is probably a good thing: state is a very hard thing to get right, and, in your words, tried and true absolutely beats novel and new. All of this is what makes Torus's ignorance of what comes before it so exasperating; one gets the sense that if they understood how thorny this problem actually is, they would be trying much harder to use the proven open source components out there rather than attempt to sloppily (if cheerfully) reinvent them.

[1] http://dtrace.org/blogs/bmc/2010/03/10/turning-the-corner/

[2] https://github.com/joyent/manta

[3] https://www.joyent.com/blog/manta-postmortem-7-27-2015

It seems you're disagreeing with yourself.

Even when commercially misguided, Sun always had terrific engineering talent -- and my farewell to the company captures some of that.[1] In terms of why did the company fail, the short answer is probably that SPARC was disrupted by x86, and by the time the company figured that out, it was too late to recover.[2]

[1] http://dtrace.org/blogs/bmc/2010/07/25/good-bye-sun/

[2] Longer answer: https://news.ycombinator.com/item?id=2287033

How is them developing their own technology not competing on merit? Did they steal the technology? Did they claim that anyone ought to use this in production? Why all the negativity? Yeah never mind let's just discourage everyone from trying to build new technology. Nobody is forcing you to use this. If it's not for you, move on. Going out on a rant about what you think their intentions are is ridiculous.

Really you come off as defensive. I don't understand how CoreOS has disrespected anyone by coming up with their own approach to the problem.

They're not writing blog posts or comments on HN, they're writing code.

Here is an example script that will panic anyone using NBD to server a block device (replace qemu-nbd with the nbd export provided by torus)

qemu-img create -f qcow2 f.img 1G mkfs.ext4 f.img modprobe nbd || true qemu-nbd -c /dev/nbd0 f.img mount /dev/nbd0 k killall -KILL qemu-nbd sleep 1 ls k

But: that's not a kernel panic, just a BUG() stack trace from kernel which doesn't halt the system.

Fleet died once Kubernetes became a thing, even though they serve entirely different use cases.

Persistent storage isn't a hard problem. Distributed, well performing, scalable and consistent storage is a hard problem.

Persistent storage on today's filesystems and complex hardware is a hard problem. All kinds of failures can happen during any write. Some are obvious with some silently corrupting data. There's been decades of work on approaches to dealing with this with a variety of tradeoffs. Picking the right one for a widely-deployed, portable, distributed app is tricky by itself.

They're aiming to do a lot more than that. ;)

> Also, the fact that it's a hard problem is what makes it a compelling product offering. You can't make much money solving easy problems.

If they can solve the hard problem, yes. Which I believe is unlikely.

It's described as a prototype, so I'm pretty sure there's no deployment at scale at all.

Virtualized, network block devices have all the same problems I described -- even worse, because the abstraction coneys even less about what an application is trying to do.

what is the existing alternative for storage in distributed container environments?

Competition is good, even in open-source land. By the way, I'm still hoping for an open-source competitor for elasticsearch.

See also: Piston Cloud.

Yeah, I've been smelling NIH for quite a while.

And it generally seems that, when they say "composable modular tools", one should read "includes dependencies on the rest of our stack". This isn't intended as harshness. It is just that every time a tool from COS seems interesting enough to look in to, it turns out that I wouldn't be investing in a tool, it would be investing in an ecosystem that duplicates a ton of what I'm already managing.

Rkt had good reasoning behind it, and I think it's reasonable to assume that a lot of the recent changes to Docker came because they forced Docker's hand.

For Etcd I'm of two minds. I like its simplicity. On the other hand, I have had far more problems with Etcd than I've ever had with e.g. Consul (though I dislike Consuls "and the kitchen sink" approach), up to and including data loss and more than once having to re-initialise clusters. I'm a couple of years of no further problems at least a way from trusting Etcd with my data.

For this? They have to have very compelling stories for why they couldn't e.g. modify Ceph or Gluster to achieve whatever it is that they want. And it's no clear what they want. Gluster for example is built as a stack of pluggable translators that is relatively easy to extend.

Even if they have a credible and compelling story for why we need something else, it will take years of production use by others before I'd trust any data to a new distributed file system.

Gluster and Ceph have those years. I've had a Gluster volume running without maintenance or data loss for more than five years. Even then it took several years before I truly started to trust it.

> Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems.

Being built atop something else doesn't mean it inherits all the stability of the underlying system; it's an upper bound not a lower bound.

It sounds like etcd is in the I/O path. Is that correct?

I've managed to lose data with Etcd, and have regularly had issues with membership issues requiring maintenance. Meanwhile I've had Glusterfs volumes remain available for 5-6 years without maintenance at all.

To me at least, having it "backed by etcd" is a big red flag, not a feature.

"haven't figured out consistency yet" is the issue. Seemingly very small decisions have huge impacts that you don't expect, and the state of the art is far enough along that you don't get better than the existing solutions except by exploiting newly-identified workload characteristics (or acceptable ways of losing data, loosening consistency, etc based on the workload) that you've planned out to make use of ahead of time.

One example: Both Gluter and Ceph have erasure-coded storage. Gluster's looks just like the replicated storage, only it involves more nodes and less overhead. Ceph's is severely limited in comparison: it's append-only, it doesn't allow use of Ceph's omap kv store, "object class" embedded code, etc. The reason is because distributed EC is subject to the same kind of problem as the RAID5 write hole: if a Gluster client submits an overwrite to 3 of a 4+2 replica group and then crashes, the overwritten data is unrecoverable and the newly-written data never made it.

Torus won't hit that particular issue because it is log-structured to begin with, which has all kinds of advantages. But garbage collection is really hard! Much harder than seems remotely reasonable! Getting good coalescing and read performance is really hard! Much harder than seems remotely reasonable! There's one big existing storage log-structured distributed storage system which has discussed this publicly: Microsoft Azure. They have a few papers out which hint at the contortions they went through to make block devices work performantly — and Azure writes first to 3-replica and then destages to the log! They still had performance issues!

https://github.com/coreos/torus/blob/master/Documentation/re... points to a bunch of HDFS research and replacements; HDFS is designed for the opposite (large files with high bandwidth and nobody-cares latency) of what I presume Torus is targeting (high IO efficiency, low latency). Mostly the same for the Google papers they cite. There's no mention of Azure's storage system papers, nor of Ceph, nor anything about the not-paper-publishing-but-blogging stuff from Gluster or sheepdog; nor from academic research into VM storage systems (there's tons about this!).

Can they fix a bunch of this? Sure. But the desires they list in eg https://github.com/coreos/torus/blob/master/Documentation/ar... go towards making things worse, not better. They aren't talking about how etcd can be in the allocator path but not the persistence path[1] and how every mount should run a repair on the data to deal with out-of-date headers. They talk about adding in filesystems, but not any way of supporting read-after-write (which is impossible with the primitives they describe so far, and really hard in a log-structured system without synchronous communication of some kind). They discuss network partitions between the storage nodes, and between the client and etcd; they don't discuss clients keeping access to the disks but losing it to etcd.

[1] Using etcd for allocation would be a reasonable choice, but putting it in the persistence path is now. Right now a database in your container would require two separate write streams to do an fsync:

1) data write. It doesn't say in docs and I didn't look to see if replication is client or server-driven, but assuming sanity the network traffic is client->server1->server2->server1->client, with a write-to-disk happening before the server2->server1 step.

2) etcd write. Client->etcd master->etcd slaves (2+)->etcd master->client, with a disk write to each etcd process' disk before the reply to etcd master.

This is a busy-neighbor, long-tail latency disaster waiting to happen.

Coming from the company that raised hell about Docker's lack of a spec, this is an interesting approach.

Will each disk in the system be weighted so that different size disks can be used in the system over time? OpenStack Swift offers this and is useful to not lock-in disk size at design time.

Docker supports native RBD since version 1.8. Which should be both faster and more reliable than using the S3 gateway (though that has other benefits and becoming pretty good too).

How do you say that with conviction? I looked at the code and it has no bearings of being able to build a fault tolerant elastic block storage system.

That's interesting. When I was looking at it, it seemed overwhelming. Do you have some pointers for the basics?

Many thanks, that looks doable. I'll give it a try.

> but I would have hoped for better from HN

Don't worry about it. Some guys just mistake their experience with traditional storage systems as meaningful here.

6 months is overnight in terms of a filesystem, there are literally millions of dollars of work that was done with glusterfs and probably a magnitude less man hours. I understand coreos wants.to innovate but what does torus bring to the table besides negatives and NIH?

Disclaimer: I work on etcd at CoreOS.

CockroachDB is built on top of etcd/raft package, not etcd itself.

I think that reflects the complexity of a storage product more than CoreOS itself. I use CoreOS. I like it. I love the update mechanism, and the tight focus on running as much as possible in containers.

But I also have spent enough time using it to run across certain warts, and Etcd clusters refusing to start without manual intervention etc. have been a frequent enough problem that when they go after something incredibly hard such as distributed storage, and it's relying on Etcd, I see that as a scary combination.

Personal attacks are not allowed on HN, nor single-purpose accounts. Novelty accounts for personal attacks are right out, so we've banned this one.