krenoten

> Generally any program that can handle a suspend and resume can handle being persisted across reboots.

This is not the case for very many of my processes. Are you suggesting DMTCP or something? That won't work with most of the stuff I'm running in terminals.

Badass! This would have saved me about a week of my life if it had come a few years ago!

Prometheus is for smaller problems. You have to manage your own sharding with prometheus, so it doesn't work for handling large volumes of data unless you are willing to do complex automation for a large number of disparate replication topologies. This works fine at Soundcloud, where it came from, because the teams there like keeping their monitoring systems isolated from each other because they don't trust each other.

For problems where you require a single source of truth for high volumes of data, you can't really justify that kind of effort for the data management when distributed databases have been created to solve this problem.

Prometheus is nice and featureful, but it makes me laugh when the proponents (usually [ex-]soundcloud folks) brag about how scalable it is.

Side note: they love to say that because it's pull based, that magically makes it more scalable. This is a crock of shit. Push vs Pull doesn't get you out of capacity planning, and from this perspective they are identical in terms of throughput requirements. Pull means you need a single source to know about a ton of endpoints. Push means you need a ton of sources to know about a single endpoint. Which one is simpler to operate? They say it's more scalable because when they failed to properly do capacity planning, it hid their dropped data more effectively and their dashboards kept running, oblivious to the degredation.

The price performance curve is bad at low traffic, but pretty good compared to most other things out there at the higher end of things.

I love OpenTSDB, but I think it's highly unusual for people to have "absulutely no issues at all". I ran OpenTSDB at the hundreds of thousands to low millions of ops per second range for several years. OpenTSDB has historically crapped itself when you have writes hit it for rows that it already ran its all-columns-for-an-hour-into-a-single-column compaction on. Now it just drops data on reads.

If you have a large team of engineers writing data into it, they will sometimes abuse the schema by overloading single metrics with many tags. Every permutation of tag*value creates a new row per hour of data. This creates extremely hot shards when engineers inevitably do something like store a client IP address into a tag. When this happens, you may have to write some web-UI scraping code to figure out which regions are getting the most traffic (assuming you have "extremely high traffic setup" numbers of region servers), or script up something like misra-gries on a tcpdump of its inbound metrics to see where the hot shit is so you can get somebody's deploy reverted. I know of some companies that have forked OpenTSDB and prefixed each row with a hash such that it spreads overloaded metrics around the cluster much more evenly. KairosDB solves this by not using a lexicographically sharded database (Cassandra).

When run properly, it runs great. But that takes some really painful learning experiences to learn how to do, generally.

It depends how much time you have to burn, and how much traffic you handle. If you have trivial scaling needs or want to spend 10x the human time managing that setup, then go ahead. You use something like opentsdb when you have hundreds of thousands or millions of ops per second and don't have the time or desire to deal with a database that you have to build your own partitioning + resharding + failover + recovery systems on top of.

B+ trees are not optimized for the kinds of write-heavy workloads that you typically stick in HBase (LSM-tree).

They have mostly moved away from it.

That's pretty hand-wavey. All databases can be made to lose all guarantees under failure conditions.

Redis is single-threaded, so everything it does on a per-node basis is implicitly atomic. You can also force the single thread to handle a block of commands from a single socket at once by using MULTI (otherwise commands can be interleaved with other commands from other sockets).

This is a pretty balanced article. Absent is Nomad, which claims the world but when you dig into the code pretty large advertised chunks are simply not there yet. Nomad seems like a much more straightforward implementation of the Borg paper, and one day may be interesting once they write the rest of it. A nice Kubernetes feature that is similar to what you can do with fleet is the “Daemon Set” which lets you run certain things on every node. Some cool Mesos features that are pretty new and haven’t been talked about much yet:

* persistent volumes: let frameworks have directories on agent machines returned to them after an availability event happens, which is nice for replicated stateful services

* maintenance primitives: schedule machines to go offline at certain times, and ask frameworks if it’s safe to take nodes offline. This will soon start being used for stateful services so that they can vote on when it’s safe to take out a replica, and to trigger proactive rereplication when maintenance is desired.

* oversubscription: if you have an agent that has given away all of its resources, but the agent detects that there is still some unutilized CPU, it can start “revokable tasks” to fill up the slack up until it starts interfering with existing workloads.

That link isn't working for me, unfortunately. I'm assuming that the HOL you're referring to stems from the assumption of the default DRF allocator of fast decisions, which can cause issues for mixed workloads if there's a lot of hoarding happening. This is something that could be addressed with an allocator module, but nobody has yet to my knowledge, leading me to think it's not a massive problem for many people. I think the people who care about this use case end up running multiple mesos clusters.

One note on priorities a la Borg (what I think people are actually using at Google today, but I don't know to what extent Omega was merged with it after the Omega paper or if Borg currently runs as an Omega scheduler or what), is that it sometimes causes annoyance for people at Google, as the optimal priority is not always assigned to certain workloads, leading to starvation or overpreemption of things that were not given a healthy number. I believe that Mesos' work on oversubscription took concerns around this issue into consideration.

It's really fascinating to compare the architecture of Nomad vs K8s, because both claim heavy descendence from Omega + Borg. If I squint, it seems like Nomad is more of a direct implementation based on those papers. K8s more explicitly modularizes more components, which I think is why a lot of companies are jumping behind it, as they can mold it in the ways they want.

Mesos is a two-level scheduler, where the elected master is more of a broker for offers from agents to external schedulers that decide to act on the offer or not. By default it's pessimistic, but this is pluggable and there's no reason you can't write an optimistic allocator module that will hand out offers to multiple schedulers.

Modern operating systems are better at handling many threads.

It was a terrible idea with old OS's and engineers never learned about their schedulers. Many of your favorite systems run at high frequencies on Varnish using one thread per session.

It's like many things that are declarative - you can only do what has been explicitly enabled in the declaration handling logic. This is good and bad. It's good because it restricts what is possible (protecting you slightly more from bad users) and can make learning and using the system easier. But you can't go beyond this. With a non-declarative scheduler you can implement far more interesting (for better or worse) management systems.

One area where we've been seeing some interesting work happening is in stateful services running on top of schedulers. With something like postgres, there is interesting logic that must happen for responsible management beyond just throwing up 3 of them. You need to manage replication carefully. When a node goes down somewhere, you need to react somewhere else in a controlled way. With Mesos, you can write a scheduler that encodes your runbook. With kubernetes, you need a lot more out-of-band stuff.

Kubernetes is very much still under development. Most of the work up to 1.0 has gone into refinement of the API, rather than stability. For instance, HA is only now being seriously worked on - and if your API server goes down and you need to start it on another host, if you don't have an external reconvergence mechanism you have to restart every kubelet so they can point to the new one.

Mesos is much more mature, but is more low-level, and schedulers like Aurora or Marathon do not give you the same service abstraction, but they have many features not currently present in Kubernetes. But, in return for being more low-level, you can run things that are simply impossible in a declarative system like Kubernetes. Personally I see Kubernetes-on-Mesos as being a really bright future choice, but like pretty much anything other than Marathon/Aurora on Mesos, it's pretty nascent.

TLA+ model checking and an implementation that follows the spec as closely as possible.

Raft has a dirty little secret: your cluster breaks when two nodes become partitioned from each other (but not the rest of the cluster) and one of them is the leader.

Script: nodes {1, 2, 3*} where 2 and 3 are partitioned, and 3 is the current leader.

Node 2 fires its leader election timer, broadcasts RequestVotes to 1 and 3, only 1 gets it, but now 3 is ignored for the term of its leadership and 2 is the only node capable of quorum. In a decent implementation 3 will be nack'd the next time it sends an AppendEntries to 1, and 1 will pass along the current term. 3 isn't hearing from the leader so broadcasts RequestVotes (either failing once due to failure to hear the current term from 1, or jumping straight into the next term, which actually increases the ratio of cluster livelock). Leadership bounces back and forth rapidly, making your cluster worthless.

The current hotness is spec paxos, which gets the positive trade-offs of both cheap paxos and fast paxos, if you're able to actually implement it in your DC (has networking assumptions).

Have any of the popular Haskell libraries for building distributed systems seen high-scale production use for more than 5 years?

No system can achieve 100% consistency and 100% availability in the presence of partitions. It's kind of wacky that Aphyr compared those replicated consensus tools and eventually consistent stores with HBase. HBase does not use a consensus protocol for replication, it uses HDFS. HBase is not eventually consistent. There is a single authoritative server for reads and writes of a lexicographic range of keys (a region) which writes immutable store files and a WAL to HDFS. Partial availability may be achievable for reads with a significant amount of effort and latency and limiting of total cluster size by allowing non-authoritative regionservers to read the HDFS WAL + Storefiles, but this really isn't realistic. I've personally been burned by the client retry thing though, and there's not really a better solution when you consider the types of workloads HBase is actually used for and it's incredibly variable latency (it aims only for consistency and very high AVERAGE throughput, at the cost of extremely high HIGHEST latency). One solution here could be the use of a configurable filesystem queue for clients. This is how you build resilient high-throughput pipelines. HBase is used in some places for OLTP, but only when the readload is very very low. So the effort to make reads more highly available would be in vain.

Yes, please do look at them, because Google, Facebook, Twitter, Yahoo, and eBay use MySQL pretty heavily for many of their core storage needs (look at the contributor list for WebScaleSQL). Spotify uses Postgres for the same. All of them also use other things as well, but only for cases where they are willing to make performance or reliability trade-offs like for colder or analytical data. Of course they also use things like Sherpa, Cassandra, HBase, etc... but they lose consistent low latency or consistency or availability when they do so.

The point is, if you are going to bet your business on a technology, it helps if it has been tested with production workloads in many different conditions and scales. You want to know about as many shortcomings as you can. For many of the use cases that people use things like Cassandra for, they will be tolerant of 30ms++ reads and potential read inversions. Redis is used pretty heavily, but it is relatively simple code and you can trace through the entire writepath pretty easily and get a sense of its limitations (being single threaded is a blessing and a curse, you really need to be careful about bad tenants because a single slow query will cause an availability event for everyone). HBase is used in a few places, but usually only for cold data after they expect it to be read only occasionally and they don't want to use up space on their MySQL pci flash devices for it anymore. There are a bunch more, but they all have some latency, consistency, or availability downsides compared to a traditional sharded B+ tree backed transactional store.

When building reliable systems, you expose them to simulations of as many failures as is economical. It's not clear how many failures the pilot dealt with in those 5000 hours. Proper simulation teases out faults in handling failures with a high success rate in many situations. Maybe it will become clear that the training regime could be improved for this airline.

Like everything else, it's a tradeoff. Knowledge about certain market topologies actually can constitute a tremendous advantage, although NDA's are employed far more than is actually necessary.

The two things that matter are performance and reliability. Are you able to go into how lattice improves over either Mesos or k8s in these regards?

It's a plus until it isn't :P

Strong type systems absolutely save lives.

Aye. Timeouts cannot save you from the recursive ack problem.

I believe the parent is thinking of the fact that FLP applies only in an "asynchronous" system in which messages may take infinite time to arrive, and without regard to sending order.

code stubs and tests:

git://g.csail.mit.edu/6.824-golabs-2015

paxos assignment: http://nil.csail.mit.edu/6.824/2015/labs/lab-3.html