Hacker News Reader

jolynchon Monitoring Cassandra at Scale

A deep dive into how data is stored on Cassandra rings and how to use this knowledge to give your operators notice before cluster issues become site issues.

Any feedback is super welcome. If folks like this we may consider working on upstreaming the approach into nodetool.

jolynchon Scaling Down Gracefully with AWS Lambda and HAProxy

You can down servers or backends over the stats socket, but you can't add them (adding servers requires a reload).

The DNS resolution work that is going into 1.6 releases may lead to a workable approach where you over allocate servers in a backend and then update them to have the right host/port. Last I checked, however, you still couldn't set the port dynamically.

jolynchon 50 Shades of System Calls

Sysdig makes me giddy like dtrace used to.

I also sometimes feel a bit ... challenged ... by translating the questions I have (e.g. why did this arbitrary program start using a lot of memory and then OOM) into actual sysdig chisel invocations, but I'm learning slowly but surely. This command line spectrogram looks like a really nice addition to the existing toolset!

jolynchon Linux TCP will have lockless listener processing 3.5M SYNs per sec

I think the latest number is looking more like 6M Mpps SYN (http://marc.info/?l=linux-netdev&m=144431864418007&w=2)!

This is also awesome because it fixes the issue with SO_REUSEPORT where the 3WHS gets screwed up by the per listener queues. I think that we still need some extra help from another patch (http://marc.info/?l=linux-netdev&m=144331389206170&w=2) to allow SO_REUSEPORT applications to drain connections, but the original patch plus the SO_REUSEPORT_LISTEN_OFF patch, or whatever Eric and Tolga end up doing, should be enough to allow applications to finally do graceful shutdowns of listen sockets. This is very relevant to my interests because it means that HAProxy will be able to do a natively supported hitless reload on Linux for the first time ever.

This stuff is brilliant.

jolynchon HAproxy in the era of Microservices

I mean, I wouldn't encourage anyone to use HAProxy for microservice load balancing unless they're running it on every node, at which point clients are always talking to localhost and everything is egress.

That being said, you can turn ingress into egress with an ifb. A coworker of mine created a proof of concept for using our strategy with an external facing load balancer but I don't think he ever tried it in production so I'm not sure how well it works.

To be fair I'm scared of both the iptables and tc solution on external LBs because you might never know that you're refusing connections accidentally. Linux 4.4 is coming with some patches that should help with this and afaik BSDs have done it right since the start.

jolynchon HAproxy in the era of Microservices

Has anyone actually done this at scale and seen how it works? We've talked about it a few times because Consul looks really cool but it doesn't appear to have the full capabilities of SmartStack yet, in particular Synapse (https://github.com/airbnb/synapse) for intelligently doing stats socket updates, intelligent reloads, or controlling how HAProxy reloads over time.

Also consul only has two levels of services right, "local" and "remote"? I'm not sure that is sufficiently powerful to describe SOA's which need to be able to differentiate services on the machine, rack, colo, datacenter, region locality? I suppose you could do what we do in SmartStack and just register the same service instance at a few different places since I think that consul can register arbitrary services.

Maybe all we need is a consul watcher in Synapse, and a consul-template for the Synapse configs, and we can get the best of all worlds. With Linux 4.4 SYN handling in Linux might even be fixed by Eric Dumazet being super hardcore (e.g. https://lwn.net/Articles/659199/) and we won't even have to worry about reloading HAProxy.

jolynchon HAproxy in the era of Microservices

The most significant issue with HAProxy and large SOAs is the lack of really good dynamic re-configuration. You don't want to have to manually change your configuration every time someone wants to add or change load balancing to a service, and if you do it automatically you run into HAProxy reloads being a reliability problem as they drop traffic on the floor due to Linux being goofy in how it implements SYN handling.

It looks like there is a lot of good advice on this thread for how to mitigate the configuration problem (bamboo, consul template, etc ...) but then you run into reloads, and even if you use zero downtime restarts as I described in my HAProxy deep dive (http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...), you still end up with periodic ~50ms blips in your timings.

We've found that Synapse (https://github.com/airbnb/synapse) is best in class at managing HAProxy for us in a fairly large SOA. It is really intelligent about doing as much as possible dynamically over the HAProxy stats socket as opposed to reloading for every configuration change, and we can finely tune how often we are reloading HAProxy. The best part is that it is fully pluggable. Use DNS, marathon, zookeeper, etcd, etc...: Synapse can manifest that service registration information into HAProxy configurations.

jolynchon HAproxy in the era of Microservices

Check out the deep dive I wrote into how you can help prevent HAProxy reloads from dropping traffic without a large 99% hit over at http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...

Looks like bamboo encourages folks to use this strategy via a custom reload command: https://github.com/QubitProducts/bamboo/issues/152

jolynchon Microservice Discovery with SkyDNS

I'm excited for all the buzz going on around service discovery, it's often one of the most crucial design decisions for a SOA architecture. I'm really bummed, however, that so many of them rely on clients doing DNS lookups. I vastly prefer a system like airbnb/Yelp SmartStack, consul template, or really any system that reflects discovery state in real load balancers such as HAProxy, nginx, or even those new fangled service load balancers like vulcand.

Honestly just don't do DNS. Why?

1. Performance

Service discovery should not happen during the life of a HTTP request. The number of times that I've had to debug performance problem that ultimately ended up being DNS requests taking like 100ms (which for most internal applications is too damn long) is staggering to me. Yes you can mitigate this, but it is not easy because even local DNS has to miss the cache sometimes.

2. DNS propagation sucks.

This article talks about one side of the DNS propagation problem ("it's down" -> removed from DNS), but they seem to completely ignore the other side of the DNS propagation problem (removed from dns -> clients stop talking to it). At any kind of reasonable engineering scale one of your libraries (watch out for ones that cache DNS "for performance"), programming languages (java), or infra tools (nginx/haproxy) are going to not re-resolve DNS like you expect them to.

3. More Performance

Client side load balancing is legitimately hard, and if you're just doing round robin over DNS records you are likely screwing your high percentiles. HAProxy/nginx/vulcand/etc all support better load balancing strategies like least connection load balancing that really make a difference here. This can be the difference between a 500ms 99% and a 50ms 99%.

jolynchon Show HN: Baker Street – A simple client-side load balancer for microservices

To be honest my favorite part of SmartStack is that you are not tied to a single discovery backend or mechanism. Both Synapse and Nerve support custom backends using whatever system you want (zookeeper, etcd, DNS, etc). At the end of the day both just expose basic configuration files and we exploit that at Yelp to do pretty cool stuff like allowing multiple systems to inform nerve/synapse about services (e.g. marathon or puppet) and allowing us to control service latency using a DSL that compiles down to those configuration files.

Just to clear something up, we have not found it necessary to run zookeeper at a cross datacenter level to get multidatacenter support. We're still working on writing up the details but the general gist is run zk in all datacenters and then cross register from a single nerve instance to multiple datacenters. That's why we had to remove fast fail from nerve, because by its nature cross datacenter communication is flakey. This approach has some tradeoffs however, as all approaches do.

All that being said, this is an interesting system and I look forward to more mindshare in the area of service discovery!

jolynchon Elasticsearch: The Definitive Guide

I wish they did this and had a better search interface. Their current "all in one bar + suggest" functionality works somewhat poorly for me.

Also I feel like they could reduce the needs for super great API docs if they had something like a query optimizer so that I don't have to care about the exact way that ES translates my queries into Lucene queries. My favorite bit is how if you permute your filters and queries in the wrong order you end up completely killing your performance (filter then query or query then filter).

jolynchon Elasticsearch: The Definitive Guide

Elasticsearch is built on top of Apache Lucene and is very similar to technologies like Solr but with a nicer interface (really more like SolrCloud these days).

It is different in that it provides higher level APIs and distributed functionality on top of Lucene. For example, one Elasticsearch index might contain multiple Lucene indices spread across multiple machines, but your application treats it as a single "index".

jolynchon Zero Downtime with HAProxy

No, nginx should not have this problem as it uses fd passing to gracefully handoff connections.

HAProxy only has this issue on Linux because Linux's SO_REUSEPORT implementation unfortunately introduces a race condition between accept and close. While I haven't personally tested it, HAProxy on one of the BSDs should not have this small window of downtime.

jolynchon Zero Downtime with HAProxy

I've been a little confused by the claims of how complex the SYN delay solution is. Is it actually all that more complex?

The original blog post wraps a restart command in two iptables invocations (and relies on a hacky sleep interval which may or may not work sometimes). The SYN delay method wraps a restart command in two tc invocations. The concepts are more or less identical in complexity as one is telling the kernel "drop SYNs now please" and the other is saying "delay SYNs for a bit please".

All the complexity in the qdisc solution is in the one time setup of the queuing disciplines. I think the largest drawback of the delaying SYN solution is not complexity but that getting it to work with external load balancers is more tricky than getting it to work with internal load balancers. Honestly, you're right that if an org doesn't have to restart HAProxy a ton, then it doesn't make a lot of sense to invest in solving this problem; although if it were me I'd just make sure I was on the latest Linux kernel so that the period during which HAProxy can cause RSTs is as small as possible and not bother with either the iptables or tc solutions.

jolynchon Zero Downtime with HAProxy

As other posters have mentioned the solution in this link is actually suboptimal. I showed how the iptables solution leads to 1-3s latency for connections that establish during the restart but you can avoid it by getting a little more hardcore.

See http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h... or http://inside.unbounce.com/product-dev/haproxy-reloads/ for solutions that are even better than this one.

jolynchon True Zero-Downtime HAProxy Reloads

Patching HAProxy to reload config is really hard. There have been ideas, patches, and discussions on the HAProxy mailing list for a few years now trying to get zero downtime reloads natively supported in HAProxy, but the reality is that it just is not as easy as it might seem.

For more details check out the mailing list: http://marc.info/?l=haproxy

jolynchon True Zero-Downtime HAProxy Reloads

Heh, the number of times DNS has bitten me in production... My main objections to DNS are:

1) You have no control over clients, there is absolutely no rule that says clients have to respect TTLs (I'm looking at you Java)

2) We are talking about hundreds of HAProxy load balancers, thousands of clients and hundreds of backend services which are moving around all the time. I just honestly didn't want to deal with DNS propagation limiting my flexibility.

3) At least at Yelp, we don't really have particularly nice control apis for DNS. This is sort of specific to Yelp, but it was a factor.

jolynchon True Zero-Downtime HAProxy Reloads

Totally agree, I mention this as the largest drawback of this approach in the post.

jolynchon True Zero-Downtime HAProxy Reloads

Josh gets all the credit for finding the plug qdisc. I was originally planning on using netem and having a fixed delay of 50ms or something, but Josh decided that wasn't elegant enough and went looking for a better solution.

As for the patch, I can certainly ask John but I do know that the proof of concept was written during a hackathon so I imagine there could either be fundamental flaws or just need a lot of work to get ready to merge. From reading the HAProxy mailing list I think they have been working on this for a while, but the issue seems to be merge risk (it's a fairly large architecture change). I'll mention it to John though.

I'm glad you think it's elegant :-) I tried hard to find a solution that was minimally invasive (no code changes, no significant infrastructure, etc ...).

jolynchon True Zero-Downtime HAProxy Reloads

I am not familiar with IPVS, I'll look into it.

From your description though I imagine it might have the same drawbacks that exist with the multiple HAProxy + iptables swap solution. With a solution where we run multiple HAProxies and move traffic between them I worry about port exhaustion, long running connections, and the maintenance costs of the infrastructure. It could just be the case that we're talking about different problems. At Yelp we have hundreds of services that are added, changed, moved, and removed from physical machines causing our internal HAProxies (which listen on hundreds of ports and many have long running connections) to reload pretty constantly.

As all engineering decisions, I could very easily be wrong and be overestimating the complexity involved with the multiple HAProxy instance solution. At the end of the day I made a call based on the data available, and I decided the solution I talked about in the blog post was lowest risk.

jolynchon True Zero-Downtime HAProxy Reloads

This was more complicated that I hoped it would be

I also hoped that it would be less complicated, but after prototyping a few different options, many of which I mention in the blog post (e.g. a patch to do fd passing, huptime, or running multiple parallel HAProxies), I decided that this was the lowest risk given the engineering constraints that we had. Also I wasn't entirely sure of an implementation plan for how to cleanly do port mappings for the hundreds of services. For example our service provisioning system adds, change, move, or removes services in various Yelp environments dynamically, so the port mapping logic would have to keep sync with that. It is certainly possible but I stand by my concern.

To put this in perspective, I got this working in production at Yelp in about two or three days, and it has not had a single issue since. With this solution we apply some TC rules and some iptables rules once, and then the act of restarting is just a set of TC commands. It solves all of our use cases and does so without significant infrastructure that someone else has to maintain two years from now.

If a month from now HAProxy releases native zero downtime restarts, I don't feel bad about throwing all of this away.