A Google Cloud support engineer solves a tough DNS case

I'm stealing this.

Cattle is a threshold, but when the same problem keeps coming up it's time to call the vet. http://rachelbythebay.com/w/ has many good examples of this, some submitted and voted up here.

The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the same issue.

Servers as cattle can cover a variety of sins.

And even if you could somehow bisect every single configuration change in your configuration management, there's the added complexity of how many configuration changes are actually needed to test if the problem is still present. In this case it would probably be fairly easy because DNS is such a core "feature", but if this is something more application-level, you're really going to be lost.

It's often a useful exercise to dig into the root cause - a lot of times the problem you're seeing is just the tip of the iceberg.

Of course, anything necessary to get your production box producing, but a well-engineered server is worth the debugging time.

+ if (rmem > (size + (unsigned int)sk->sk_rcvbuf))

However in reality this would have worked too - + if (rmem > (unsigned int)(size + sk->sk_rcvbuf)) (The bit pattern of the result remains the same and it's still casted as unsigned int during the comparison)

However, signed integer overflow is undefined behavior in C and unsigned integer overflow isn't. Hence, the submitted patch is the correct solution

https://cloud.google.com/support#support-plans

$250/month/dev is the minimal for phone calls on technical issues, $150k + 4% of GCP spend for 'come running' support.

There're more details here

https://cloud.google.com/support/docs/procedures#additional_...

though they use the old names for the support tiers.

If a service you depend on is down, your support agent will be able to tell you it's down, but not speed up the fix.

Cloud support will have more information about performance black holes and limitations that the documents don't describe. They also will be able to advise on "is this design or that design likely better". They generally know how the backends of GCP services work, and their common failure modes, which is pretty hard knowledge to get from the outside.

Free customers did report bugs, and we would replicate, triage, and fix them as usual. These tended to be more obscure bugs (more severe ones would usually surface in the paid queues first), but we didn't discard them immediately.

Whether we effectively ignored bugs depended more on the ability of the customer to provide an actionable report. Some users provide exactly what we need up front, but there is a lot of "it doesn't work!" white noise from users that aren't able to or aren't willing to put in the work to accurately describe their issue and/or action feedback from us. There's usually not a whole lot support can do in that scenario if we don't see any obvious issues, but we'd go a bit further to placate paying customers--I fondly remember joining a call between some very technically inept user and their ISP who was adamant that either we or the ISP were at fault, after we guided their network team through taking local packet captures showing unanswered SYNs past their network border.

I actually started putting assertion checks about overflow issues almost everywhere but it requires great discipline. I wonder if there is a better solution available.

The difference for users can be fairly small, but as someone who used to carry a pager for Amazon, the difference is really huge for the support person.

Probably this bug would not have happened if this comparison were written as `rmem - size > sk->sk_rcvbuf`?

So it is saying that the simplest sanity check is "if the buffer was already full before we staked our claim we should drop this packet immediately." As the "goto" indicates, there are then a bunch more checks on other circumstances where we should also drop the packet. Due to the quirks of multi-threading it is of course possible that some packets get unnecessarily dropped between when we stake the claim and when we discharge it, which the code just accepts -- the thinking is presumably "yeah if the buffer is full a lot of packets are gonna get dropped and that's just life -- it's much less important that we dropped some extra packets when we were already dropping packets, and much more important that we don't mismanage the buffer's memory when it's nearly full."

A comment suggests that part of the reason for this awkward phrasing is that it is possible for rmem = size, in other words the buffer was empty when we staked our claim--and in this case we don't want to drop this packet even if it would overflow a small max-buffer-size. I think the idea there is "we already have the socket buffer allocated, obviously this thing fits in memory, so let's just handle it if the queue is empty rather than dropping every single packet that is larger than the queue size."

Whereas the commonly used socket functions recv/send construct the required headers for TCP, UDP and whatnot, they handle encapsulation/buffering/connection/etc so they're easy to use for developers, just read and write application data.

By nature raw sockets skip TCP/UDP libraries and a good chunk of the network kernel code. Including the place where the bug was located.

In my experience, Google Cloud is better than most organizations about escalating hard issues up to the chain. Admittedly, this happened at a company with substantial spend, and I can't say one way or another whether a smaller player would get the same quality of support.

On the flip side, it's encouraging that they have people somewhere in the support chain who are capable enough to read Linux kernel code and submit fixes upstream.

We have the expensive off-the-shelve support option (I think 450/seat/month) for 1h response.

In most cases, we spend more time back and forth with support than it would take you to figure it out. I'm talking about issues that span over weeks with tens of hours spent. We end up reiterating the original support case problem (i.e. the support engineer doesn't bother reading the actual problem) whenever the engineer changes.

We've had: P1s where the support engineer told us we'll get an update the following day, only to figure out that there was a breaking release on their side that exactly matched our description. While investigating a load balancer issue, the support engineer looked at the LB logs, saw a ton of logs coming from penetration scans (e.g. GET /phpmyadmin), and suggested that the solution was to open up those addresses.

https://en.wikipedia.org/wiki/Reverse_DNS_lookup#Uses

This aside, even though we had too many support cases so far with AWS, and having highest support level, they mostly cannot access user data, just the metadata. We had a major problem with RDS once, and they specifically requested to load that snapshot to an internal instance to reproduce. It can happen in AWS, but not very common, in my experience.

When I was a new engineer working in telco, one of the longest investigations I worked was when connectivity broke between one of our regional roaming partners and 1/3 of our nodes (I'm summarizing to try and keep the story brief). We called them and asked if they changed anything, reviewed the configuration and secrets used on the tunnels, etc. And were working with the vendor to go through any problems with the implementation. Saturday morning and probably 20 hours of investigation later, a new engineer at the regional partner see's there is a work order for changes to the connectivity to our nodes (we were adding some new ones) that was supposed to be executed that week. A typo in the change overwrote the secrets used by an existing tunnel instead of creating a new secret for the new peer. The person we were working with to investigate, was the person who implemented that change and told us several times nothing changed. He was also the one we worked with and read through all the secrets for typos or issues and didn't notice anything. Saturday morning he get's into the office, is shown the work order, and goes, oh yea, I did that at exactly the time the tunnel went down. Fix of typo'd secret later and everything comes right back up.

So just in my experience, I find it quite plausible that buffer size was not mentioned. And even besides this story, I know I've personally missed connecting causes with potential effects when investigating a problem, it's very easy to dismiss some setting, like the buffer size, as being connected specifically to DNS behaviours, especially if they are not noticed together or with a strong change management system that helps connect the timelines together.

1. They need guaranteed delivery, but chose to use UDP 2. They jacked up the default rmem buffer to ~2GB which is insane. Also, applies to all sockets not just UDP, so I wouldn't be surprised if they where also running into issues with memory pressure especially under load 3. Support didn't seem to let them know that's a pretty unconventional configuration

That was an interesting debugging story, and catching a bug like this is always good IMO. But, there is just so much WTF in this setup.

As another commenter mentioned, this was the result of customers never actually mentioning their weird sysctl tuning in the original issue description. It's not like they're trying to screw you over or anything - there's just an awful lot of config options in an entire system that does anything interesting, and in the case of a big enterprise appliance, it's likely that dozens of people have had admin on it at one point or another.

We rarely have to do so, but whenever we did, they went out of their way to figure it out.

YMMV, I guess?

If you're 5 people at a dentist office with GSuite, that may be a different story. That's always a problem when small entities buy direct -- that's why VARs exist to provide more handholding for smaller orgs.

I've managed some pretty significant vendor relationships -- if you think they are the worst from a support POV, you're young or exceptionally lucky!

We got put on their best support plan for a few months for reasons. The difference was insane. I gave them permission/creds to log into the problem box, along with steps to repo things. In a few days they had reverse engineered what was going on in our code, without having our code, figured out where the bottleneck was in the kernel, and gave me detailed steps to build a tweaked kernel that wouldn't be a problem.

Even their product teams are just... Depressing. There was a support ticket open for TEN YEARS for people asking for WebSockets support on Google App Engine.

I often wind up with DevOps responsibilities, and I'd never recommend building more on them, and I'd help in every conceivable way to minimize money given to Google, in addition to aiding the transition to more reliable providers.

And on a more personal note... My husband recently tried recovering his email. He no longer had the password, the phone number he had it registered with was no longer owned by him (a major security issue, btw), so he tried his recovery email. And even after clicking the link from his recovery email, he was denied access, and sent to the same help page that couldn't be more unhelpful if someone actively tried. And there was no apparent way to contact support from that screen.

I hear tell of Gmail users who could reach Google support, but... We don't put much stock in those stories round these parts.

Moral of the story? Never trust Google with your emails, or other important information. Always make backups if you must continue using them, and forward your email to an email provider you trust (ideally one that you pay for, own, and has a decent support department of any kind).

Computer Networks and Internet

Internet working with TCP/IP Volume III

The Linux Programming interface

Or even when you are writing low-level code on a known platform (e.g. an LP64 platform).

It does show that embracing good change management techniques is important. The VM in question clearly had runtime configs updated manually or other VMs would have exhibited the same problem. Ideally orgs should be able to identify when the change was introduced and the rationale / tests that were used.

By the time it stopped transferring, it was already diagnosed as a hard problem and the time pressure was off.

It was sent to the customer to collect experimental results while investigation continued in parallel. Per the post, production mitigations had already been put into place, and the customer was knowingly participating in an investigation of an unknown issue. The customer provided the requested experimental results.

It can also help because it shows the customer you're making progress.