Blending complex systems made my latency 10x higher
srvaroa.github.io
srvaroa.github.io
In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.
Better (more accurate) title would have been: "Merging complicated software systems made my latency 10x higher"
I also get the sense that that team has a better than average allocation of resources. Some teams I've been on, this type of problem would be the responsibility of one person within an afternoon, or with impatient product people and managers checking for status after that.
I don't know. k8s is pretty complicated. How many small/medium apps need more than this nginx/Terraform/Docker example? This would be a lot more difficult to set up in k8s (pods, ingress, etc.)
Terraform:
resource "docker_container" "app" { count = 2
name = "app-${count.index}"
hostname = "app-${count.index}"
image = "app:${var.app_version}"
restart = "always"
env = [
...
]
ports {
internal = parseint("${var.app_port}", 10) + count.index
external = parseint("${var.app_port}", 10) + count.index
}
networks_advanced {
name = "${docker_network.private_network.name}"
aliases = ["app-${count.index}"]
}
depends_on = [
...
]Nginx:
http { ...
upstream app {
server app-0:3000;
server app-1:3001;
}
server {
listen 80;
root /usr/share/nginx/html;
location ~* ^/api/ {
rewrite ^/api/(.*) /$1 break;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_pass http://app;
}
}
}Is there some epidemic of developers setting up complex, multi-node Kubernetes clusters just to run a small web app ?
Why?
Because too many engineers I've worked with would bump into this situation and this would be their answer. They wouldn't take the time to debug the situation deeply enough and they'd blame k8s, or blame the network, or blame...
In my experience, the most common issues with complex distributed systems are much more likely to be due to misconfiguration because of a limited understanding of the systems involved than such issues are to be caused by core, underlying bugs. And I believe that's why some engineers shy away from otherwise valuable frameworks and platforms: they have a natural and understandable bias to solve problems via engineering (writing code) than via messing with configuration parameters.
Title should be: My configuration made my latency 10x higher.
It would be more accurate to criticize AWS's Kubernetes support. Both KIAM and the AWS Java SDK are specific to AWS.
It is after all running the control plane in multiple AZs.
For service behavior (e.g. request/connections), timeouts provide value for services and clients. For services, if you never time out then under failure conditions you either end up saturating your max concurrent request limits or growing your concurrent connections indefinitely until you hit a limit (connections, threads, RAM, CPU). Unless all of your clients are offline batch processes with no latency SLA there's a good chance that the work clogging up your service was abandoned by your clients long before it completes.
Timeouts also help clients decide if/when they should retry. Even if the service never times out, clients can't really tell if their request is just taking a long time or if something is wrong and the request will never succeed (e.g. network partitions, hardware failure). There's at least implicit latency SLAs on most things we do (1 second? how about an hour or week?). Given that there is a limit somewhere, it makes sense to use that limit to get benefits like resiliency in services.
>>>Your timeout is not my timeout.
Absolutely. Client deadlines are a great way to reduce wasted work across distributed systems. e.g. service has 60s timeout, but client has a 5s timeout for an online UI interaction. The client can tell the service to give up if it's not completed within the client's SLA.
Companies can either choose to have their devs take on ops responsibilities or continue having dedicated ops jobs.
In either case, whether or not dedicated ops jobs exist, ops responsibilities always will. I'll be there to pick up the slack because designing and maintaining systems is an interesting job that has to be done that a lot of people can't/won't do and it pays accordingly.
1) Developers don't code on prod
2) Prod needs to be protected
3) Therefore, developers need to be restricted from prod
Source: My employer is currently undergoing SOX compliance
I market myself in interviews as a software engineer (on the same team as the other SWEs) that just happens to focus on what actually getting this product into production and keeping it going looks like.
I got my start in this by working at a startup where we were responsible for everything just by the nature of the limited staff. I think that's how a lot of people in Devops get their start.
I didn't even know coding without a thought about deployment was a thing until I joined a huge company.
And here lies the problem.
We had "Ops" before. People would throw stuff over the wall from Dev to Ops, and now it became "Ops" problem. Operations would stonewall releases – because releases bring problems, and they want to keep things stable. Developers would push for releases, because that's what they are paid to do.
To solve this conflict, someone came with the idea of "DevOps". You don't have a split organization now, you have some developers in your team that can wear the "Ops" hat. That way, they experience production issues first hand, and that allows them to architect the application better – as well as ensuring it is thoroughly tested – noone wants to wake up at 2AM.
And, because they are developers at heart, they will do things to make their job easier and more predictable, like infrastructure as code. And heavy automation.
Somewhere along the way, people misread the whole idea, and now think that "devops" == Terraform or some other tool. Then they rebranded their Ops org as DevOps and hired a few automation folks and called it a day. Ops rebranded is still ops.
It was not supposed to be like that. If you have a "devops" team, you are not doing it correctly. Call it operations like it should be called. You may even have a dedicated team working on common tools or similar – which is fine – but you still need developers doing operations and ultimately responsible for their own shit, otherwise this is not devops.
> I got my start in this by working at a startup where we were responsible for everything just by the nature of the limited staff. I think that's how a lot of people in Devops get their start.
Yup. I got into that by trying (and ultimately failing) to launch my own startup. We didn't have dedicated operations folks, or time to manually tinker with servers, or to respond to repeated production issues. When you are sales, operations and engineering, you try your hardest to not have incidents (so that you don't lose customers), but you are trying to move fast to deploy features (so that you can sell).
To be fair, most ops people i know worth their salt have been automating since forever. Doing operations at scale and still staying sane is impossible without automation.
The largests issue in ops is that reality in production is usually far more messy then development. Some minor issue which can be easily fixed in dev by something simple (take a reboot of a service for example) can be a major pain in production because of several factors. Usually related to interdependencies or customer impact.
Also, production (especially on the networking/hardware side) is usually hard because of things like hardware failure, physics itself or simply human error. These gems for instance[1][2].
[1] https://www.cisco.com/c/en/us/support/docs/field-notices/636...
https://www.gartner.com/en/information-technology/glossary/d...
It's common to mistake DevOps meaning devs wearing ops "hats".
Because I've lived with both, and to hell with both of them.
It's a delicate balance to strike, especially considering having a product with a ton of features but lackluster stability doesn't get you anywhere, and neither does having a super stable product that lacks basic features.
"move fast and break things" doesn't work in a lot of sectors. It might work for an internet startup, but good luck applying that principle to fields like large infrastructures and datacenters.
AWS have recently added the feature natively [2], so once that version of the SDK is in use by all your pods you won't need Kiam any more.
[1] https://github.com/uswitch/kiam/issues/191 [2] https://aws.amazon.com/blogs/opensource/introducing-fine-gra...
That was compounded by the metadata requests using DNS and the metadata IP, and until recently Kubernetes didn't have any built-in local DNS cache[3] (GKE still doesn't), which in turn overloaded kube-dns, making other DNS requests fail.
We worked around the issues by disabling metadata concealment, and added metadata to /etc/hosts using pod hostAliases:
hostAliases:
- ip: "169.254.169.254"
hostnames:
- "metadata.google.internal"
- "metadata"
[2] https://github.com/GoogleCloudPlatform/k8s-metadata-proxy
[3] https://kubernetes.io/docs/tasks/administer-cluster/nodeloca...
I suggest more people read the first few chapters of Specifying Systems by Lamport. Maybe the rest is good also, but that's as far as I got.
It works through a trivial system (display clock) and combines it with another trivial system (display weather).
Nothing Earth-shattering, but it really stuck with me. Thinking about it at that level gave me a new appreciation for what combining two systems means.
> DNS resolution is indeed a bit slower in our containers (the explanation is interesting, I will leave that for another post).
I would like to see this expanded upon or hear if anyone else has suffered similar.
Forwards compatibility is painful.
I know a small percentage of clusters makes their service networks public with dns resolution (so a.b.svc.cluster-a.myco is reachable from most places).
The “namespace” auto-injected file was created long after this was settled, so that wasn’t an option. I believe most of the input was “the auto env var injection that docker did we don’t like, so let’s just keep it simple and use DNS the way it was quasi intended”.
Certainly we intended many of the things istio does to be part of Kube natively (like cert injection, auto proxying of requests). We just wound up having too much else to do.
The title is a bit misleading, kub didn't cause the 10x latency - also latency was lower after they fixed their issues
TL;DR version - Migrate from EC2 to Kub; due to some default settings in Kiam & AWS Java SDK, latency of application increased, fixed after reconfiguration and kub latency lower than EC2
That said it is a very common deployment strategy in ec2 to run kiam or kube2iam. I wish the kube core teams took over the development of an aws iam role service since issues like bad defaults would be solved much quicker. Your only other alternative is to use iam access keys and nobody likes that (security wise and it’s a pain to configure).
I'm just saying the title is misleading, and clickbaity.
It’s sort of like cursing Microsoft every time an application crashes... because it’s running on windows.
More complicated than what? Without a baseline for the comparison it's not that useful. In our case we transitioned over the last four years from running hundreds of VMs provisioned with puppet and jenkins to running K8S workloads (on a lot fewer nodes than we had VMs) provisioned with helm/kustomize and using gitlab ci/cd pipelines. In my opinion the current platform is much less complex to understand and manage than the old one was. Yeah there are layers of complexity there that didn't exist in the previous platform, i.e the k8s control plane, the whole service/pod indirection, new kinds of code resources to manage, but it's all pretty consistent and works logically, and isn't really any harder to internalize than any other platform-specific complexity we've had to deal with in the past. And in terms of day-to-day development, deployment and operations k8s has been completely transformative for us.
No it doesn't.
This is only true if you don't need the features in the first place, and you haven't gone through the learning curve yet.
So I can deploy a single container. Works fine. You can tell docker to restart it. Works fine. If that's all you need, you don't need k8s.
But you may now want to deploy two containers. And let's say they are servers. Now, you need incoming connections from a single port going to them. So maybe you are going to deploy something like haproxy – which now you have to learn.
Also, you now need a way to tell if your service is down. So you add a health check. But then you want to kill the container if it goes bad. So now you add a cronjob or similar.
At some point, one machine is not enough. So do you replicate the exact workloads across machines? If you do, it is a bit easier - but potentially more expensive. But if you need to spread them across different machine types, it gets more difficult. And now you also have storage distributed which you need to keep track of. If you haven't automated your deployment, now it is the time.
Now you have this nice setup. It is time to upgrade. How do you do that? Probably some custom script again.
Instead, you can forget all of the above. You can spin up a k8s cluster with any of the currently available tools (or use one of the cloud providers). Once you have k8s, a lot is taken care for you. Multiple copies of the service (with the networking taken care for you) ? Check. Want to increase? kubectl scale deploy --replicas=<n>. Want to upgrade? Apply the new YAML. Health checks and auto-restart? Add a couple of lines to your YAML. Want to have different workloads over different machine types? That's also an easy change. Want to have storage (that follows your workload around)? Easy. You are in a cloud provider like GCP and want a new loadbalancer (with automated SSL cert provisioning!)? A couple of lines of YAML. Want granular access controls? It's built in. I can go on.
Of course, there's a learning curve. But the learning curve is also there if you stitching together a bunch of solutions to replicate the same features. Once you get used to it, it's difficult to go back.
Again, my point was that rather than trying to do PR damage-control it would be better to work to improve things so this doesn't understand. Someone went to the trouble of posting a detailed examination of a real problem with a fix and some references to upstream improvements. That's a lot more useful than trying to draw a line between one of two tools commonly used to meet security requirements when operating Kubernetes in one of the most popular cloud environments.
KIAM and the Java SDK have bad timeout overlap. That's the problem, has nothing to do with Kubernetes, and looks like it's well documented and now resolved.
So would disagree that is specifically is a necessary component.