Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP
blog.jetstack.io
blog.jetstack.io
These kind of reverse dependencies, where app level changes have to be reflected in infrastructure are source of endless bugs and headaches.
To do it right, you'd want to encode app and infra changes in the same ubiquitous tool, but "infrastructure as a code" tools such as terraform suck big times at app level deploys.
Pulumi takes steps in right direction, where it is actually not that painfull to manage everything, that is what I ended up using for very similar problem configuring GCPs load balancers for my k8s apps.
The only and most frustrating missing piece of the terraform kubernetes provider is CRDs. I hacked around it by using some yaml provider bit I would like a cleaner solution.
Well, that's exactly what we do with Dhall, which lets us put static types on infrastructure tooling that prints out unpredictable output but in a predictable format and transform that output into the format that other dependent tools expect. Everything is then glued together with simple shell scripts and run in a CI environment.
Once you understand the pattern - use an uber-config plus typed tool outputs to generate downstream config and apply all config in an idempotent way - you wonder why you ever did things any other way.
If Dhall enables safer bash glue, I'd be happy to read how
In other words, there is no way to express sequence of changes of the same object in the single run of IaC tool. We are forced to wrap IaC tools in layers of bash to simulate it.
What you're describing sounds a bit like migrations for databases. Do you agree?
We use terraform end to end and tried the http/https L7 loadbalancer first in our setup but I had a heck of a time with:
- there is no API for the annotation mentioned in the article so if you miss it while setting up a backend, nothing works - gRPC was hard to get going. Most gRPC examples out there use some port such as 50051 so you assume that is needed. gRPC does work over 443.
We currently use istios ingressgateway as a loadbalancer that you set in your kubernetes setup. It works but I don't know if it is better or not. We had to run an ingressgateway pod as a daemonset on every node so that we could get the real IP addresses from requests for security logging. That was a pain.
> Kubernetes Accelerated Your Path to Enterprise Cloud Native
I have no idea what they want from me.
Just for fun, I used Grover to generate an article based on the headline:
> What is the cheapest way to optimize the cloud? Going globally? Using open-source cloud computing platforms? Choosing AWS in the U.S. and for the remaining countries? Choosing SaaS? Or, using AWS and establishing a global presence on the EC2 Container Manager platform?
> I know because I did all four. And I won’t do them again.
> I am not arguing that globalizing your compute resources through a community cloud is any less important than globalizing your compute resources through your enterprise cloud (though I am somewhat of a skeptic about the value of community clouds). The important point is that migrating to a global cloud requires a state-of-the-art approach to container-native load balancing, container-native load balancing on AWS, and container-native load balancing on other clouds. And you need to be prepared to pay for it.
This article is talking about routing traffic from a single IP globally to multiple backend Kubernetes clusters that can be running in any region. Traffic will automatically go to the closest region (with available capacity) or otherwise fallback to other endpoints, while also bypassing much of the Kubernetes network stack to go straight to the running pods via GCP's software-defined network.
The complexity here is warranted if you need the flexibility and features. If you don't then you can just stick with nginx.
However, this is embarrassing to GCP that this isn’t easier. These were very basic requirements from this customer, yet so much rigmarole was needed to get it set up.
AWS requires stacking the Global Accelerator on top of their ALB/ELBs but doesn't have smart routing across clusters and Azure only has Frontdoor which requires completely manual setup and has no backend integration.
The authors had to go read source code to figure out obscure annotations at one point to get container aware networking to work, which then wasn’t compatible with Google’s tools to setup global ingress.
What other cloud vendor or PaaS offers this? The only one that has seamless global deployment is Cloudflare's Workers but that's limited to Javascript code in a lightweight functions environment.
It’s common enough they are trying to support it: https://cloud.google.com/blog/products/gcp/how-to-deploy-geo...
Their tool is at least three years old and still has this disclaimer: “Caution: The kubemci tool is a temporary solution intended to help users begin using multi-cluster Ingress. This tool will be replaced by an implementation using kubectl that delivers a more Kubernetes-native experience. Once the kubectl implementation is available, you will need to manually migrate any apps that use kubemci”
AWS is the VMWare of our age, you'll never get fired for suggesting it. There are workloads that are inappropriate for it (see: Dropbox) but those are few and far between.
I can run all of my container infrastructure for a property for the cost of the control plane for EKS (before you attach hosts and before you spend how much time figuring out why hosts won't attach to it). It's just bad.
Also not having load balancer support built in kills it for me. Yeah you can do metallb or nginx ingress but it means they punted one of the major components needed to make it, 'cloud agnostic.'
Meanwhile in AWS we have ECS which works out the box on fargate, their EC2 hosts, or your own EC2 hosts and on hardware I can easily get (and dev) on docker swarm.
There are lots of production-quality load balancer implementations for Kubernetes, such as Traefik, as well as operators that configure cloud-native load balancers (e.g. the one that configures GLBs on GCP). I don't see this as missing functionality. The ingress story isn't great, but the available options (Nginx being a very solid example of something tried and tested) are good enough that it's more a problem of standardization, not implementation.
Neither of those two points seem to sufficiently argue against using Kubernetes. Having used it for a few years now for all sorts of workloads, I would never go back to plain VMs, nor to something like ECS or Mesos. The only alternative I might entertain would be "serverless", but the available offerings don't seem to cover all the bases (e.g. batch jobs).
ECS with Fargate behind ALBs talking to RDS/SQS/Elasticache with Scheduled Tasks as my cron layer is 99% of what we need without standing up a single host we have to maintain.
I put that in my calculator and it makes a happy face.
At scale I agree, however if you're a small company (or, in our case, 6 small companies) and don't have dedicated DevOps (or, in our case, have 1.5 DevOps people split 6 ways), it's fantastic.
A place I used to work at migrated from GCP to AWS because they don't allow port 25 incoming; if you're doing e-mail, you 100% need this.
The point is to make sure that anything sending email from GCP has someone properly attending to it, so that GCP does not become a source of spam, rather than to prevent email service providers or the like from using GCP.
For many companies that aren't an email service provider, it's often a better use of IT funds to send via one anyway, given how much work is involved in maintaining security and deliverability of an outbound email server in 2020. Most of those support sending through ports that aren't 25, and Google has some nice free tier deals for GCP customers.