898 karma · joined August 31, 2010
Reminiscent of Knight Capital losing $440 million in 45 minutes via feature flags: https://dougseven.com/2014/04/17/knightmare-a-devops-caution...
I don't agree with your comment about UK baguettes though. In London at least, you can get very good baguettes from Orée, le Pain Quotidien, (to an extent) Paul, and a bunch of small local bakeries here and there. As a Frenchman, I find these to be just as good as what you'd get in any French bakery.
The HN headline is also "wrong" (or at least imprecise) - the fine amount is in NOK, so the euro figure is ~6.418 million depending on exchange rates.
Terraform does this very well too, using either the Kubernetes provider or the Helm provider.
Fargate Spot is about a third of the price of Fargate (at least in eu-west-1 now according to: https://aws.amazon.com/fargate/pricing/ ); so the savings are roughly identical.
Re risk of running out, our current strategy is to use different-but-closely-similar instance groups; so for example we have an autoscaling group running a mix of:
- m5.large - m5dn.large - m5n.large - m5ad.large - m5d.large
Which are the same price on Spot instances, but I'd wager it'd be pretty rare to have all these families reclaimed at once.
(We also use some on-demand only ASGs with lower priority in the cluster-autoscaler to ensure that if it _does_ happen, then we'll have a fallback)
Not sure if it's temporary down or fully off.
> Why the worker count varies
> The rule I used for deciding on what the optimal number of worker processes was is simple: for each framework I started at a single worker and increased the worker count successively until performance got worse.
> The optimal number of workers varies between async and sync frameworks and the reasons are straightforward. Async frameworks, due to their IO concurrency, are able to saturate a single CPU with a single worker process.
> The same is not true of sync workers: when they do IO they will block until the IO is finished. Consequently they need to have enough workers to ensure that all CPU cores are always in full use when under load.
- Control-plane performance on a cluster with a large-ish number of pods (thousand+) can be hit-and-miss, and "what to scale up" was hard to pinpoint (though admittedly it seems to be getting better)
- Istio upgrades often are a pain, but mostly around the actual way of deploying the upgrade, rather than the upgrade itself. For a long time there was no official Helm chart, then there was a Helm chart, then two Helm charts, now it looks like Helm is deprecated and will be removed; instead installing via `istioctl` is recommended... Some of it is due to the pain of upgrading CRDs, which is a general Kubernetes issue, but there's still a _lot_ of churn to keep up with.
- Adding a new VirtualService registered to the same hostname as an existing one will be accepted by Istio (at least as of 1.4), and will proceed to _silently break all routing for new pods joining your istio cluster_! This was a bear to debug too, given how noisy and confusing the Pilot logs are, and we ended up stitching up a custom Prometheus alert around this given it bit us roughly every other week
- HA for control-plane components isn't explained in the docs. Is it safe to run two Citadel pods? We did it and it seemed fine, but who knows?
- We sometimes ran into pathological cases where traffic would for some reason completely drop after a new deploy, and gradually pick up after the config was stremed onto sidecars, over a span of ~10-15 minutes. We never managed to debug this issue (which happened probably half a dozen times over a year), and that mere fact turned me off the complexity of Istio in general.
(Some of these might have been fixed in Istio 1.5+, as 1.4 was my latest experience)
Of course, once your setup is stable, everything is awesome: sidecar injection works flawlessy, observability is awesome, distributed tracing is a breeze, Kiali is a great crowd-pleaser when showing off features, mTLS + TLS origination mean full on-the-wire encryption without losing any of the previous benefits... A lot of features that meant we carried on with it, but if I had to start again I'd probably have a good hard look at Linkerd before recommending Istio for any prod setup.
Re the AMI - yes you could use a data block, but that would immediately apply the latest version of an AMI when executing terraform, leading to unreproducible builds. Zeitgeist instead would just notify you that a new image is available, but let you control when you choose to deploy it, for example once you’ve confirmed that the new image passes all your tests in a dev environment.
(And if the AMI is for an individual EC2 instance instance rather than an ASG, you’ll have to plan downtime while the machine reboots with the new AMI, so a data block can be pretty dangerous there)
Re Terraform itself; where you’re running it is irrelevant - it needs to be kept up to date somehow. As mentioned elsewhere in this thread, some of these updates (like 0.11 -> 0.12) need to be carefully planned and managed... and how would you do that if you don’t know updates are available :)
Keep up the good work. :)
Despite thinking I knew SQL reasonably well, I wouldn't have fared very well at all in an interview setting. :/ Took more time and googling than expected.
:(