Amazon EKS Anywhere | Hacker News Reader

Amazon EKS Anywhere

github.com

158 pointsdidip54 comments

I hope this will allow us to contribute toward fixing some of the longstanding issues. One that I'm still surprised hasn't been addressed is the way that it grants irrevocable admin permissions for the IAM security principal that created the cluster. And it does it invisibily–there's no way to see what IAM role or user has system:masters since the permission doesn't appear in the configmap or any other user-visible configuration. Last I checked you can't even see it in the dashboard and have to make note of it yourself for future reference.

Its even worse than you describe, its by name. It doesn't have to be the same IAM user, that user just have to have the same username as the original creator. Support told me this when we got locked out of our cluster and the creator had moved on to other things.

Oh. Wow. That is honestly insane.

Yeah, unfortunately for security minded folks the unique ID of the IAM security principal isn't used instead of (or in conjunction with) the ARN. Or perhaps fortunately if you are blindsided by a security setting that makes you wonder how thorny it is on the backend if this shipped to production in the first place. I experienced something similar. We tried recreating it on a whim thinking it wouldn't work but needless to say everyone on that call was surprised.

I know they did (do?) some weird stuff exposing the control plane with VPC Endoints that seem like some kind of weird hybrid of AWS-managed endpoints and privatelink. I always wondered how scrappy the team had to be at the beginning when they were still pushing ECS hard.

Agreed. This is extra bizarre considering their own documentation on the purpose of IAM unique IDS states:

>"However, every IAM user has a unique ID, even if you create a new IAM user that reuses a friendly name you deleted before. In the example, the old IAM user David and the new IAM user David have different unique IDs. You can create resource-based policies that grant access by unique ID and not just by user name. Doing so reduces the chance that you could inadvertently grant access to information that an employee should not have"[1]

[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_i...

My searches on this are coming up empty. Do have more info or perhaps a link that further explains this issue?

https://aws.github.io/aws-eks-best-practices/security/docs/i...

To save a click/scroll/combination-there-of:

https://aws.github.io/aws-eks-best-practices/security/docs/i...

> When you create an Amazon EKS cluster, the IAM entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster's RBAC configuration. This access cannot be removed and is not managed through the aws-auth ConfigMap. Therefore it is a good idea to create the cluster with a dedicated IAM role and regularly audit who can assume this role. This role should not be used to perform routine actions on the cluster, and instead additional users should be granted access to the cluster through the aws-auth ConfigMap for this purpose. After the aws-auth ConfigMap is configured, the role can be deleted and only recreated in an emergency / break glass scenario where the aws-auth ConfigMap is corrupted and the cluster is otherwise inaccessible. This can be particularly useful in production clusters which do not usually have direct user access configured.

Emphasis mine.

Thank you!

EKS Engineer here. This is something we have lots of thoughts about and want to make better in EKS on AWS.

For EKS Anywhere, you can configure your cluster with OIDC auth today, and IAM auth is coming very soon. https://github.com/aws/eks-anywhere/issues/90

Could you shed some light on why this was done this way? Was there a technical reason for not uasing ARNs or unique IDs?

This is probably the most interesting part: https://anywhere.eks.amazonaws.com/docs/reference/faq/#can-i...

> Can I connect my EKS Anywhere cluster to EKS?

> Yes, you can install EKS Connector to connect your EKS Anywhere cluster to AWS EKS. EKS Connector is a software agent that you can install on the EKS Anywhere cluster that enables the cluster to communicate back to AWS. Once connected, you can immediately see the EKS Anywhere cluster with workload and cluster configuration information on the EKS console, alongside your EKS clusters.

My guess is that this is similar to Google's Anthos Connect agent https://cloud.google.com/anthos/multicluster-management/conn... that initiates the bi-directional connection from the on-prem to cloud (as opposed to opening a firewall rule for the cloud to access on-prem). That way, the cluster shows up and is manageable through a UI that is hosted on AWS Console.

Another interesting point is that I don't see a fee or price for installing this to your local VMware set up. I am not sure if it has to dial to AWS APIs to work (my guess: not really). So it can be treated as just another Kubernetes packaging that runs the AWS' distro by default.

Props to AWS for putting this out there and doing it in the open. I look forward to reading on how it compares with other Kubernetes installation methods given this doesn't manage the on-prem Kubernetes for you.

This has been available with ECS for a few months as well: https://www.theregister.com/2021/06/01/aws_ecs_anywhere_goes...

It is by far the easiest installation method I’ve encountered for vmware/vcenter. I plugged in a few variables (Datacenter, VSAN storage pool name, Network name, and resource pool) and in 30 minutes I had a fully production ready cluster installed with all the vsphere cloud provider / csi plugins / etc. installed also.

AWS is lagging behind on managed K8s on-prem for air-gapped environments, consider Google had GKE on-prem (now a part of Anthos) long ago, and Azure offered AKS on Azure Stack HCI. Finally AWS caught up, feature parity between the big 3.

Adoption will be interesting to see.

I've worked on gravitational (gravity / telekube) flavoured K8s distribution designed for on-prem data centre deployment models for the past few years. Those customers will eventually move to 1. managed K8s (good on them) or 2. continue to self-host K8s to satisfy their building one's own PaaS goal lol (in that case, it's better be managed K8s' on-prem variant from GKE/AKS/EKS). For the same K8s solution's other model - BYO K8s, I see rapid adoption of AKS (vs GKE, EKS is the dominate from beginning as it was the first supported).

On gravitational: Recently I heard gravitational has shifted focus (and rebranded) to teleport (secure cloud access gateway - used to be a part of gravity), around their Series B funding round, after all, investors are looking for a business model that can bring them profit (and exit). On the other hand, `gravity` - `pull the stack from cloud` - where the name came from - which addresses specify niche market does not seem to have generated much revenue over the last couple of years.

>"I've worked on gravitational (gravity / telekube) flavoured K8s distribution designed for on-prem data centre deployment models for the past few years."

This sounds interesting. Can you say what K8S distro this is?

Telekube (renamed to gravity by Gravitational - which has again rebranded to Teleport ;-)

Open source edition: https://github.com/gravitational/gravity

Note: We have been using its 5.x branch (tracking K8s 1.13.x) for a long time. That's why later on BYO K8s model was introduced ;-)

Thanks. Oh that is confusing. I think I actually became more confused after I read the announcement of their name change :)

Wasn't Teleport the original name for the product?

I'd be curious to hear what's your plan going forward now that gravity is no longer under active development.

The existing appliance model will continue to be maintained (until upstream EOL it - or when the support agreement ends).

In the meantime, to address the same requirements from customers who wants some control (K8s stack, VMs, networking and storage are theirs), managed K8s' on-prem distribution may be the answer, I'll be exploring EKS Anywhere see how it works and see if it's a viable option to replace the appliance model moving forward.

Super excited to finally be able to run Kubernetes in places other than my AWS account!

Another k8s distro that’s hard to administer is exactly what I wanted, definitely not what I used EKS in the first place

(Real talk though, never heard of anyone at Amazon using EKS internally)

Internally do they use ECS?

(I used to work at aws)

yes! specifically, quite a number of teams use fargate and its one of the 2-3 "obvious" ways to build services.

Literally everything is a Lambda nowadays it seems, so it seems like it's just Lambda -> ECS -> -> -> raw EC2 unless you're building directly on bare metal.

Yeah, it really depends on where your service falls on the dependency chain.

Services that Lambda depend on are heavily encouraged to use lower level options etc...

Thanks!

We've been running on ECS for 6 years now (basically within a few months of GA). When EKS launched we talked about maybe migrating over but ECS just feels more first class wrt cloudformation and stuff. It makes sense that internal teams rely on it.

The big selling point for k8s vs ECS is that it (theoretically) allows you to lift and shift your cluster to another cloud provider without too much pain because load balancers/storage/etc sit behind an abstraction layer. When you are AWS that's not really something you're even going to consider, so you may as well build with the underlying primitives because its absolutely guaranteed no one is ever going to ask you to migrate to Google Cloud.

AWS SA here, I know several AWS Services that are now built on top of EKS or using EKS. One example is managed Apache Flink on Kinesis Analytics.

"Kinesis Data Analytics deploys Apache Flink using Amazon EKS. Multiple Kubernetes pods are used in Amazon EKS for each AWS region across availability zones." https://docs.aws.amazon.com/kinesisanalytics/latest/java/dis...

Amazon.com runs on EKS (and other services). You can see public customer references here https://aws.amazon.com/eks/customers/

ECS is 3 years older than EKS (2015 vs 2018) and a lot of AWS services launched (or planned to launch) during that time which is a very valid reason lots of customers picked ECS.

(I work on the EKS team)

Where else are you going to pay those bandwidth rates?

My local loanshark.

It's like Google and Amazon decided to swap roles in the "clearly naming your product" game for this line of products.

"EKS Anywhere" is a lot more clear than "Anthos".

Granted, "Anthos" has grown into something of a larger brand for GCP, but GCP is usually a lot more literal than that.

For what it's worth, the analogue to Anthos is AWS Outpost.

But you could argue Outpost still has a more clear name for what it is, although I wonder if they actually thought through the definition of the word: a small military camp or position at some distance from the main force, used especially as a guard against surprise attack.

Wasn't at all clear to me what it does.

"EKS Anywhere" doesn't make sense.

to me the 4 main benefits of EKS is 1. managed control plain. 2. ebs storage integration. 3. amazon vpc network integration. 4. IAM integration.

EKS anywehere provides none of these, at best maybe it is providing similar networking behaviour, but that is actually happening with a 3rd part CNI, so even if it is consistent which i am no convinced it is, i don't need eks anywhere for it at all.

I would have expected it to be some sort of system that would let me use my EKS master to run nodes anywhere

Number 1 benefit of EKS to me is Fargate pods, which don't look to be available in EKS-A.

Kubernetes: lets commoditize public IaaS by containerising workloads thus making them IaaS-independent!

Public IaaS: lets commoditize Kubernetes by making a multi-cloud Kubernetes control plane! (AKA Google Anthos, AKA EKS Anywhere, AKA AKS on ARC)

This is how I see Kubernetes:

https://imgur.com/dsiNh5d

How does this compare against simply using Gardener [0]?

[0] https://github.com/gardener/gardener

How is this different from Azure's AKS on ARC which you can run on prem nodes but managed via Azure management plane?

In 8-12 months EKSA will either be very successful or will die, because it’s up against incumbent on-prem k8s so teams will either quickly see the benefits or it’ll quickly be dismissed as useless.

The target market seems to be infrastructure admins who have on-prem VMs and an AWS account and don’t like running their own K8s clusters. No idea what that market looks like in terms of size or interest in something like this, but I’m sure AWS did their homework.

My company committed to AWS-native EKS, didn't want to deal with the management headache of on-prem or self-deployed Kubernetes in AWS/GCP, presumably AWS support is pretty good, DevOps is happy so far. One anecdotal win for Amazon.

Not directly related, but ECS Anywhere has been a pretty OK experience.

Same - locally no one wants to run the admin plane, but folks are happy to see some beefy machines in their AWS account.

Problem is usually if data is in AWS, you want compute locally - the on prem has the bandwidth charges so need to be careful there.

A cloud-hosted AWS console changing configs on an on-prem vSphere cluster doesn't sound like something that's going to appeal to a broad audience to me. There's also the option of having your on-prem EKSA cluster have a hard dependency on the cloud AWS IAM. Youch.

I'm not convinced the target market is people who run their own data centres, those people are highly unlikely to migrate the control plane to AWS. To me the big target markets here are:

* People running compute on AWS, but content distribution on their own hardware co-located at ISPs, those CDN nodes need to have the software running on them managed, and I can see the appeal of doing so with the same control plane everything else uses.

* Organisations running heavily distributed compute, either for the purposes of latency or resilience. Many large retail chains have small remote nodes running the PoS systems at each site so that they can continue making sales and tracking inventory in the case of a central outage. Again, seems like it would be nice to just treat those nodes as an extension of the central servers on AWS.

* (Somewhat more dubiously) IoT providers. AWS have a couple of services which are much better suited to managing IoT devices, but I could see some arguments for treating an IoT hub as a k8s node and just deploying new containers to it when updating.

This is pretty cool and I think outstrips GKE on-prem/Anthos in developer centric experience.

GKE on-prem uses VMware's vCenter Server. But I like that EKS Anywhere is basically GitHub repo with their CI and their custom kubernetes distro.

Far better than what everyone else is doing.

GKE on-prem doesn't have to use VMWare or a hypervisor. Anthos also supports bare metal:

https://cloud.google.com/anthos/clusters/docs/bare-metal/1.6

yes im aware of bmctl. however you dont have control over the kubernetes distro that's running.

is there anything similar to https://github.com/aws/eks-distro ?

I'm not understanding the question. EKS Anywhere installs EKS-D on-prem and Anthos installs whatever the GKE distro that Google maintains onto your on-prem hardware no? I didn't think you could bring you own distro with either provider. I thought the whole point was to have somewhat unified environment between both sides of a hybrid cloud.

its not evident what is the distro that GKE installs. is it even open source ? i havent been able to find it. Happy to be corrected here.

Reply on news.ycombinator.com