Hacker News Reader

Comment by Avalaxy | Hacker News Reader

> all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platform of choice

I don't agree. I spun up a Kubernetes cluster in Azure, which was indeed easy. But then I had to figure out how to write the correct deployment scripts to deploy my docker containers to it, and how to configure all the security stuff. After more than a week of trying to figure it out, I decided to ditch the whole solution and go for Azure Container Instances instead. It was too much for me to learn about all the concepts of Kubernetes, how you configure them, how to make it work for solutions that are not as simple as the example on the website, and how to navigate through the various different methods of deploying stuff.

Maybe I'm just too dumb. But I wasn't going to invest a month of my time into doing something that should be simple enough for an average developer to accomplish.

View on HN

You're not dumb, you're just new to it, and it's fundamentally hard stuff anyways, and if you can find a higher-level abstraction that lets you get work done faster, then all the better. However, the question is comparing Kubernetes to traditional VM-based infrastructure (especially with pet nodes) whereas you're comparing Kubernetes to a higher-level abstraction.

For what it's worth, deploying in Kubernetes is pretty easy once you figure it out (and often finding the information is the hardest part). All you need to do is update the Deployment resource's "image" parameter. You can do that with `kubectl patch` like so:

    kubectl patch deployment foo -p '{"spec":{"containers":[{"name":"main","image":"new-image"}]}}'

Kubernetes will handle bringing up a new replicaset, validating health checks, draining the old replicaset, etc.

Please, do not manage deployments with imperative kubectl commands, I beg of you.

I probably wouldn't do this, but what problem does this cause?

The same problems any imperative management within declarative config causes -- drift. If the tool you're using supports declarative configuration, all changes should be made exclusively via the declarative interface to prevent that drift. In this example, the new image should be added to the original manifest itself, not via a CLI update.

It depends on how you manage your changes. A lot of people don't have their infra-as-code manage the deployment's image field--rather, that's updated by the application's CD pipeline. There's no drift to worry about.

So upon deploy the CD pipeline calls kubectl with the proper deployment image and that's ok?

Yes.

Drift, as the other child comment mentions, but also loss of version control. I do not want to have to trawl through someone's shell history to figure out what they changed, nor do I want to have to redirect `kubectl get foo -o yaml` output into diff.

If everything is in code, and you have a reasonable branching strategy, it's much easier to control change, to rollback bad merges, to run pre-hooks like security checks and configuration validation tools, etc.

You know kubectl has a built in diff subcommand?

I do now! Thank you.

It's not drift, because the infra-as-code doesn't manage the image field (the application's CD pipeline does). You don't trawl through someone's shell history', you look at the CD pipeline history. Rollbacks are easy--you just deploy the prior version via your CD tool.

I think you're assuming that invoking kubectl means invoking it directly from a user's command line, but kubectl can also be called in a CD script.

If you're using ArgoCD or something then sure, but bear in mind the original statement you made was directed at someone who is new to K8s, and given a command that can be executed from their shell, they would likely assume that's what you meant.

It doesn’t have to be Argo, it can be Jenkins. Whether or not you use a CD pipeline is orthogonal to whether or not you use k8s. The best practice is to use a pipeline whether you’re targeting k8s or bare VMs or a higher level PaaS abstraction.

I'm not new to Kubernetes and I've been using containers since Solaris zones were introduced.

The declarative approach is a more sustainable way to run Kubernetes. If you define some desired state in manifests and apply them to a cluster, they can be applied again to new clusters or the same one and Kubernetes will attempt to maintain the desired state.

This state can be version controlled, written in stone, whatever you prefer and it can always be attained.

When administrators start issuing imperative commands to a cluster, state starts being changed and there is no record[0] of the state Kubernetes is being asked to maintain.

[0] Not entirely true, the state can always be retrieved from the cluster so long as it hasn't failed.

I find this to be a common misconception, stemming from a misunderstanding of what "declarative" means (especially common when people are discussing tools like Terraform).

Firstly as you point out, there is a record of the state Kubernetes is being asked to maintain: it's in the API server as the spec of each resource.

Secondly, using `kubectl` "patch" in the manner described is not making changes to the cluster state directly, it's making changes to the specification of what should be maintained, and the various controllers effect the state changes.

Fundamentally, the argument seems to come down to "you don't have a record of what you once asked the API server to do", and that's fair enough - you don't. But that has nothing to do with imperative or declarative models.

I'm not advocating actually doing this on a day-to-day basis, but the arguments against it are not ones of imperative vs declarative.

Given that Kubernetes' docs[0] discuss using imperative commands, I think it's a fairly reasonable way to describe it.

[0] https://kubernetes.io/docs/tasks/manage-kubernetes-objects/i...

Why is applying a full manifest “declarative” but applying a patch is “imperative”? That’s clearly an error.

Then open a PR.

This is a strange way to concede an argument, but I’ll take it.

Applying a full manifest is "this is your state now".

Applying a patch is "make these changes to your existing state".

That dependency on existing state is a difference, and it seems to map reasonably well to what declarative/imperative seem to usually be used to mean in this context.

1. A strategic merge patch says "this is your state now" (the only difference is the scope of the state in question, with a full manifest including a bunch of extraneous stuff)

2. "make these changes to your existing state" is still declarative

> Applying a full manifest is "this is your state now".

It isn’t though. It’s “please make the state look like this eventually”. You do not patch state, just spec. Controllers effect the changes.

You're both correct, but you're talking about different states. The parent's "state" is the state of the etcd database and your state is the actual state of the resources that the controllers are managing.

That said, the parent is wrong that a full manifest is declaring the (etcd) state in a way that a (strategic merge) patch isn't--both are declaring etcd state, but a strategic merge patch is doing so in finer-grained increments. A strategic merge patch can declare zero state or many full manifests, while applying full manifests can only work in increments of complete resource manifests. But both are telling Kubernetes "this is your (etcd) state now".

> Fundamentally, the argument seems to come down to "you don't have a record of what you once asked the API server to do", and that's fair enough - you don't. But that has nothing to do with imperative or declarative models.

You do actually have this record. First of all, because k8s has an audit log, and secondly because deployments maintain a revision history (so you can always rollback--kubectl even supports this via `kubectl rollback undo`).

You're correct, but, when approaching this from the human perspective, it's less about the technology and the reality and more about "do I have a record of the last state I asked the cluster to be in before I fucked it up?". Ultimately that's what matters. Auditability is great, but in practice, when the shit hits the fan, can I get the application that brings in our company's money, and our customers rely on to breath, back to the state it was after 5 years and 73 updates?

Updating the manifests, pushing them to version control and having CD deploy them encourages humans to "do the right thing".

It would be quite easy though to just tweak that one environment variable though while I patch in the new image version to update that one service; until the entire cluster dies and I can't retrieve the last definition and need an equivalent cluster up within the hour.

This is really more about the practice of writing down the state you want the cluster to be in (the spec) and showing it to the cluster, than just ordering it to do one thing without context.

In this sense, Declarative Vs Imperative is just a proxy term for "do I have a record of the state I asked the cluster to keep?"

People are incorrectly assuming that using kubectl implies invoking it from an administrator's laptop. Of course, you can and should invoke kubectl from your CD pipeline. The CD pipeline maintains its own record of runs, and Kubernetes deployments have a revision history.

Moreover, people in this thread also don't know what "declarative" means. The patch is declarative, and "declarative" doesn't provide the claimed benefits. For example, as with the patch command, I can create, apply, and delete a Kubernetes manifest file (indeed, I can apply directly from stdin!) and there is no additional record of the change beyond that which would exist if I had just applied the patch.

A patch is declarative in that it's an idempotent command which requires you to fully define the patch. It's still something that is being manually defined rather than contained in a file to be applied. Of course, if that file is just being locally held and modified on someone's laptop instead of committed to version control, it's a moot point.

I will grant you that `kubectl delete -f` breaks my argument, since it's imperatively modifying objects given a declarative manifest. Re: changes via stdin, I mean sure; you can also pipe a deployment manifest into `ex` or `sed` to make changes on the fly and then echo it back to `kubectl apply -` but I wouldn't recommend it.

Yeah, my point is that whether you're changing a manifest and doing an apply versus a patch isn't relevant, the relevant bit is whether or not you have a version history and in both cases you do via your CD pipeline and via k8s deployment revision history. You can also commit to git, but I don't think there's much value in committing every image version change to git.

Also, working in raw manifests isn't a panacea; if you delete a resource's manifest, Kubernetes won't delete its corresponding resource (maybe this is what you were referring to with your bit about `kubectl delete -f`). You need something that diffs the manifests against the current state of the world and applies the requisite patches (something like Terraform) but this isn't related to how you change image versions.

I agree with your points, but as I mentioned in another comment, if we take "imperative" and "declarative" as proxy terms for "I do or do not have a reference to the state I requested of the cluster OUTSIDE of the cluster"; my point is that updating the state, in full, by modifying the manifests and committing them to be deployed by your CD pipeline or otherwise is a better approach to ensuring you or someone else can rebuild your "empire" were it to unceremoniously burn to the ground.

I think you’re assuming that just because we’re updating the image via kubectl (which itself is invoked via CD pipeline) that the infrastructure isn’t codified and persisted, which isn’t the case. You can (and should) still have your infrastructure manifests saved in your infra repo/directory and version controlled; that’s orthogonal to how you update the current image version.

Spot on!

This is a declarative kubectl command.

Kubernetes' docs[0] disagree.

[0] https://kubernetes.io/docs/tasks/manage-kubernetes-objects/i...

They’re not being precise with their language. Applying a full manifest is no more “declarative” than applying a patch.

The question is: what tool to use? I'm a solo developer running a very small kube cluster for my hobby project. I very much wanted to have a declarative version controlled state of my cluster. Every time I try googling solutions I get flooded with some enterprise Saas offerings that do nothing I want.

I managed my stateful sets/services for a while with terraform, but my experience was absolutely terrible and I have stopped that eventually. I now use "kubectl patch" and "kubectl apply" with handwritten yaml, but the workflow feels very clunky.

Intuitively it seems obvious to me that there must be a tool helping with that, but for some reason I was absolutely not able to find anything that would be even a little bit helpful. I am considering writing a couple python scripts that will automate it.

Fluxcd, or argo CD if you want a nice UI

I'm a fan of emrichen, but simple text templating (Jinja) works almost as well. (j2cli means you can just provide a yaml file with per-stage configs.)

kubectl apply on a directory full of yaml works fairly well for small stuff. Check it into git and it's version controlled.

If you need something more flexible than that, try making your own helm chart. Helm will give you some text templating, pre and post hooks, some basic functions, and some versioning and rollback functionality.

You can start simple by just pasting in your existing k8s yaml, and then just pull out the pieces you need into variables in your values file. If you need to change an image version, then you just update the variable and `helm upgrade mychart ./mychart`

kubectl apply on a directory doesn’t work because deleting a resource manifest won’t delete the corresponding resources.

'kubectl delete -f dir/' will delete all resources in the directory.

Right, but the only resource you want to delete is no longer in that directory, so you’ve now deleted every resource you except the one you actually wanted to delete. :)

Ah, I slightly misunderstood. Still, you should always have your manifests under version control so this shouldn't ever be a problem :)

It does work, as long as you’re not deleting anything. :) That might be good enough for a “very small kube cluster for my hobby project”

I only use ‘kubectl apply’ for small stuff where I only have a couple resources. Anything more complicated and a tool like helm is much more useful.

As an outsider, that command looks really easy to mess up.

* shell interactions with quotes * double quotes * interpolating into image namespace with no autocomplete * easy to forget an argument * do you get autocomplete against the deployment name?

Comparison: C# declaration of a complex type - it's less complex than the `kubectl` command above, but IDEs offer way more support to get it right.

* var x = new List<Dictionary<CustomType,int?>>()

This will light up warnings if you get anything wrong;

You get: * go to definition of `CustomType` * autocomplete on classnames * highlighting on type mismatches * warnings about unused vars. * if you initialize the var, the IDE will try hard to not let you do it wrong

So structurally,

1) in the code realm, for doing minor work we have really strong guarantees,

2) in deployment realm, even though it's likely going to hit customers harder, the guarantees are weaker.

I think this is behind the feeling that the k8s system is not ready yet.

1. You’ll use `kubectl set image deployment/gitlab gitlab=gitlab/gitlab-ce:14.9.2-ce.0` in production instead

2. kubernetes will abort the change if it doesn’t syntax check or the types don’t match.

Hah, I didn't even know about `set image`.

You put it in a bash script that gets called from CI and move on with life. No one is typing this into their terminal every time they want to do a deploy.

> I think this is behind the feeling that the k8s system is not ready yet.

Whether you patch the deployment image from bash or from C# doesn't indict the k8s ecosystem.

About quoting rules: We desperately needs static analysis or IDEs (Emacs plugins, whatever) that "explain" exactly what quotes do in a particular context. When I returned to C++ recently, I was blown away by Clang-Tidy (JetBrains CLion integration). Literally: Clang-Tidy seems to "know what you really want" and give intelligent suggestions. For someone whom is a very average C++ programmer, it instantly leveled me up! I could see the same for someone writing /bin/sh, /bin/ksh, /bin/bash shell commands. If the IDE could give some hints, it would be incredibly helpful to many.

The quotes here have nothing to do with k8s, it's just how you encode JSON in shell.

apt-get install shellcheck

https://www.shellcheck.net/

correct! Here's why: Kubernetes is an operating system (a workload/IO manager) and its lineage is 16 years old. So it's like Unix around 1979, or Windows/DOS around ~1996. Imagine it's 1996 and you can pick between Windows NT (representing 16 years of MS development) or System 360 (~40 years out from IBMs first transistorized designs).

What I'm trying to say is that as an operating system, Kubernetes is now a young adult, and historically speaking, operating systems at this level of maturity have been adopted and ridden for decades, with much success. But, ya know, if you chose OS/360 in 1996 you would have a point.

You wrote: <<if you can find a higher-level abstraction>>

This a stretch, but to me Kubernetes is like the C programming language for infra. If you look at the entire software stack today, drill down (all the turtles), and you will eventually find C (everything goes back to libc or a C/C++ kernel). I assume any commercial (or non-!) "higher-level abstraction" for infra is already (or will be soon) built on top of Kubernetes. I am OK with it.

I write this post as someone who is uber-techincal, but I know nothing about actually using Kubernetes. I can do vanilla "hand-coded/snowflake" infra just fine in my constrained, private cloud environments, but nothing that scales like Kubernetes.

Kubernetes may eventually underpin lots of higher level infrastructure, but most stuff currently isn't running atop Kubernetes. None of the higher level container-as-a-service offerings by the major cloud providers run on Kubernetes, for example. Nor does Heroku. And moreover a lot of people are still working with lower-level abstractions (VMs and auto-scaling groups) or no abstractions at all (pet servers).

> If you look at the entire software stack today, drill down (all the turtles), and you will eventually find C (everything goes back to libc or a C/C++ kernel).

I might be nitpicking, but I'm not sure that's necessarily true. You could in theory write a compiler for a new language in C (or even assembly!), and once you have a working language, re-write the compiler in that new language. Now that there is no C code involved in the stack anymore, would that still count as a C-turtle?

Haskell for example, has some "bits" written in C, but a lot is written in Haskell or assembly[1]. So if you look at the WHOLE stack you'll find C _somewhere_ (probably most places), but I don't think _everything_ boils down to C.

Granted, a LOT of stuff is written on top of C.

[1] https://stackoverflow.com/questions/23912375/what-language-i...

Great point and not a nitpick at all! I hope the next generation of languages is written on top Python, Ruby, C++ (ultra modern), Java, or DotNet/C#. I wish more languages would figure out if they can host in JVM/CLR, which would provide some crazy interactions!

After seeing so much great work done with JavaScript in the form of "transpilers", I think a lot can be done in that area. I feel Zig is a crazy good idea: A brand new language that produces binaries that are 100% compatible with C linkers. If all goes well, in a few years, why would anyone use C over Zig? It seems like the future.

Lots of people think C++ is bat sh-t crazy complex (me too, but I still like the mental gymnastics!). What if there were different C++ dialects supported by transpilers that intentionally restricted features? I think kernel and embedded programmers would the primary audience.

Don’t do that, use

    kubectl set image deployment/foo main=main:new-image

instead

TIL. Thanks for the advice.

It also accepts "*" if you don't know or care about the container name(s)

    kubectl set image deployment/foo "*=the/new/image"

and then, my other favorite (albeit horribly named, IMHO) "rollout status" to watch it progress:

    kubectl rollout status deployment/foo

It took me 4-5 hours to read and understand first few chapters of Kubernetes in action, I annotated a lot text as notes on the Oreily site, but didn't have to go back to it. I didn't have to do any security and account management stuff, it was being done by someone else. After 3 years of working with plain Kubernetes and OpenShift, I still didn't have to go back to the book. Basic concepts in Kubernetes are easy to understand, if you're working as a developer to deploy your apps in it and not in DevOps managing it.

https://www.oreilly.com/library/view/kubernetes-in-action/97...

I hear you about feeling dumb. I think some early decisions in the k8s ecosystem led to a lot of wasted time and effort, and this frustration.

YAML: significant whitespace is always unwelcome but YAML also introduces unexpected problems, like how it deals with booleans. For example, say you have this array:

    foo:
    - x
    - y
    - z

You might think this is an array of strings, but you'd be wrong.

It's also difficult to read through a YAML config and understand the parent of each key, especially in code reviews or just on GH.

I believe k8s life would have been easier with JSON configs, where it's impossible to confuse e.g. booleans for strings and where it's easier to understand the object's hierarchy.

Helm's use of gotpl: this choice exacerbates the problems with YAML. Now you're treating a structured language with a text template library. You have to spend energy thinking about indentation levels and how the values will be interpreted by both the templater and k8s.

I think helm would be less frustrating if they chose some templating library that made objects first class citizens. Where you can inject values at specific locations with one-liners or simple blocks of code (e.g. `ingress.spec.rules[0].append(yadda yadda)`)

I'm sure there was debate about these choices early on and I don't have any unique ideas here, so I don't want to be too critical. These are just a couple of pain points I've personally experienced.

for what its worth, and if im not mostaken, there is some support for JSON, ie. kubectl get deployment xyz -o json

JSON has a different set of problems, e.g. not having multi-line string support.

Is that yaml not an array of strings?

["x","y","z"]

I don't happen to have a YAML parser at hand, but I believe it's...

["x", True, "z"]

It's not, y is treated as boolean true.

Is that a k8s thing? Because when I load it in this page[1] it comes out as a string...

[1] https://yaml-online-parser.appspot.com/?yaml=foo%3A+%0A-+x%0...

In YAML 1.2 they fixed it (and also the Norway-problem of "no" meaning false) but many places are still using an older version.

Technically YAML is a superset of JSON - all valid JSON is valid YAML. So you could write all your configs in JSON and they'd work just fine.

This discussion from a couple of weeks ago suggests it isn't that simple: not all valid JSON is always valid YAML if your definition of superset requires that JSON would be parsed the same with a YAML parser as with a JSON parser.

If you're going to use JSON for config, it's better to use an actual JSON parser.

https://john-millikin.com/json-is-not-a-yaml-subset https://news.ycombinator.com/item?id=31406473

Thanks! That's really interesting.

You could mitigate some of the issues and get JSONs "strictness as a feature" by passing the document through e.g. `jq . $file` as a CI step, but I don't think that'd resolve the 1e2 issue. TBH I didn't know you could write numbers in JSON like that, so I imagine it'd be an issue that doesn't come up often. But it's disappointing that it wouldn't just work.

> Technically YAML is a superset of JSON

That's false. http://p3rl.org/JSON::XS#JSON-and-YAML

I'd like to amend my statement:

Technically YAML is (supposed to be) a superset of JSON - (almost) all valid JSON is valid YAML. So you could write all your configs in JSON and they'd (probably) work just fine (assuming you keep things relatively simple).

Similarly, we switched from self-managed k8s on EC2 to Fargate. Took about 2 months part-time with some consultants to get thing squared away.

Once we deployed, we ran into all sorts of SRE-issues. Turns out AWS sets all those “sane limits” that our own folks never did. Still hunting ghosts from the rollout 6 months ago.

Makes for good resume fodder, and makes me laugh at the prestigious titles and positions the folks who built this system went on to receive at big name firms.

Guess it is someone else’s management problem now. :shrug:

Azure is generally the worst (their new container offering looks better though)

If you try digital ocean k8s offering it's fairly straightforward.

Google cloud was the first to offer a decent k8s as a service, if I recall correctly. We didn't have DevOps back in 2015 and we were on GCloud.

Personally I don't pick k8s just because it's heavy to run and I don't want to waste machines (plain docker is good enough for a large part of what people actually need). Sometimes in a project when I can't figure something out with just docker, I just bite the bullet and install k0s.

And you probably didn't even get to TLS and or authenticated communication between containers, CI/CD, canary deployments, observability, monitoring etc.

What we need is a Next.js for Kubernetes. Something that delivers a full stack solution on top of base Kubernetes.

The core system is great, but we need to replace these DevOps with a framework or platform.

> What we need is a Next.js for Kubernetes. Something that delivers a full stack solution on top of base Kubernetes.

Doesn't Rancher fit this description? It's pretty resource-heavy though.

Rancher seems to now be a "multi-cloud container management platform".

- Deploys kubernetes clusters - includes Fleet for CI/CD - install apps with helm - ISTIO (awesome)

It feels like it is positioned for orgs with large needs. I'm looking for k8s for small nimble orgs.

I will look at Rancher more, thanks for reminding me of it!

Google's Anthos is also hard to describe now as they cover similar "everything" product features.

GCP now has Autopilot which lets you pay just for the cpu you use (no cluster management at all).

Anthos includes ISTIO which may at some point work on Autopilot. This would mean not having to fiddle with GKE Ingress (which I found unpleasant) and instead use the new standard Gateway.

I believe that eventually GKE Autopilot will offer running individual pods on GPU/TPU, pay as you go.

But when is this all as easy as using Next.js ?

I feel you. Took me 2 years of dabbling with k8s every once in a while on the side to finally "get" it. and if I stop looking for a few months, suddenly it already gained another set of features and deprecating some other.

Definitely a pain point I've seen in my past work as well. Even though spinning up a basic cluster has gotten easier with canned services like EKS, deploying and developing on the cluster is a major challenging for most developers without a higher level framework.

My cofounders and I are working on a solution to this at https://www.jetpack.io/. If you're interested in early access, we'd love your feedback!

Container instances are pretty good really, the app service for containers is pretty good too. I’ve been playing with k8s because that’s what everyone thinks they want and need to be able to speak to it, and use it when the time comes, but I’ve yet to run into a case where I really thought it was necessary, for the platform I work on (millions of users, big number of transactions per second).

Do you use ACI at that scale? It's interesting how Microsoft also promotes AKS rather than ACI. For Azure ML they even explicitly state that ACI should not be used for production purposes, which I find quite curious.

There are tools that let you convert a running docker/podman container to a kubernetes deployment manifest which you can then deploy with one line command to your cluster. Kubernetes is the new data center and I don’t expect anyone with <10 years of enterprise experience to master it. Certainly one can use it and deploy things to it if you work at a place where there are mature pipelines, but deploying your own cluster and all associated services plus networking plus security is the domain of experienced/senior engineers. How quickly one achieves that “status” is up to the individual. It took me 15 years before I could build an enterprise network from scratch. It took me two years to understand how to do the same with K8s.

Reply on news.ycombinator.com