Docker, Kubernetes, Terraform, and AWS crash course series

To be fair. Just last night i was googling kubernetes and k3s tutorials.

All of them on page 1 of SERPS are super simple and brilliant and ONLY show you how to add nodes to clusters.

Ok now what ?? Ive drank the k-Koolaid signup for the newsletters but how do i get value from this 'thing' ???

When i code a webapp how do setup/config the db ? How do I do a regular LAMP app ?

Lol i never thought i say this but i need a kubernetes tutorial on 'best practices' for common patterns. No for google scale but how do i kubernetify my run of the mil lamp app with 100kpage views a month ??

There seems to be a gap between the beginner and advance (kubernetes all things) levels ?

Maybe that is the answer ?? The fact that the middle-situations(lamp apps with low-med traffic) tutorials dont exists means i dont need it ??

This is no doubt a useful commment, since you provide other sources. However, it's not particularly a mark of courtesy to denigrate others' contribution to the space. If you had substantive criticism of the OP offered tutorial, it might be helpful. However, you offer none.

Author here. I tried to answer your question in the first two paragraphs. But to add some context, given the nature of my work, I hear from developers on a nearly daily basis who are struggling to get started with the technologies mentioned in this blog post series, which include not only Kubernetes, but also Docker, AWS, and Terraform. In part, they are struggling because they are too scared to ask for help, and comments like yours only make that worse: you seem to be implying that the materials out there for Kubernetes are so good, that if you don't get it, there must be something wrong with you. And yet, there are thousands of devs who don't get it, so maybe for different people, there are different ways to learn?

In discussions like this, I'm a fan of what Steve Yegge wrote about blogging [1]:

> This is an important thing to keep in mind when you're blogging. Each person in your audience is on a different clock, and all of them are ahead of you in some ways and behind you in others. The point of blogging is that we all agree to share where we're at, and not poke fun at people who seem to be behind us, because they may know other things that we won't truly understand for years, if ever.

That's why I write: to share what I know, from my particular perspective. Hopefully, that's useful to some people out there. If it's not useful to you, no problem!

And for the record, I agree the Kubernetes docs are great, including those interactive tutorials: if you read the series, you'd see I actually recommend those exact docs at the end of the post [2].

[1] https://sites.google.com/site/steveyegge2/you-should-write-b... [2] https://blog.gruntwork.io/a-crash-course-on-kubernetes-a96c3...

I just don't want to do that, no thanks. I've spent so much time learning about how an OS can solve these problems and these lessons have paid dividends many times over my career. Instead, I will continue to invest in the core foundation. I'm not interested in re-learning these abstractions reading [1]-[5] tutorials plus many more and studying an unrefined, new layer of complicated abstractions built on top of the OS--the OS is difficult enough. I'll only _use_ k8s when it's managed and supported by a team of 10+ engineers, which is what it requires. Plus, it's not just k8s; you have concourse, spinnaker, artifactory, some sort of cluster templating, kops? to template your k8s deployments across DCs and environments, (I don't know what the community uses, we built our own in-house tool). It's all so gross.

My rejection of these systems has lead me to invest in FreeBSD. There was a learning curve and I'm certainly not as fluent in this system as I am with Linux but I'm in a place where I'm in control of the OS and have at my disposal solid, refined tools to help me masterfully construct the infrastructure to solve my problems. And, every time I solve a problem in this space my foundational understanding grows and these lessons will pay dividends into the future, or so I hope.

I'm utilizing FreeBSD because docker / k8s and I'll add systemd to the list, are missing and thus the community solves problems in a different way that better aligns with my values. FreeBSD isn't easier, I've been stumped on problems countless numbers of times but when I find a solution, it scales. What I mean by that is I have a better understanding of a system that can solve a larger class of problems than if I was in the docker / k8s ecosystem.

YMMV.

>> I wonder what drives someone to write a 101-level tutorial on K8s.

When I was going thru k8s tutorials several years ago, one issue was how quickly the tutorials became outdated with changes in k8s. Then you'd spend only half your effort on the the tutorial, and the other half debugging why the tutorial wasnt working.

Sadly last I checked there were relatively few tutorials for setting up on bare metal (or rather, they neglect important topics like setting up net worked volumes or other things that are very likely to be required by an application.

How does ClusterAPI compare to Helm charts (or just managing Kubernetes YAML manually or through Kustomize?)

Why do you need clusterctl / clusterawsadm? Can't you just kubectl apply once you have your .kube/config set up for your AWS k8s cluster (or Azure or GCP or whatever)?

For training purposes, Azure has cost-limited subscriptions.

Frustratingly, the other clouds have dragged their feet and largely refused to implement their own equivalents. This comes up here semi-regularly as one of the #1 complaints about the public cloud, and it's always hand-waved away by some Silicon Valley FAANG employee making half a million a year and using their parent company cloud account.

I know of several ways it would be possible for a malicious actor to deliberately bankrupt a solo dev limited to a normal credit card budget.

And as you've said, it's very fiddly and complicated to implement custom "stop" scripts, and this is futile anyway because the billing metrics in all clouds have huge delays on them.

My $0.02 is try to learn using Azure first because there's a true safety net, and then use customer accounts to continue upskilling.

Can completely relate to your situation. Basically, that’s the main reason I run everything on bare metal server, running in my basement. And use cloud providers only if my clients ready to pay for it or take risks.

There is nothing stupid about this fear at all. Having to fight a $20k mistake is highly stressful. Not being able to prevent it via policy is a design decision by the provider.

> The only downside I can tell, is the pricing, but compared to dev hours saved, it's probably a net benefit.

dev hours is already paid - i say that a hosted service for an app is gonna cost even more than the dev in the long run.

The difficult with backend is dealing with state. It's easy enough to provide a simple experience when dealing with stateless frontends, backends are a very different story.

You're right that we need better tools. I'm the founder of Encore [1] which is all about bringing the simplicity of Vercel/Netlify to backend development. Not by substituting the backend for a BaaS but by building a developer experience hand-crafted for dealing with cloud infrastructure.

[1] https://encore.dev

The downside is vendor lock in. What will you do when vendor will ban your website?

For backend, if you have a platform that allows you to run arbitrary containers, you will need to have a way to express how they are connected together, how much resource they should get, which parts should receive traffic from the Internet and so on. Any solution will tend to have about the same inherent complexity as kube yamls, this seems to be an iron law of devops.

IMHO the thing you have to trade off to get simplicity is generality. I agree that's probably the right tradeoff for 90% of apps, whether that's BaaS, heroku-alikes, language specific hosting or whatever.

Cloud providers are definitely keen on the idea that your web framework and your cloud platform could basically become the same thing.

What do you think people use to build systems like these?

A lot of medium/large shops build their own deployment and BaaS platforms on top of Kubernetes/Terraform/AWS. You just don’t hear about them as much because they often aren’t provided as a service externally, and aren’t open sourced. (I work with and on such platforms)

The mistake I see people often make is assuming it’s worth using such complex tools directly with a small project. People like the flexibility but don’t understand cost to build/maintain it.

For the average developer, it's okay not to interact with these tools. All you need to interact is with the `bin/deploy-production` script the above-average developer built.

And I'd argue that with managed kubernetes solution, such as GKE Autopilot, you get roughly the same "DX", and much more flexibility to use whatever tools you fancy. GKE Autopilot manages the cluster and node availability, ingress and certificates, all you need to do is write the kubernetes yaml. As long as what you are working on fits in a docker container, you're good to go. And you're fairly good to move off of their setup, because those same yaml configs will work on AWS or self-hosted solutions, albeit with additions that fill in gaps where Google managed services lived.

Now you have to hire not just for a specific language but a specific backend platform. Idk man.

If you can use a Platform as a Service (PaaS) offering like Vercel, Netlify, Heroku, etc, you absolutely should! I always recommend those types of tools as the first stop tools to anyone building software these days.

But there are many use cases that don't fit into those neat PaaS molds: typically, as a software company grows beyond one team, one service, one database, etc, they start to hit limitations with the PaaS solutions out there. As you scale, you often find you need more control than you can get from a SaaS: you may need more control over the hardware (e.g., for performance or cost reasons), or networking (e.g., you need service discovery or a service mesh to allow microservices to communicate with each other), or security (e.g., to meet compliance standards), or a hundred other items.

That's when many companies find themselves migrating to an Infrastructure as a Service (IaaS) provider like AWS, Infrastructure as Code (IaC) tools like Terraform, orchestration tools like Kubernetes, and so on. I'm guessing every software company with more than 50-100 developers ends up moving from PaaS to IaaS, and that's when developers need to understand how to use the tools covered in this blog post series.

Perhaps, some day, the PaaS tools out there will be good enough that you never have to migrate off of them, regardless of scale or requirements, but we're not there yet, and probably won't be there for a while longer.

Depends on what u consider avg developer, aren’t most developers employed by some company or enterprise? Who would definitely lock into big3?

God damn, I love this comment. We manage 8 Linux servers total for an application that is lightly used by our customers but heavily used by our internal workers that constantly reach out to internet services and check for things. Everything we've done has been in Go and bash, and our deployments are easy and take seconds.

I'm tired of seeing so many posts for Docker, Kubernetes, terraform, and even so many of Amazon's service offerings. Guess what? All these tools are abstractions, and it's insane to me that so many would rather learn to work with these abstractions, as opposed to learning some Linux basics, which would allow them to solve the same problem with much less overhead. We have two webservers, four workers, and two database servers. All of our code is written in Go so our dependencies are limited, and this has been my reason for not needing Docker. However, we use Docker locally to spin up our local dev environment, so developers don't need to have MySQL installed on their machines. The primary benefit I see with Docker is dependency management, and when your dependencies are minimal, it's not worth the overhead.

All of our servers are built from bash scripts that are executed on the server on the first run. This bash script creates folder structures and user accounts and sets permissions. If a server needs to be destroyed for whatever reason, it can be rebuilt as new from these scripts in < 10 minutes.

Code deployments are done using SCP and rsync and systemctl to stop and start services remotely. If we were using an interpreted language where dependencies are a bit trickier to manage in prod, certainly we'd use Docker then. Still, for the time being, this Go \ bash ecosystem works for us, is easy to manage, and limits dependencies overall. Debian & Go all the way, baby.

The longer I work in this industry, the more I realize a bigger hindrance to accomplishing goals is actually overthinking and over-engineering. I see startups with no customers using kubernetes and terraform. People building noahs ark when they've never experienced rain. The pain points in your process will become obvious. Let the process tell you it's time for these more advanced tools. If you can manage production code and deploy updates easily without them, then don't waste your time. If you don't know any other way, then maybe it's worth spending the time to learn Linux a little bit better.

> Go can handle about 500 million visitors a month on 12 year old hardware per 1 server.

Doing what? I'm absolutely no fan of Kubernetes, or overcomplexity, but this is just meaningless platitude.

The feasibility of 12,000 requests per second (assuming evenly distributed throughout a 30 day month) largely depends on what the system is _doing_ for each of those requests.

"You don't need X because I don't need X and everyone else must have exactly the same requirements and constraints as me" is one of my least favorite takes.

> You don't need Docker, just a version control system (e.g. git).

I've worked where we had to run a lot of systems that we didn't write ourselves and they came in a variety of languages, frameworks, etc. Some Python, some Go, some Ruby, some PHP, some Java, some Fortran, some C++, etc. We did that before Docker and when Docker came out it was a massive simplification for us to be able to just run standard containers where we know there's an entrypoint, some ports, environment variables, etc. rather than have to figure out how to configure all of those things for every kind of environment that we might run into. We could use the same basic build/deploy pipeline for all of them, run them on the same shared servers and know that they weren't conflicting with each other and updates to one wouldn't blow up the .so files that a different one used, etc. Docker doesn't really compete with a version control system like git; if anything, it's closer to an application package format like .deb or .rpm, but with a bit of a different focus.

> You don't need overpriced cloud offerings in the first place.

If you've done the cost analysis and the cloud offerings are cheaper than running your own datacenter, then they're not really "overpriced" and maybe you do need them. If you're just thinking "rent a server from Hetzner or a colo or run a workstation under your desk" and that those options are always cheaper than the cloud then you've clearly never worked somewhere that needed regulatory compliance (HIPAA, PCI, SOC2, etc).

> First of all, pick the right language.

If you're on a greenfield project and have that luxury, great. A lot of developers have to work with existing codebases and products though.

> Kubernetes, you do not need it. Go can handle about 500 million visitors a month on 12 year old hardware per 1 server. You will almost certainly not reach these numbers.

It must be wonderful to live in a world where all requests scale the same. What if each visitor is using your service to transcode 2GB video files? Or running massive climate modeling simulations? Or training an ML algorithm on a few TB of data? 500 million of those per month on one 12 year old server? Awesome. Sign me up.

I've done all of these things with and without tools like Docker and Kubernetes. Yes, there are workloads and environments where they'd be overkill and add more complexity than they're worth. But there's also very good reasons beyond hype that those tools became well known and widely used.

Or, you can just keep assuming that all those developers out there are just idiots compared to you. I doubt you'll change your mind, so I'll just hope I never have to work with you.

Counterpoint: there is a niche in the industry where going with containers would be easier than following the best practices without, given how likely people are to slip into doing things the lazy way, or due to how you might have limited control over the applications that you might have to run. Containers at least limit the fallout from various mistakes and nudges people towards certain architectural choices - e.g. 12 Factor Apps, so you don't cripple your app into being a "singleton app" which can only have 1 instance running, instead of just a "monolith".

It doesn't matter whether you're running Podman, Docker, Docker Compose, Docker Swarm, Hashicorp Nomad, K3s/K0s/minikube or a full K8s distro like RKE, the OCI standard is a hill that I'll die on because unless you're the person who's shipping and managing their code 100% of the time, someone else will have to figure out how to setup and run your Java/.NET/Node/Python/PHP app and moreso, what to do when one needs JDK 8, another needs JDK 11 and there are inevitable conflicts in some other system/runtime dependencies. With containers, you can limit the the impact of relatively badly written and sometimes even somewhat insecure applications, by not exposing the rest of the underlying host to them, especially when running rootless or with host remapping.

Sure, in practice it would be exceedingly nice to just deploy a single packaged executable to the server. It would be really nice to even have the choice of picking a technology that plays nicely with such simple setups, like Go, as opposed to having Java shoved down your throat, especially in combination with a separately managed Tomcat instance or something. It would also be nice to have well written applications that don't misbehave and don't trash the CPU/RAM and don't cause the server to have to OOM kill processes, and also that have proper port configuration/remapping available and also applications that have proper health checks built in, or even just well written systemd services and Ansible to manage the configuration.

But in the real world you don't always get that. So unless you like moving around different companies a lot, you need ways to manage the fallout of sub-optimal circumstances (especially legacy code that still needs to run) and containers are one way to achieve that. I wrote down some of my experiences with the subject (albeit with details changed) in my blog post "My journey from ad hoc chaos to order (a tale of legacy code, services and containers)": https://blog.kronis.dev/articles/my-journey-from-ad-hoc-chao...

That said, developing applications poorly will also mean running them poorly on Kubernetes, or not being able to set the environments up at all. Thus, I believe that it's a slippery slope with options on all ends of it being bad, be it running things directly on the server and running into the aforementioned issues, or trying to go full steam ahead with Kubernetes and failing.

The sweet spot in my experience is using containers, but in a simple setup - currently that is Docker Compose or Docker Swarm, but might become Nomad in the future. Maybe K3s in 5-10 years (single node cluster worked fine, except for RPM distros where cgroups v2 support had problems and the Traefik proxy had bad documentation for wildcard SSL/TLS certificates). Then you no longer need to even care that much about the specifics of the technologies inside of the containers either.

Edit: As an example, here's all that I (or anyone else) would need to deploy my homepage, as long as the server can run OCI images or Compose/Swarm stacks:

  version: '3.7'
  services:
    homepage:
      image: REFERENCE_TO_MY_CONTAINER_IMAGE
      restart: "unless-stopped"
      environment:
        # All of the configuration goes here, for example:
        ANALYTICS_URL: LINK_TO_MY_MATOMO_INSTANCE_GOES_HERE
        ANALYTICS_ID: SOME_IDENTIFIER_SET_UP_IN_MATOMO
      volumes:
        - SOME_DIRECTORY/homepage/app/storage:/app/storage
        - SOME_DIRECTORY/homepage/app/db/files:/app/db/files
      networks:
        - prod_ingress_network
      deploy:
        mode: replicated
        replicas: 2
        update_config:
          parallelism: 1
          delay: 10s
          monitor: 10s
        placement:
          constraints:
            - node.hostname == SERVER_HOSTNAME_GOES_HERE
        resources:
          limits:
            cpus: "0.5"
            memory: "512M"

  networks:
    prod_ingress_network:
      driver: overlay
      attachable: true
      external: true

And I no longer have to worry about systemd slices, configuration files, file permissions (for the most part), Ruby or Rails versions, or install any other stuff on the server itself directly. I can just run the image and if it doesn't work, I can just let the developers know what to fix (though in this case I am the developer myself). And if it goes rogue and eats too much resources? The runtime will cap the container without it affecting the rest of the system outside of resource limits that I've set.

Oh boy, another HN engineer who cannot imagine a technical situation other than the ones they're already familiar with telling everyone else what they don't need.

Yes, learn the basics. That way you'll know that Docker and `git` solve for almost entirely different domains, that Terraform and Ansible solve for almost entirely different domains, that "AWS" is a huge suite of services that makes no sense to compare with a single server, and that some of us do indeed work in distributed environments serving 10s or 100s (or more!) of thousands of queries per second.

Companies hiring the people probably also have a need if they are hiring and looking to grow and scale out. I made a "you don't need X / Y / Z" point because I strongly believe most people don't need these tools and really need a better understanding of Linux basics. You can also learn these tools on your own if they're mentioned in job descriptions of jobs you're applying for, but that doesn't mean you have to use them when you're the decision maker, or for a personal project.

As a Devops engineer I'm staying away from Kubernetes at home. It would require too much constant maintenance.

I use docker only.

I get to play with kubernetes enough at work.

Because then your endpoint is the standardized kubernetes rather than ECS.

Also I don't think you can use Kubernetes logic with ECS

Finally I think with ECS you have quite the vendor lock in.

You can host your kubernetes cluster anywhere. Or multiple places (as in you can host both at azure and AWS or your own data center and Google etc).

I'm not a fanboy of kubernetes, I think it makes some really trivial things stupidly complex, but avoiding vender lock in is huge. It can be massively better but oh well.

I like and use ECS, but it lacks the tooling and ecosystem around Kubernetes.