Hacker News Reader

Comment by digitalsushi | Hacker News Reader

I am growing my anxiety as more layers to the Docker onion grow faster than I can stay current. In the past 2 years my company adopted kubernetes to manage our already confusing Docker infrastructure design, and then they added Rancher to it because no one was making sense of kubernetes.

Meanwhile we're shipping on five year old containers that everyone is afraid to update and no one remembers how to build them. We're building a skyscraper on a floating pier and trying to reach the moon before the tide changes.

I know this is the way, but I am having a hard time because it just doesn't make sense. This doesn't feel like process. It feels like compounding of reactions. Is it just me? My company? Or is this a general feeling?

View on HN

You aren't alone. But if it wasn't Docker, it'd be VMs, unreproducable golden images, and whatever VM orchestration you use. If it wasn't VMs, it would be bare metal, and piles of unmaintainable post provision scripts.

You can always do it wrong, it's not always the tech. Your description sounds like frustration mixed with some amount of wrongness. But I do agree that after being in the industry for about 15 years, there seems to be a diminishing returns of sorts of complexity to net org gain with the current "meta".

Always remember, KISS.

I don't consider myself a docker fanboy, but between bare metal, VMs, and docker... I think docker works better than most solutions thus far primarily because developers can run and test their code locally... But as you move more towards complicated solutions, docker becomes more than a 1-3 day training session, if it can even be tested locally anymore...

Despite dockerfile being essentially shell scripts on a clean basic install... It seems more approachable to developers than ssh and working with real servers.

> I think docker works better than most solutions thus far primarily because developers can run and test their code locally

That can also be done with a locally run VM on the dev machine and provisioning it with config management like chef or ansible.

Indeed, and the meta at the time was Vagrant on top of Virtualbox!

> It seems more approachable to developers than ssh and working with real servers.

I think the problem with "ssh'ing into real servers" is reproducibility, right?

I say this as someone who does not use Docker and does not understand it and is scared of it, heh.

But the reason I think I probably should/will have to eventually learn it and use it, is I'm not sure there's a better solution to reproducibility, meaning basically "infrastructure as code", which really means "the state of the infrastructure as code".

Ansible/chef/etc are not great solutions. I haven't used Docker much yet, but my impression is that this is what leads people to Docker, liking it better than these alternatives specifically for reproducibility/"infrastructure as code"?

I think that's exactly right.

Yeah, you nailed it. Definitely take the dive yourself. Docker has made it so that we can run any of our tens of services locally with nothing more than a git clone and make up. It’s a great feeling!

You can "ssh" into a docker image and then commit it. Than you can ship the binaries __or__ you can write down all the steps and ship the source (Dockerfile).

So it's something like a mix out of infrastructure as code, vms and baremetal. Something like vagrant but with lesser overhead.

If you can't rebuild the docker containers... I think you're doing things wrong...

I always start from an official container base from a reputable source (Debian, Apache, nginx, Alpine)... Or I branch off one of my creations that is based on these... If I want to use someone else's work from an untrusted source, I make my own image and build pipeline for it so I'm in control.

This is my philosophy... I don't have any containers I'm afraid to rebuild... But I don't use kubernetes or rancher, just raw docker, docker compose, and ansible/terraform

I like and align with your simple approach: Docker Compose, Ansible and Terraform.

K8s is too complicated for my competence and needs.

> shipping on five year old containers that everyone is afraid to update and no one remembers how to build them.

That's bad. This is not the way to do it. And honestly, this isn't the fault of Docker, Kubernetes, Rancher, or any tech. Sounds like the org is broken...

I second this. This can also be applied to a jar/script that somebody built 5 years ago and nobody knows how to update it.

Your company, certainly.

I've been to a number of places, big and small, and the ability to maintain and build every part of prod has been seen as important everywhere.

I'm pretty picky, though; one of the questions I ask at interviews is whether the potential employer does deployments on Fridays.

Well, don’t leave us hanging. When do you if the answer is yes or no to a Friday deploy?

If they say that they do deploy on Fridays, I ask whether they do multiple deploys a day as a routine. Some SRE orgs have their gears oiled so well that deploying stuff is trivial, and rolling stuff back is also trivial (and may even be automatic). If so, I'm glad and continue with the interviewing process.

If the answer is "yes, when the deadlines are tight", then I cancel further interviews. This means inevitable and regular overtime on weekends (because deadlines are always tight), and I need my weekends for my family.

And if the answer is No ?

That would mean that their deploy/release procedure is so brittle that they have frequent outages/problems or expect deploys/releases to introduce issues, so they avoid deploying on Fridays so no one has to work during the weekend.

you are not alone.

i am growing very tired and weary of all these abstractions. i use them because, well, i want people i work for and esteem to succeed. but all this feels like non-sense from engineers at these big techs trying to justify what they did for their performance review.

folks, we are not actually solving much!

i feel like we should start over at this point with everything we learned. but that's just too much to ask because many have invested so much in all this already.

Maybe I am stupid, but is there a problem in Docker build design? By default you should get both a Docker image and an artifacts.zip of binaries that it downloaded so you can pin them? The cache isn't just for speed. It allows for reproducible builds.

Also, Docker should have a reproducible flag that creates binary reproducible images. Timestamps and local system info would have to be nerfed to some default or fed in statically with a config file.

You can already make reproducible images of e.g. NixOS. I don't know if there's a good way to tackle the problem at the docker level without sacrificing container OS agnosticism.

My OS is a docker image, booted bare metal.

https://godarch.com/

Perhaps mount an external directory and extract all artifacts to it as a zipfile at the end of the build.

https://access.redhat.com/documentation/en-us/red_hat_enterp...

It's your company. I've seen it a lot. They want to go whole-hog on some new technology, but they never learned how. So then they adopted more new technology to solve not knowing how to use the previous technology.

> no one remembers how to build them

For a long time I didn't know about:

  docker history

That's like losing the source code and blaming the language ?

The technology industry is broken in every level... we layer more and more abstraction to try to fix issues with previous technology...

Reply on news.ycombinator.com