Show HN: Dockerfiler: declarative management of images built from Dockerfiles | Hacker News Reader

Show HN: Dockerfiler: declarative management of images built from Dockerfiles

github.com

35 pointsjbergknoff49 comments

Hi, I'm the author. Thanks for taking a look. This is basically a tool to help manage a "Dockerfile" repo (along the lines of https://github.com/jessfraz/dockerfiles), where you build any tools you want into images that you control. This can be really useful for personal use or within a company.

Why build tools into Docker images? Love it or hate it, there are many senses in which Docker is currently the best medium that we have for distributing and running dev tools. Here's an article making that argument: https://jonathan.bergknoff.com/journal/run-more-stuff-in-doc....

this description is a lot clearer to me than the one from readme :)

Thanks for that feedback! I'll update the readme.

We've made all this so complicated. Instead of the beautiful simplicity based on UNIX philosophy (arguably), modern devops feels like a mishmash of ideas plopped together, tied with a bunch of yaml tape and god forbid if you ever want to look inside the stack of mess. Containerization is a great idea but flawed in its interface.

I like what Jim Keller (chip architect) says about complexity - that we need to throw away everything and start from scratch to which the interviewer asks, how often? Jim responds that currently chip architectures are re-done every 10 years but it should be more like every 5 years[1].

Just like any evolutionary process, there comes a point of diminishing returns because mistakes made cannot be corrected due to many other things that get piled up on top of it. So, it is difficult to track back. What happens is more shit gets piled up on top just to patch up old mistakes. Like our laryngeal nerve that loops around from the brain, all the way to the cervical area and goes back to the voicebox[2]. It is even more evident in a Giraffe. A good architect wouldn't design anatomy like this. The reason why it is the way it is, is because evolution has no hindsight and marginal cost of undoing the nerve is higher than just slightly increasing the length of the nerve. This is what we do in software. A good architect wouldn't design software like this. Sorry for the diversion, but I just feel so much pain with Docker, Kubernetes, Terraform and a whole load of AWS complexity. Holyshit.

[1] https://www.youtube.com/watch?v=1CSeY10zbqo

[2] https://en.wikipedia.org/wiki/Recurrent_laryngeal_nerve#/med...

> Instead of the beautiful simplicity based on UNIX philosophy, modern devops feels like a mishmash of ideas plopped together, tied with a bunch of yaml tape

So... it's simpler tools tied together? Are you arguing that simple tools must instead stay apart? Is the developer supposed to build their thing invoking each basic tool one-by-one? Maybe makefiles and shell scripts should also be verboten, then?

> we need to throw away everything and start from scratch

Wait a minute, this part is even funnier. So Docker allows you to build your quasi-VMs using all those Unix tools, by talking to Linux that's been put inside your Linux. But apparently this is not what you meant by ‘Unix philosophy’ and it all should be thrown away. Huh.

Sure it seems complex, but we're solving insanely complex problems that didnt even remotely exist during the halcyon days of "beautiful simplicity" of the ol' Unix philosophy. That philosophy was borne out of tools to process text files. It's closest proxy in the ops world might be microservices, but if you compare the amount of labor it used to take to set up a datacenter in even something so recent at the first dotcom era, its insane what we can accomplish with a cloudformation yaml file or a terraform configuration file in a similar amount of time. The reproducability, the change management, all of it is light years better than it used to be. Every one of these 'yaml messes' all evolved out of drastically shittier shellscripts, duct tape, hope, and weekend pagers blowing up.

It might seem complex, and obtuse, but its turtles all the way down, and everything all the way back to to the original von Neumann architecture could definitely be looked at again and redesigned, but otoh, we're still doing weird shit like dividing our day into 24 hours and 60 minutes and 60 seconds for no other reason than 'its always been that way' and if we had a chance to actually design a sensible time system we could eliminate a lot of the complexity that has piled up on top of that system.

But yeah, no way would I ever want to go back to the way things were before all these layers existed.

There were just as many tools that existed during those "halcyon" days that didn't adhere to the UNIX philosophy. They just didn't tend to survive.

The one thing that hasn't changed is commercial (& ego) pressure to make the "one tool to rule them all", which has been pretty much immutable throughout the years, while on the other hand there is the reality of trying to string all of these things together.

Small, self contained tools with good APIs have persisted and will stand the test of time while the monoliths always die eventually (maybe with the exception of excel, which has really managed to cling on).

I don't think kubernetes or docker really have a bright future because they both have an ambition to be "the one tool to rule them all" and they simply can't.

> we need to throw away everything and start from scratch

This has already happened back in the 1990s with Plan 9 from Bell Labs[1] and Inferno[2], but it didn't take due to various factors. With Docker and Kubernetes we are slowly wrangling the current Linux ecosystem to be more akin to Plan 9 environment with private namespaces and networked environment, with "compute" and "storage" servers. But the issues these tools are trying to fix and improve are buried deep in the stack, at the kernel level - fixing them would mean throwing away years of work put into Linux and starting almost from scratch.

Plan 9 was definitely ahead of its time. The original authors learned from their UNIX choices back in the 1970s and looked into the future with a fresh set of eyes. Private process namespaces, lack of a superuser account with full privileges, transparent networking, split of the monolithic system architecture into CPU (compute), fileserver (storage) and terminal with CPU and storage being in the data center on a fast network and terminals being out there used by the users... I would say that this is pretty spot on with 2020s world where cloud and data centers are used almost all the time and end users rely on lightweight (both in a "weight" and in a "computing power" sense) devices to engage with the system at large.

Of course there were missteps as well, nobody's perfect. A byzantine graphical user interface with overfocus on the mouse (today's world overwhelmingly uses touch-based interface instead) probably hinders the newcomers the most. But apart from that, its hard to pick anyhing else that is wrong with the platform in the current world, and most of the things offered at the operating system level are very attractive in today's environment where user applications need to be protected from each other.

Today Plan 9 still lives on[3] and waits for its time to shine. If enough people are fed up with Kubernetes, I would hope that they could see the future elsewhere. There are definitely issues with current Plan 9 ecosystem and its forks that prevent many people to consider it for work, like its graphical interface stuck in the 1990s era when the rest of the computing world evolved and improved. But this can change, new graphical interface can be implemented to scratch someone's itch - that's how most of the current Linux server and desktop environments came to be. So I would say, look at what the OS provides, trim the bad parts away if you don't like them and start working in a fresh environment with good ideas.

[1]: https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs [2]: https://en.wikipedia.org/wiki/Inferno_(operating_system) [3]: https://youtube.com/watch?v=6m3GuoaxRNM

I am growing my anxiety as more layers to the Docker onion grow faster than I can stay current. In the past 2 years my company adopted kubernetes to manage our already confusing Docker infrastructure design, and then they added Rancher to it because no one was making sense of kubernetes.

Meanwhile we're shipping on five year old containers that everyone is afraid to update and no one remembers how to build them. We're building a skyscraper on a floating pier and trying to reach the moon before the tide changes.

I know this is the way, but I am having a hard time because it just doesn't make sense. This doesn't feel like process. It feels like compounding of reactions. Is it just me? My company? Or is this a general feeling?

> shipping on five year old containers that everyone is afraid to update and no one remembers how to build them.

That's bad. This is not the way to do it. And honestly, this isn't the fault of Docker, Kubernetes, Rancher, or any tech. Sounds like the org is broken...

I second this. This can also be applied to a jar/script that somebody built 5 years ago and nobody knows how to update it.

If you can't rebuild the docker containers... I think you're doing things wrong...

I always start from an official container base from a reputable source (Debian, Apache, nginx, Alpine)... Or I branch off one of my creations that is based on these... If I want to use someone else's work from an untrusted source, I make my own image and build pipeline for it so I'm in control.

This is my philosophy... I don't have any containers I'm afraid to rebuild... But I don't use kubernetes or rancher, just raw docker, docker compose, and ansible/terraform

I like and align with your simple approach: Docker Compose, Ansible and Terraform.

K8s is too complicated for my competence and needs.

you are not alone.

i am growing very tired and weary of all these abstractions. i use them because, well, i want people i work for and esteem to succeed. but all this feels like non-sense from engineers at these big techs trying to justify what they did for their performance review.

folks, we are not actually solving much!

i feel like we should start over at this point with everything we learned. but that's just too much to ask because many have invested so much in all this already.

Your company, certainly.

I've been to a number of places, big and small, and the ability to maintain and build every part of prod has been seen as important everywhere.

I'm pretty picky, though; one of the questions I ask at interviews is whether the potential employer does deployments on Fridays.

Well, don’t leave us hanging. When do you if the answer is yes or no to a Friday deploy?

If they say that they do deploy on Fridays, I ask whether they do multiple deploys a day as a routine. Some SRE orgs have their gears oiled so well that deploying stuff is trivial, and rolling stuff back is also trivial (and may even be automatic). If so, I'm glad and continue with the interviewing process.

If the answer is "yes, when the deadlines are tight", then I cancel further interviews. This means inevitable and regular overtime on weekends (because deadlines are always tight), and I need my weekends for my family.

And if the answer is No ?

That would mean that their deploy/release procedure is so brittle that they have frequent outages/problems or expect deploys/releases to introduce issues, so they avoid deploying on Fridays so no one has to work during the weekend.

You aren't alone. But if it wasn't Docker, it'd be VMs, unreproducable golden images, and whatever VM orchestration you use. If it wasn't VMs, it would be bare metal, and piles of unmaintainable post provision scripts.

You can always do it wrong, it's not always the tech. Your description sounds like frustration mixed with some amount of wrongness. But I do agree that after being in the industry for about 15 years, there seems to be a diminishing returns of sorts of complexity to net org gain with the current "meta".

Always remember, KISS.

I don't consider myself a docker fanboy, but between bare metal, VMs, and docker... I think docker works better than most solutions thus far primarily because developers can run and test their code locally... But as you move more towards complicated solutions, docker becomes more than a 1-3 day training session, if it can even be tested locally anymore...

Despite dockerfile being essentially shell scripts on a clean basic install... It seems more approachable to developers than ssh and working with real servers.

> I think docker works better than most solutions thus far primarily because developers can run and test their code locally

That can also be done with a locally run VM on the dev machine and provisioning it with config management like chef or ansible.

Indeed, and the meta at the time was Vagrant on top of Virtualbox!

> It seems more approachable to developers than ssh and working with real servers.

I think the problem with "ssh'ing into real servers" is reproducibility, right?

I say this as someone who does not use Docker and does not understand it and is scared of it, heh.

But the reason I think I probably should/will have to eventually learn it and use it, is I'm not sure there's a better solution to reproducibility, meaning basically "infrastructure as code", which really means "the state of the infrastructure as code".

Ansible/chef/etc are not great solutions. I haven't used Docker much yet, but my impression is that this is what leads people to Docker, liking it better than these alternatives specifically for reproducibility/"infrastructure as code"?

I think that's exactly right.

Yeah, you nailed it. Definitely take the dive yourself. Docker has made it so that we can run any of our tens of services locally with nothing more than a git clone and make up. It’s a great feeling!

You can "ssh" into a docker image and then commit it. Than you can ship the binaries __or__ you can write down all the steps and ship the source (Dockerfile).

So it's something like a mix out of infrastructure as code, vms and baremetal. Something like vagrant but with lesser overhead.

The technology industry is broken in every level... we layer more and more abstraction to try to fix issues with previous technology...

It's your company. I've seen it a lot. They want to go whole-hog on some new technology, but they never learned how. So then they adopted more new technology to solve not knowing how to use the previous technology.

Maybe I am stupid, but is there a problem in Docker build design? By default you should get both a Docker image and an artifacts.zip of binaries that it downloaded so you can pin them? The cache isn't just for speed. It allows for reproducible builds.

Also, Docker should have a reproducible flag that creates binary reproducible images. Timestamps and local system info would have to be nerfed to some default or fed in statically with a config file.

You can already make reproducible images of e.g. NixOS. I don't know if there's a good way to tackle the problem at the docker level without sacrificing container OS agnosticism.

My OS is a docker image, booted bare metal.

https://godarch.com/

Perhaps mount an external directory and extract all artifacts to it as a zipfile at the end of the build.

https://access.redhat.com/documentation/en-us/red_hat_enterp...

That's like losing the source code and blaming the language ?

> no one remembers how to build them

For a long time I didn't know about:

  docker history

See also Nix, which can declaratively generate Dockerfiles (example on home page) :-)

https://nixos.org

> declaratively generate Dockerfiles

This is inaccurate, assuming you are referring to the 'dockerTools.buildLayeredImage' function in the nixpkgs repository.

That builds docker images declaratively, not dockerfiles.

So this is for me to easily manage all my linters, formatters and such in one image from a group of upstream, published images, right?

Looks like it. When developer tooling causes conflicts to install globally and tricky to configure with the project (say phpunit) then this can all be taken care of by building it within a docker container. However if the linting also dependent on a specific version of a programming language to test its code now you are no longer testing the production environment.

From the article:

> Docker is an excellent means of distributing those sorts of tools.

No, package managers are excellent means of distributing and managing installed tools. Docker is an excellent way to package the tools, but it's distribution and management are terrible. There isn't even a command to simply show which containers are considered "outdated" without having to repull all your images, which can take 10+ minutes on 30 images.

I guess we are fixing our lack of compatibility story by encasing our tools in fixed environments. Then write new tooling when this is no longer usable.

That seems like a short term view of things.

Pretty sure the name is ungoogleable.

That is what I use Skaffold for

I think I may be a bit slow -- I don't get it =/

You write declarative manifests that include source repos/tags and destination repos/tags and build arguments, and this thing figures out what commands are needed to get your container registry in the desired state. It outputs those commands. You'd then pass then to an environment that has access rights to make the necessary changes.

Ohhhh.

So the example in the repo being, you have some set of tools which you may want to run in CI/CD, or maybe every developer to have the same version of (linters, deployment tools, test runners, etc) and this will figure out the optimal way of pulling them so they're available?

Thanks, makes sense now I think.

Yep! Not only pulling them, but also building them and pushing them if needed.

Reply on news.ycombinator.com