Hacker News Reader

Comment by lykr0n | Hacker News Reader

I've been tasked a couple of times trying to debug issues in 3rd party docker images. Every one of those images did something different. Placed configuration, binaries, libraries, or logs in different locations. A simple task on a standard CentOS 7 VM that would take 5 minutes turns into 25 minutes with most of that googling ("Where the hell is your working directory?!"). It's not a fault of docker per se, but it's the result (I think) of the docker "it just works!" mentality.

What's great about my examples is that two of the applications were single binary Go applications that could be downloaded and run on almost any linux box. But they were distributed with docker ... for reasons?

Docker is great when I want to run a local Redis database in one command for development. But past that- ehhhh. Each new feature seems like Docker is just digging itself deeper.

View on HN

I think this is really a problem you see in every technology that tries to make things easier. JS. MongoDB. The list goes on.

When a technology democratizes access to something that was previously only accessible to those with more specialized skillsets, it will attract people who don't know enough to know how to use it right. But, well, they're still using it. Thus, left-pad.

Its uncovering something that was always there; incompetence is rampant in software engineering. Ask the average engineer to properly set up an Ubuntu box with CICD for deployment and they'll likely fail, even with all the help of the internet. Ask them to get a docker image onto a server and they can make it work.

Wider access to technology is... well, I don't know if its a "good thing" or not. We as an industry pretty strongly believe in "fail fast"; we write tests to make sure our code works, we structure our startups to recognize and pivot around failure, and if that failure comes we stop and say "somethings fucked, stop what we're doing". So I do get scared when a technology enters the stage which allows hard failure to be delayed as long as possible. Maybe we need more technology to be harder, have opinions, and be willing to say "you're doing this wrong, just stop and go get an adult." Because the alternative... Equifax? Yahoo? Ashley Madison?

My personal design philosophy (from a System Admin, SRE, DevOps, Ops- whatever) is no black boxes. I'm not talking about knowing the inns and outs of every little component in my infrastructure/system, but instead of saying "This is Graylog" being able to say "This is Graylog- a Java app that uses MongoDB to index and Elasticsearch as a backend. We have to monitor JVM usage." The former is saying we depend on a system we have no idea how it works, and the latter is saying we depend on 2 java applications and a NoSQL database.

It horrifies me that people run stuff in production not knowing how it's configured. But, this happens and is why there is just a booming market for DBaaS and other like productions. Docker is an amazing tool, but does make somethings a bit too easy.

> Its uncovering something that was always there; incompetence is rampant in software engineering.

That's what I took away from the article. Production was setup and they don't know what kernel version it has, or what JVM version it's using, or which C library, etc. Or, now we want feature X, so we need to upgrade dependancy Y. I already did it on my local machine, but Bob from DevOps doesn't like upgrading things on QA/integration/prod without a change control...

Granted, apps have thousands of dependencies now, so keeping track of it manually is impossible. Docker came along and promised that only whatever's in this one config file is what's installed. Easy!

So it eliminated lots of the server-specific configuration management but created other problems.

I think it really comes down to developer tooling. Devs, in the absence of tooling will make their own and default to the latest stuff. If the company doesn't provide a fully mirrored developer environment, then everyone is asking for trouble.

Where I work, we have a specific team dedicated to managing this tooling.

We support Python Django, Java, and Go. You can clone the base repositories which include vagrant scripts to setup everything. You develop in that and it gets sucked up into the pipeline. You want something newer? If it works locally in our environment, the SLA says it will work in production. Doesn't work locally, then tough shit.

In personal projects I do with others, I try and do the same thing. Python? 3.6 is our version target. Java is OpenJDK 1.8.0 latest. Rust & Go are compiled with the latest versions. It might not be incompetence in software engineering as a whole, but more of an issue in defining version and software targets.

I think a lot of that can also come from people mis-using Docker as if it were a VM instead of trying to do the smallest amount of stuff possible in an image. But at the same time, there's stuff I wouldn't even try to manage in containers in production. Databases and message queues in particular.

We've had great success with Docker and Kubernetes for our stateless apps.

As a developer, one thing I absolutely love is being able to run an image without having to install anything (other than Docker). If I want to play around with something new, its easy to get started and easy to clean up.

Playing with something new should be as simple as unpacking a tarball and rm -rfing the directory when done. This should be possible in the comfort of my home directory and uid. If I need a whole separate copy of a computer's files just to try out something, then something very very fundamental is broken.

How do you install dependencies in this model? As far as can tell, you end up in situations like this

https://askubuntu.com/questions/193695/installing-packages-i...

  Playing with something new should be as simple as unpacking a tarball and rm -rfing the directory when done.

Needing a single install script before, and uninstall script after is acceptable.

Yes I agree things are fundamentally broken with these situations, and they're at the common OS design level.

What it that something depends on a different operating system and libraries that conflict with yours?

Also the world you describe as should is not the world we live in. I should be able to exist in the world without danger or undue prosection.

Docker containers primarily mean Linux userland. It doesn’t help you with different operating systems.

Hosting stateless apps is simple. If the underlying host fails? don't worry, just spin up a few new copies. Slap a load balancer infront and we can do a/b or blue green.

Smashing.

But where is that state held? maybe your lucky and your data can be hosted in a single RDS instance. Maybe your partition scheme is simple.

However, managing distributed state is hard_.

There it is... Docker is fantastic when "it just works" but there is a huge overhead when it doesn't.

Reply on news.ycombinator.com