Billions wasted on Hadoop startups, the same will eventually be true of Docker

Also, Docker itself is very much trying to be more than what it is right now, Docker swarm, compose, and basically everything Docker as a company does to improve their product offering moves them farther away from the way people are using "docker" in conversation right now.

Basically, when people say "docker image" they should be saying "container image" (for lack of a better term, at least).

> That is all we want. A few instructions to get a system up. Devs are sick of setting up machines. Demand is there.

This is pretty vague -- "setting up machines" can mean a lot of things. I want to make it clear that it's not the container runtime's job to set up a single machine, it's to enable a process to pretend that the machine it has access to is it's own. It's a subtle difference but it's worth noting IMO.

If you want to fix setting up your machines, you should be looking at tools like Packer, Container Linux distributions (formerly CoreOS, now Atomic I believe), Nix, Guix, and Linuxkit. Docker does not set up machines, it sets up processes, but happens to bring along a filesystem (amongst other things).

A linux container is not a machine (virtualized or otherwise), it is an isolated process, I agree that devs are sick of worrying about machine configuration when setting up the required dependencies to run their processes, though, containerization is here to stay -- it was around before docker and will be here after.

I've found the opposite to be true. If you have proper observability into your isolated services you can more easily pinpoint the origin of a bug.

Containerization of processes definitely increases complexity but if you can take the time to understand VMs then you can (and should, IMO) take the time to understand how containers work as well, they are lighter and simpler (for example, you don't need to build a kernel or make an initrd). I would argue that people who think VMs are simpler are actually being fooled by huge advancements in tooling over the years and the fact that it's become "easy", not that it was ever simple.

I also want to point out that containers should actually make tracking down some bugs easier, but it does so in a counter-intuitive way -- it removes whole classes of bugs from ever occurring. You'll never have two programs clobber some shared folder or resource, you'll never have programs fight over dependencies, or struggle for locally-bound ports if you're running them in containers.

Containerization definitely represents an increase in complexity, but it is well worth the effort, most of the time, granted you understand the tooling.

That isn't pedantry, it is an extremely critical point and the one most people miss when figuring out Docker. Both from a security context (docker doesn't provide vm level promises about isolation) and from a resource management side (docker is really close to no overhead).

It is not uncommon to deploy containers on VMs in the real world...

I agree, I think the next evolution of software deployment is definitely heading to sub-program sizes. I do want to point out that we've seen this before, it was called CGI. It's not exactly the same, and things will be better this time (more isolation, better tooling), but if we're doing functions-as-a-service now, we were doing scripts-as-a-service much earlier.

I think the great unbundling of the future is definitely coming and in fact it's already here -- it's just unevenly distributed.

> I agree that Docker doesn't have a huge and profitable future ahead of it, because it's providing commodity infrastructure. Rather I think it's interesting to think about what the next level of software deployment decomposition will be, and I'd wager that it's FaaS (ie serverless).

This was not what I meant to get across -- Docker may have a huge and profitable future ahead of it, but is only tangentially related to the near assured continuance of containerization. Docker as a company and project's goals are different, they do more than simply offer a way to run containers, and they have for a very long time -- my point was that the literal use of the word "docker" should die, because we should be just referring to containerization in the general sense (no matter which lib you're using). It's like the "tissue" vs "kleenex" debate, in a way.

You rarely see hypetrains for old, established things :)

Feel free to make analogies between two other similar sports, such as Competition Karate and Taekwon-Do.

Docker is far from betting on the wrong horse. I can build a docker container and deploy/orchestrate it however I wish - via a docker runtime, kubernetes (which can vary in its underlying implementation), microVMs, VMs, or bare metal.

But then the containers don't need to be docker either. Theres many options available there too.

Either way, there are many options available. And the horse(s) are still in the running.

As for "serverless" vs containers/other, if you can provision accordingly, containers/other can be a better option at scale. Horses for courses, I guess.

It does isolate a process's view of the filesystem, it does isolate a process's view of the PID namespace, and that is valuable. Let's say it's a spectrum:

raw processes -------- processes with a certain user ---------- namespace+cgroup isolated processes ---------------- VMs

Docker is certainly an improvement for processes that you don't want running completely contained but want somewhat isolated. For isolation in every sense, you want a VM (a lightweight one if you can get it, i.e. some stripped down qemu).

> MicroVMs start faster and provide better isolation.

Agreed on isolation -- VMs are hands down better at isolating, but I'm a bit skeptical about starting faster.

> Meanwhile, none of this is relevant unless you're building your own cloud platform, which is a huge waste of time for most companies.

Again, this is exactly why it's relevant, 99% of people aren't building their own cloud platform, so they don't need the full isolation of VMs -- most of the time they're just trying to prevent program A that their devs wrote from clobbering program B that their devs also wrote when they both run on the same machine, and making it easier to deploy the dependencies that come with each.

> MicroVMs, containers, VMs, zones and bare metal are places to execute code. Serverless makes all those distinctions irrelevant.

OK I don't even really know what this means, you know serverless runs on MicroVMs right? and most of the time it's actually containers in MicroVMs? MicroVMs are just stripped down versions of regular VMs, and no one is in a zone unless they're running Solaris.

The distinction between these things is still very important, unless you mean that the future is everyone just deploying functions for their applications? But even if you mean that, cold start is basically the first stumbling block you see and it literally exists because of the distinction between how these technologies work (and how fast they can be started on demand, with how much isolation).

> Sorry if you spent 2015 getting really into Docker. You bet on the wrong horse. It's OK, this happens in tech.

Is there anyone that spent 2015 really getting into Docker and isn't better for it now? The technologies that have sprung out of this part of computing are very valuable to know and are getting more valuable, not less. You don't have to install a VM to run an isolated postgres instance on your dev machine because containerization exists -- if you're still doing this you should probably look into updating your tooling.

Also, don't forget that containerization is how some of the richest and supposedly best (due to their ability to spend money on engineers) companies in the world have been handling deployment for nearly a decade -- 2015 is late for realizing containerization is a good thing, not early.

Yeah given how often people mistakenly call containers VMs or assume they have VM-like properties (like the ability to run a different kernel), this is tripping a lot of developers up, not just lay people.

> Isolation and sandboxing is virtualization. In a container, the applications seem to have an operating system and machine to themselves.

I don't think this is quite true, generally you can isolate and sandbox something without virtualizing anything. In this case to isolate and sandbox without a loss of functionality, we are virtualizing and swapping swap implementations out from underneath an application/process. That is not always the case though -- if you unplug a machine's ethernet cable it is isolated, but you did not virtualize anything, if you deny access to a folder on disk for a process you are isolating without virtualizing, unless you mean that any kernel interaction (like blocking an `fopen()`) is "virtualization".

> A single Unix process and its address space is also a kind of virtual machine, creating the illusion that the process has a machine all to itself. Thanks to virtual memory, other processes are not even visible; they are in a different name space of pointers. That concept breaks for multi-process applications: processes are aware of each other through manipulations of shared resources like files. Or effects like not being able to bind a networking port because some other process is tying it up. The next level of virtualization is to have namespaces for resource-related namespaces in the system beyond the address space. As far as just the filesystem space goes, we can virtualize with tools like chroot. A group of applications can have their own global /etc configuration, their own version of the C library in /lib and so on. That's the beginning of "containerization".

This is a good summary of how containerization works, and the layers of isolation provided by the kernel -- the approach here is isolation by providing a fake, but that does not mean that isolation + sandboxing = virtualization, that just means that these forms of virtualization can be used to provide isolation & sandboxing.

This is less true these days (and on macs that had to use the docker machine "hack" it was barely ever true, per say) -- rootless containers are on the way thanks to user namespaces. For example LXC can run fully rootless containers that act more like VMs themselves (as in they will have systemd as pid 1 inside) -- kernel support, user namespaces, and File systems in Userspace (FUSE) make this possible.

IMO One of the biggest benefits to running containers is actually E2E tests -- I don't see it done as often as I should, but it has become drastically easier to run an entire postgres instance for a single local E2E test runs -- I do this on almost every project I start now, I set up E2E tests that spin up the actual world I expect in production (so all the backing services, at the version they will run at), and interact with my application -- this is a huge step forward compared to a huge wiki that has "how to set up the local test VM" -- you can spin up and shutdown these services so fast that you can use them ephemerally (I do) without much worry.

So it's an innovation in packaging.

The two primary use cases for Docker is, as far as I can see, is simplifying deployment on varying environments. Variations can happen because of many reasons. Sometimes you have clusters of various sizes in production. Sometimes the environment is a developer laptop. And so on.

> applications that have weirdly specific, non-standard system level dependencies

Spot on. 99% of software the world needs can be written against libraries provided by OSes. And then packaged properly.

I can give my coworker a docker image and it mostly "just work" without failing because she happens to be running a slightly different version of Ubuntu with different system libraries present.

A slightly smarter .tar.gz would have solved the problem just as well.

It's great that you compile postgres but I just want to run it in a clean and portable way, along with several other programs, and without learning new workflows for each one. Docker containers give people more options to package and run software in a simple standardized process while offloading the tedious system details that don't matter. That's progress.

As an ops person myself, docket saved me lots of time.. defeated can run their containers locally then hand then over to me to stand up. As we move to hosted services, I don't even need to maintain a server. My role of shifting from spending lots of time on ansible and monitoring servers to helping look at code and spending more time investigating weird bugs outside the developers capacity.

I was an early Hadoop adopter as well... And I agree with people's sentiment here -- it was a tool looking for a problem (outside it's specific use case). I used it for it's intended purpose, and I have with it too make it a web crawler to. It actually kinda worked in that regard, but it's not the right usage. It might be able to expand into new use cases though.

Docker solves (again) a real problem in the industry that had existed for decades... And The problems solution keeps going back and forth. Nowadays we train developers, not systems engineers (I've been trying to hire a systems engineer for almost a year and have nearly no bites... Or developers positions get 3 good candidates worth interviewing in 2 weeks or less). This means we have lots of available developers and not enough ops people. Containers help shift the burden to work in this dynamic to -- it simplifies the process to get the devs application to work in isolation. This means 1 ops guy could support a dozen developers and 30 apps on one server relatively easily compared to before. It shifts the burden of the developers runtime environment to the developer... We can still step in too help, but when file that environment is codified in git.

I've been an ops guy for a decade and unlike my positional colleagues I love Docker, it's let me focus on more important things.

Many people use/used Hadoop for problems that did not warrant the overhead and complexity that comes with Hadoop. I've seen it countless times with my own eyes that people pre-emptively use tools like Hadoop and Spark because of a chance that they will hit a massive scale in the future.

This happens in both startups and enterprises alike: people like to think they have big problems too often.

- Fits on one computer (most of the market)

- Fits on several computers (most of the rest)

- Requires a significant cluster of machines (50+ to store it)

Hadoop only really solves the last one. It has huge overheads in terms of speed and in terms of resources and headcount to run it properly, so it only makes sense at a particular scale. It's like a mainframe – most companies shouldn't buy one.

If you add to this the fact that Hadoop was about batch processing, and its "realtime" capabilities were poor, there really aren't that many potential customers, and many of the potential customers would rather run it in-house, or build their own system.

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

My guess is that almost all programming jobs are in fields that produce no more data than a gb or two a month.

I'm happy to be proven wrong, but I would guess that there are far more companies making project management software, time tracking apps, invoicing software, etc. than there are facebooks, googles or reddits obsessively logging every user mouse twitch.

And that's data that's much better sitting in a nice, normal, relational database.

Even Postgres got this right recently with the introduction of the BRIN index, which is a lot more lightweight.

Look at Netezza, Oracle Exadata, and (disclaimer: I work on this) SQream DB, which can absolutely handle hundreds of terabytes without too much fuss.

How are they then querying over big data these days?

We don't know, do we? Or did they open-source their search engine?

By using Hadoop people are trying to not reinvent the big data wheel, partly because it's a motherfucker of a problem to have to solve and party because they want to solve the business problem, not the technical one. I don't see how that is in any way worthy of being frowned upon.

now, on bare metal server and premium providers where you're able to specify the boot image that wouldn't have been a problem, but there are lot more people dealing with cheap housing giving you whatever image they baked last than people doing bare metal deployments and for the former docker is an order of magnitude easier to deploy

Yes, Docker is a shitty solution to just packaging applications, but it exists because Linux developers keep saying "apt exists. It's solved! What's the problem? Static linking? Why would you want that? Portable binaries? But apt is the only place you need to put your app so you don't need them!"

I am yet to see a single project which is relatively new and is packaged on Debian (let alone Raspbian). And lets be honest - nobody is genuinely using Raspberry Pi to run an outdated media server.

Then, as a matter of convenience, you get packages, that either bundle everything preconfigured to their liking (i.e. gitlab omnibus) or packages, that configure shared services how they need it, and running them shared is not supported (i.e. freeipa, which will configure httpd as it needs and forget having anything else served on the same machine).

Docker provides a way to isolate these, so you can still use the same resources you have to run applications, that would not cooperate with each other on a single machine, without having to run separate OS instances in separate VMs.

You can sandbox a random executable with seccomp, you cannot effectively sandbox a whole container without breaking many things.

One of the reasons Docker is so popular is that it bypasses that garbage fire.

If you want to run 200 different services on 40 machines, you may find hand crafted VMs easier to create and forget.

If you want to run 10 services with different levels of replication and redundancy on 40 machines, then Kubernetes will do that for you.

However, docker internals have basically been replaced with containerd[0] in this point -- the two front runners in the battle to actually run your containers (and power higher level abstraction tools like docker) are containerd[0] and cri-o[1].

I personally prefer containerd, but there are a lot of people who are obsessed with cri-o (big company backers, from what I can remember), despite the fact that it's chronically behind on features (for example alternate runtimes, runtimeClass support), but they're both excellent.

Note that there are also other projects like podman[2] that also aim to serve as docker replacements.

Discussion on containerd shim in docker can be found on google group[3] way back in 2016.

[0]: https://github.com/containerd

[1]: https://github.com/cri-o/cri-o

[2]: https://github.com/containers/libpod

[3]: https://groups.google.com/forum/#!topic/docker-dev/zaZFlvIx1...

I hope WebAssembly goes in this direction, instead of trying to adapt to current programming language paradigms.

The idea was interesting, but it didn't quite work out. At some point he left the company and we had to do something about the pipeline, as it was crashing most of the time. We did some calculations, and figured that with the right approach, a good old MySQL+PHP solution would do the trick. And it did.

Having switched jobs myself in the meantime, I'm happy knowing the system is still running, and finding people to maintain it is relatively easy.

In telcos you have network telemetry data. In supermarkets and retail you have purchase data and often credit card gateway data. In banking and finance you obviously have transaction data.

And with Kubernetes you still need a compute framework. Like I don't know the standard in the industry the Hadoop Spark framework.

Whenever a new big piece of tech comes out, most of the detractors seem to either a) find a use case that isn’t fit for purpose or b) try to use it without bothering to learn how, and the proceed to say ‘see, it’s not all it’s cracked up to be’.