It's the Future
blog.circleci.com
blog.circleci.com
"Build, Ship and Run. Any App, Anywhere."
Jesus Christ. I get that you're The Future, but make the value prop for me here, at least. Why should I use Docker? What parts of my stack does it replace? When does the cost-benefit make sense? What new things can I do that I couldn't do before?
They made a separate page just to address this giant "Huh?": https://www.docker.com/whatisdocker, which I feel is equally obtuse.
Luckily, the product itself is fantastic, so that gives you a lot of wiggle room on your website and documentation. Like, a lot.
Look at, say, the homepage of Ruby: https://www.ruby-lang.org/en/. There's a clear, two sentence explanation of what it is:
A dynamic, open source programming language with a
focus on simplicity and productivity. It has an
elegant syntax that is natural to read and easy to write.
On Docker's website, there's a huge amount of confusion about what Docker even is. A platform? A runtime? Both? Which is the one I should care about?
The nerd-centric viewpoint that tool should succeed entirely on their own merits, with no affordances for the user, is crazy. It's that attitude towards UX that has lead to the following one-liner to being the only way to do something so mundane as "removing all your untagged images":
docker rmi $(docker images | grep "^<none>" | awk "{print $3}")
docker rmi -f $(docker images | grep "^<none>" | awk "{print \$3}")
docker images -f "dangling=true" -q | xargs -r docker rmi docker ps -q | xargs docker stats docker images -qf dangling=true | xargs -r docker rmi
docker ps -aqf status=exited | xargs -r docker rm
The definitions of "clear" differ depending on who the target audience is. If you're assuming a heterogenous group of unknown faces, then you aim for colloquial and simplified language.
When marketing to programmers, however, the use of technical jargon and specific concepts is an absolute necessity for something to attain clarity. It's the avoidance of such that obfuscates meaning.
Here's an example of a good software overview: http://homepage.ntlworld.com/jonathan.deboynepollard/Softwar...
The nerd-centric viewpoint that tool should succeed entirely on their own merits, with no affordances for the user, is crazy
This is a straw man. Introductions can be concise or detailed, but they must convey some of the technical intricacies and underpinnings regarding the software. Using marketing language, clouds of buzzwords and too many dumb copy-paste examples leads to cargo cult development and people who jump on bandwagons as opposed to surveying for what is technically superior.
Furthermore, there's nothing wrong with your one-liner.
Some of the time, that is true, but not all the time, maybe not most of the time -- you are putting the cart before the horse. In fact, I would argue that this is an anti-pattern. Yes, you might use (e.g. to pick 2 unrelated domains) Hadoop or Python because they are popular, but consider how they got popular in the first place.
Devtools exist to solve a problem. You should not evaluate devtools based on the webpage or how many people are using it. That way lies Oracle enterprise. :-)
The problem with Docker's website is not that it exists. It is that it substitutes sales & marketing for just simply explaining what it is to a developer. While one could classify this under the category of "marketing", it would be a mistake -- kind of like classifying man pages as sales pitches. Just tell me what the fuck it does for god sakes and I'll decide! I could give a rats ass whether Facebook uses it, etc...
To be clear, I find Docker, the tool, useful, I just think it doesn't need "Marketing", it needs a useful webpage.[1]
Ruby was not adopted because of its webpage or its user base, it was adopted because one person bothered to look into it, liked it and decided to build a very popular web framework around it. Others saw the value in that domain and it exploded. Similar situation for Linux, which started from an FTP site and usenet posting. :-)
"The nerd-centric viewpoint that tool should succeed entirely on their own merits, with no affordances for the user, is crazy."
Unrelated to what I was talking about entirely. Affordances to the user is a merit of the tool itself. Docker could be considerably easier to use in some regards, and that would improve it's usefulness as a dev tool. However, this has nothing to do with attempting to gain marketshare with no direct relation to merit.
[1] But then Docker, the organization, is selling something, aren't they?
What exactly do you think Marketing is? It's not all about BS, it's about communicating a message. If that's a simple webpage, then so be it. Often an idea or product is far too complicated to explain through 2-3 lines of text and needs more.
That sounds rational and it's what I used to think, but I think this talk (https://www.youtube.com/watch?v=FzzL_QDKv0c) makes a good case that fuzzy human factors always play a role in technical decisions, and that's not necessarily bad.
Eg, which is better, Angular or Ember? Ruby or Python? Go or C++? Haskell or Common Lisp? You can accomplish the same things in either tool. Which you like better has a lot to do with what you already know. And popularity may seem like a shallow measure, but it affects whether you can get questions answered, find blog posts and books, locate a library to do something for you, and hire developers who already know the technology.
Popularity is probably also weakly correlated with stability. If my custom jQuery code doesn't work, there's a 99.999% chance that it's the fault of the code I wrote (used by 1 person) rather than of jQuery (used by thousands). When jQuery was new and used by tens of people, there was a higher chance that it was jQuery's fault.
dhh is a brilliant marketer.
Getting people to use things requires communicating to them what the things is for, and how it is better than other alternatives, and how to use it to realize that benefit.
Therefore, devtool adoption requires messaging related to what the devtool is for, how the devtool is superior to other alternatives in the same space, and how to use the devtool to realize that superiority.
Actually having the tool is a start, but its not the ballgame if no one can understand what its for, why they should use it over other things that serve the same purpose, and how to use it.
Anyway, perhaps I'm being a grumpy old man late in the workday, I'll leave it be :-)
"messaging" isn't a condescending idea, its simply having clear, coherent means of communicating some message, with awareness of the audience that message is directed to. Like, what your product is for and why people should (and how they can) use it.
One problem we've encountered is that the audience for Docker is incredible broad - much broader than you might imagine when reading Hacker News for example. It is extremely common for CIOs, IT directors, and various business managers to just pick up the phone to ask us (or our partners) what Docker is all about. So it's difficult to tell a story which satisfies all audiences.
But I don't think that excuses everything. We are definitely better at building our product than at explaining it.
One interesting side-effect is that, if you've been exposed to Docker-related marketing (in a broad sense), most of it probably didn't originate from us, the creators of Docker. This is sometimes problematic because Docker is so polarizing: depending on who tells you about it you will get such a different, often distorted picture of reality. Either Docker is a miracle cure for every disease on Earth (indicating an over-enthusiastic Docker fan, or a vendor trying to sell something to Docker fans), or it's a scourge sent by the gods of the Unix valhalla to punish mankind for techno-hipster false idols such as javascript and php (indicating a jaded Docker-skeptic, or someone trying to sell something to Docker-skeptics). Either way, it creates a lot of noise. And as you point out, there might be less noise if we ourselves did a better job at explaining when using Docker is, or is not, a good solution.
For CIOs: click here
For developers: click here
For sysadmins/devops: click here
For platform providers (e.g. heroku): click here.That much work means that a) they're trying to sell something because it isn't obviously better and b) they're more worried about messaging than being simple and useful.
But useful means different things to different people. To a dev, Docker might mean you keep your environment clean, scripting of automated testing, and have platform agnostic deployments. To the CIO, Docker means your devs spend less time futzing around with their stuff and more time working on the product. Or that he can upgrade his infrastructure gradually and not have to worry about compatibility. Yes, it's the same thing, but different audiences need different messaging.
The question is whether or not the CIO knows that the devs are in fact futzing around with ad hoc solutions to problems that Docker solves.
I think the parent is arguing that in the right flow of things, that awareness is going to flow up to them from the people closest to the problem (the dev/ops folks working with it) rather than from a vendor with a vested interest in adoption to an exec/manager whose understanding of the problems their staff face may well be a high-level view at best. And who are prone to make decisions off of social proof plus that good messaging rather than knowing how well the solution fits their problems.
(Not that having the engineering staff involved is any guarantee that decision won't be mostly made off of social proof plus messaging. It just decreases the chances. :/ )
It's very common for developers or sysadmin to contact us and ask for a powerpoint deck, so that they can give a convincing presentation to their management about the virtues of using Docker in their new project. We even have specialized teams that go in and do everything they can to help Docker-based projects succeed.
But as you point out, it all starts with someone inside the organization who really, really wants to use your product. Otherwise no amount of messaging is going to save you.
In fact, imprecise or unclear messaging is usually a red flag.
If each message is delivered privately to each segment, prisoners-dilemma style, caution is warranted.
If all messages are public to all segments, the product story has likely become modular and coherent.
The general message could still be there (kitchy and sexy), the focus would still be on the end user (developer), but others wouldn't have a hard time finding the message that resonates with them.
Has anyone else seen this strategy succeed (or fail) for their business?
I wish the site landing page did this, too.
To me, docker can be thought of as a process wrapper. The executable is called an image, and the running process called a container. The benefit of docker is three-fold: 1) each process thinks it has an OS to itself, which is a killer feature for native binaries that have weird dependencies 2) network (port) indirection, and 3) filesystem indirection (mounting an arbitrary host dir into an arbitrary container dir).
Against all of this is the whole question of how to really use it to develop and deploy custom software! You could, for example, develop and deploy Java without ever installing Java on the host (not even the development host). But when you are finally ready to deploy, how is that supposed to work? Which pieces are static, which dynamic? Do you bake your binary into the image, or do you mount the binary from the (remote) host filesystem?
The docker docs don't answer any of these questions, and I really think it should.
I have two thoughts on this.
First, if CIOs and IT directors are calling, then it's possible they're confused by the website, too.
Second, the README is definitely better, but would be even better by being more specific and exaggerating less. If it's targeted to web apps and back end services, say that in the first place, instead of "any application." Can I run iPhone apps in Docker? Are there Docker packaged apps in the iTunes store? If my application runs on an Arduino, can it also use Docker?
I meant to say that they used to call us, back when our website looked like this: https://web.archive.org/web/20130329031911/http://www.docker...
> If it's targeted to web apps and back end services, say that in the first place, instead of "any application."
That was deliberate. Although obviously most people use Docker for web apps today, there is no reason they can't try it in other contexts too. For example, there is a very vibrant sub-community of people dockerizing desktop apps, and running Docker on embedded devices like Raspberri Pi etc. There are also (very) experimental ports to Android, and I heard of at least one person who after learning that Tesla is a modified Ubuntu under the hood, set out to try and run Docker on it (I don't know if they succeeded).
From the perspective of this longtime software developer, this isn't that great either. I'll try to give some specifics.
* The terms "container" and "containerization" seem to mean a lot to the author of the document, but they're never defined well and they're used an awful lot despite this. That kind of thing isn't half-bad marketspeak if the point is to get people thinking about this in the vague "It's the future!" manner that The Fine Article is satirizing (and/or accurately reporting as part of the social dynamics of the industry). The invocation of an unfamiliar term over and over can serve as a form of social proof and generates curiosity. But it might well be what's triggering suspicion on the part of some engineers.
* Positioning containers as an alternative to VMs is somewhat helpful in giving at least some idea of what kind of rough problem space we're working in -- someone familiar with a VM knows that they're often used to reproduce the specific runtime environment a given application needs. But by the end of the section I still have no real idea how "containers" are different other than that they're "lighter." Except for one clue: I know what FreeBSD jails are. So I might guess that something like them is involved -- but you're describing them as "primitives", so there's something else involved and I don't see an explanation of what it is. Is Docker a glorified chroot jail? If it's something more, what's the additional value?
* And... two sections down "escape dependency hell" -- that might be the additional value prop! But again, this section is really confusing. Dependency management means package management, right? But it's being done with Yet Another New Undefined Entity called "Layers" without replacing any other package manager so... we have two package managers? Or Layers aren't package management? What the hell are they? I could guess they're something like an image but I have no idea.
* "Plays well with others" Gives strong hints this is mostly a Unix thing (which, if this is some kind of enhanced chroot jail makes sense). Somewhat in conflict with the hints of platform agnosticism earlier in the document. Is there a story here for Windows, either as a host or for windows apps?
* "Real World Examples" These tell me how to "Dockerize" different server apps... but there's no context about why I might want to do this. What problem am I trying to solve?
And that's the thing -- at the end of this, I don't know what problem I might be trying to solve with Docker. I might guess Docker helps me deploy an application along with a specific normalized runtime environment, but that's from a lot of guessing and reading between the lines rather than from an upfront communication from the text itself.
If my description is accurate, a clearer version of it should be your first sentence. Followed by a second paragraph telling me enough about some specific frictions you've done away with compared to other solutions that I'm wanting to learn more. Then tell me some specific stories about situations where someone might have a problem that Docker is a good fit for, and explain the rough usage that be applied to address it.
It's a VM that doesn't need a full guest OS. It's a sandbox. I'm sure I'm wrong but I can't think of another ELI5 for this.
I have trouble with the idea of calling them 'the future'. So far, I haven't seen a REAL reason to use docker in a production environment.
Whether this is good front-page marketing is above my pay grade, of course ... it may well not be.
I'm not ignorant. I understand VM and virtualization in general. I understand chroot. I understand how The ANSI-Standard Multitasking Multiuser OS works. It still took me a few attempts to understand what Docker even is because, frankly, it isn't quite any of those things.
It's a lot more akin to what Plan 9 was doing with namespaces, but I think they take it a bit further. It finally clicked when someone described it in terms of "multiple distros on the same kernel at the same time" and then defined a distro in terms of being an init process and a userspace. That makes sense. Reading up on the clone(2) system call, which is where the 'magic' is, made it even clearer.
But that's impenetrably technical unless you already have a pretty good background in operating systems. As any marketer will tell you, being technical is poison. Ideally, you should be able to sell the product in terms of what it will allow, not how it works. Except with Docker, that's hard: "Oh, it will allow me to run multiple applications at once. It's... an OS kernel? Nope, it says it's Linux. So it's a distro? Nope... uh... is it a new VM? Nope, not that, they all have to share the same kernel... what?"
I guess my point is Docker is hard to market. The website is either going to be vague or rather dense, and vague seems to win.
A few days ago, the CTO of Soylent, the food drink, was describing their elaborate computer infrastructure. They have one (1) product and a simple web site. Based on their sales volume, they do about two sales a minute. That could run on a HostGator "Hatchling" account for $4/month, using one of the seven off the shelf shopping cart/payment programs HostGator offers.
The coolest software project I've ever been on was also the dumbest. We were trying to build a bios in-house - something that can be bought off-the-shelf for a buck. In theory, it could have saved us millions in the long run. In the short run, we were wasting man-months of engineering on something that was not our business, when the clock was ticking loudly. Fun as heck, but if I caught my own employees doing such a thing today, I'd at least threaten to fire them if they even thought about it again.
Likewise, Soylent is not in the inventory management business. Why should they be writing inventory management software? Buy that stuff. Focus development money on the core business.
Blame the employees for crappy management. Sounds like fun!
These kinds of managers are utterly toxic. At the slightest hint of this kind of behavior, I'd be reaching out to my contacts to find new employment, where I would certainly be paid better, but hopefully don't have a feudal work environment.
Cost advantage? What's that? We just want a cool project!
// feature request: poster flag's own posts as tangents that start new topics leaving only a graphic (arrow leaving square) in original thread
https://medium.com/@alando46/how-we-spent-500-on-tech-to-shi...
If you knew how most modern warehouse/shipping facilities ran their IT, your head would spin.
When your availability starts having problems or your data is getting too much for one machine, you start having problems of scale. Where you are in terms of scaling issues should lead you to the next iteration of technology required to keep your service up.
Starting out of the gates with that one idea you're not sure anyone will be interested in? Just stick to deploying something fast and not worry about Google style scaling solutions.
so might be AS400, but few are trying to learn that I guess.
Let's be fair, it's quite likely that if anyone is looking into this stuff is more because it's the technology du jour, rather then the off chance that they end up with 100 million users. It could be useful, and learning stuff is fun, but that's not the real drive.
Knowing the technology du jour is a lot more likely to get you a job, even if an older technology is better in some ways.
Ideally, managers create opportunities for people to exercise the former in a sandbox (e.g. some variation of "20% time"), without YAGNI-ing up the project.
Sorry did I say sandbox? I meant container.
Outsorcing became a Wall Street fad once a few big name companies did the numbers and found they could reduce costs by doing so.
Soon "everyone" was doing it, or announcing that they were going to, even though they had little to nothing to gain from it.
As that saying goes: when all you have is a hammer, every problem looks like a nail.
And right now that hammer, at least in terms of IT, seems to be containers.
I'll just say it--pretty much only Amazon, Google, Facebook, Apple, Baidu, and perhaps a small handful of other companies need Google infrastructure.
Everybody else is probably just chasing shiny on their investors' dime.
Having seen in enterprise incredible amount of man-hours wasted on migrating, re-writing and creating POCs on the latest fad, I sometimes wonder whether this cycle is productive or we keep at it just to keep our brains working and not go insane thinking of geo-political/financial/meta-physical questions. Its perhaps Crack for the tech crowd.
Well, at least they get the privilege of being regarded as very smart. Why be a lowly smart web developer when you can be elevated to the status of extremely smart framework author/designer? If they care about that kind of status.
If you want them to be a bit less trigger-happy, perhaps don't praise them so much for creating their own small contributions to the fragmented landscape.
If people actually have a legitimate need for really high availability and really high scalability and are willing to pay through the nose for the software development necessary to make a straightforward system into a distributed system... well, you should still feel just bad about all the complexity, but you're basically stuck with it :)
"The Architecture Astronauts will say things like: "Can you imagine a program like Napster where you can download anything, not just songs?"
hahahahahhaha bittorrent.
And the basic protocol was later adopted for more generic sharing systems, never mind the number of clones that came about after Napster was lawyer bombed.
Bittorrent is just the latest in a long string of P2P systems, with the biggest difference being the lack of a central search server.
People so deep into CVEs and such that they see every computer as requiring the digital equivalent of fort Knox level security, or civilization will fall.
More users, more developers, larger sites require more than just Jim Bob installing CentOS and tomcat on a pizza box.
It doesn't always require Docker + this tech or that tech, but you will have to put some automation in place.
Bob > -Aphyr is that guy who wrote, ‘Call Me Maybe.’ You know, the distributed systems and BDSM guy?
Anne > What? Did you say BDSM?
Bob > -Yeah, BDSM. It’s San Francisco. Everyone’s into distributed systems and BDSM.
This hilarity demonstrates the tone, and alone is worth the price of admission. Great job with this article.
I've run into this. In reality The Part-Time Parliament was published in 1998 and Paxos Made Simple in 2001.
I mean, it's great that the pieces are all there and open source now. You can get to a really great place with automation by gluing together Docker/Rocket, Mesos/Kubernetes, etcd/zookeeper, etc. But for now, a complete solution still requires you to bring a lot of your own glue.
I mean, by the time you've actually covered enough edge cases to make all these ops components work together in a meaningful way, in a way that doesn't doesn't fail randomly or trip over itself when some critical component breaks, you may as well just start charging other people money to use it.
I say this from the point of view of someone who's done it all (implemented a private Heroku on the stack described above [1]), and although it works amazingly well (for literally hundreds of internal apps), it was not trivial... not even close. We're talking probably 1-2 man-years of effort to get it to the point where it's usable, and that's with leveraging as much existing tech as we can.
To anybody else, as in, any company who actually wants to ship a product (where the product isn't just a PaaS), I just don't see how it's worth it. Just use heroku (or elastic beanstalk, or appengine, or whatever.)
[1] I'd love to make all the glue open source but I'm not really in a position to do so. But I suspect I'm not the only person who's done this... I really think anybody who's gotten this whole "the future" stack working solidly is in a similar position as me: if you really did get the job done for a single app, chances are you've invented an internal Heroku of your own.
I just want to give my project to a PaaS and let them figure out everything.
I was looking into Google App Engine, but they didn't support some language features I wanted (e.g. Java 1.8, Servlet 3.X["I know, programming in Java? You're stuck in the past"]). So I looked into their new Container Engine. But like this article points out, it makes deployment 500000000 times more complicated than it should be.
https://run.pivotal.io/features/
?
Java 1.8 and Tomcat 8.0.x, so Servlet 3.1. On a good day, deployment is one command which just does its thing.
It's not without its quirks, but in my experience they're fairly minor.
They work surprisingly well to this day and despite being horrified upon hearing that this was what they were built on, I found them surprisingly well done.
I wonder what will still be running 12 years from now and what it will look like.
Because it's too close to some conversations I've had.
I love it.
Thank you, CircleCI, for posting this!
React? I thought we were doing Angular?
Come on man, you're stuck in the past!
Edit: just wanted to point out that this post was only partially satirical. I absolutely am in love with Aurelia atm.
It seems that if you aren't running a full dockerized cluster of services or outsourcing everything to a PaaS, you're left with building all the infrastructure yourself. What did people use before this great new wave?
I wonder what's the architecture behind WPEngine and similar services. It must provide some isolation since clients can install their own plugins, but on the other hand I don't see them creating a new Docker image for each client, especially since they're self-managed.
On the one hand it is about getting more bang for your hardware buck.
On the other it is about someone getting so deep into netsec that they have developed gills.
In the bang for bucks category you have a chain of one box pr database etc.
Then noticing that the hardware sits idle most of the time, so virtualization is depolyed to pack more server on a single box to keep it in use.
Then noticing that virtualization comes with a performance overhead, so it gets replaced chroot/containerization to give the impression of unshared box.
In the netsec category it is really about namespace. Limiting the view of the world the processes gets.
This has a superficial similarity to chroot, but can go much much deeper.
And if one go deep enough, every server ends up looking like a digital fort Knox...
these days though you are looking at devops...
Never mind that more recent systemd releases can grok docker containers.
Virtualization does this too, but at great cost. I wish the kinks were better worked out at this point as well, and hope we start to converge around a few well-working patterns and toolsets. I expect it to happen. In the meantime it is chaos and easy to laugh at.
As an example, suppose you use Go. Your dev environment is 500mb of compilers and toolchain. Your production environment is (hopefully) a container with a single static binary on it.
What do the new containers add on top of that (or other than that)? Only the option for more services (not just web app, but database, different languages, whatever)?
It seems odd having to worry about that kind of thing.
It seems odd having to worry about things that aren't Web apps? Why?
Containers are also language-agnostic. .war files are only for JVM languages.
I don't really know docker yet. So if I need a database, rather than instantiating a server with a database, I would create a docker image that runs a database? Then deploy it to some server that can digest docker images (is there a docker image for that)?
The point of war files was not having to worry about the server.
http://www.dwheeler.com/innovation/innovation.html
I think Wheeler and Henney's quote applies to a lot of the advances on this list.
http://www.markroseman.com/tcl/dyingout.html (ref: https://news.ycombinator.com/item?id=9689046)
Edit: link to submitted story.