23david

Part of the problem here could definitely be communication. For whatever reason, it's been incredibly hard to follow what is going on with the Docker project. Keeping info in one place would definitely help.

I'm not able to find any details that support what you're saying. It seems that in June Eric Brewer was still publicly asking for libcontainer to merge in LMCTFY support.

I definitely could be wrong, but looking at this screenshot from Eric's June talk it looks like they were still trying to get it merged:

https://pbs.twimg.com/media/B4DKb51CYAA687s.jpg:large

In June, Eric Brewer posted this:

  We’ve released an open-source tool called cAdvisor that 
  enables fine-grain statistics on resource usage for containers. 
  It tracks both instantaneous and historical stats for a wide 
  variety of resources, handles nested containers, and supports 
  both LMCTFY and Docker’s libcontainer. It’s written in Go with 
  the hope that we can move some of these tools into libcontainer
  directly if people find them useful (as we have).

http://googlecloudplatform.blogspot.com/2014/06/an-update-on...

Not sure. Never heard of an official partnership program either, so I'm interested to know details about how this partnership program works. Could be innocent, but something about it smells like a OSS shakedown to me.

Just 'cause it's open-source doesn't mean there aren't any politics involved in what gets merged or not.

An example:

lmctfy support was added by the Google GCE team a long long time ago and I attended a meetup where the GCE team submitted the pull request right there... it was never merged. Languished for months without any public review comments from Docker maintainers.

I'd never seen anything like this before. Here we have Google engineers integrating their work with Docker on their own time and being completely ignored. Embarrassing is a nice way to put it.

There may have been outside discussions and real issues that made merging a bad idea, but as an OSS project I expect those discussions to happen in the pull request, not in some business meeting. I'm sure any technical issues would have been addressed if there had been any. I'm also sure that the GCE team would have been more than happy to maintain their driver. Politics and open-source are a happy mix.

sources:

  https://github.com/docker/docker/pull/4891
  https://github.com/docker/docker/issues/4874

You're more than fair. They made the rules, and I guess they can break the rules. But really, the value of the project lies in the community's continued support and backing of the project.

The good thing is that this is open-source Apache-licensed code. Easy enough to fork it if necessary, even if just temporarily while we wait for another project to mature and replace it.

  Hudson -> Jenkins
  Mysql -> MariaDB
  Docker -> ???

would be great if the other Docker employees were as clear as you are about that.

Are you willing to share the contract details involved in these partner agreements? What do partners have to agree to in order to be an official partner?

Speaking for the DevOps, a large part of the job is being responsible for building trusted systems from trusted components. I want to continue considering Docker to be part of that trusted stack, but from looking at the minutes, it seems that you are willing to mislead even large partners like Google while on the record.

  JB (Google): Docker is an anchor of a large ecosystem of projects. Docker Inc 
  bought Fig, is it a separate project or is it going to be in Docker?
  SH: it should be explicit. The Fig guys make proposals and those are reviewed
  as every body else's.
  JB (Google): The idea is we will be more comfortable if we know how to 
  contribute both to Docker and the larger ecosystem. 
  SH: The day after Docker Inc. acquired Fig nothing changed at all, everything
  is still in the Open. The reason for the acquisition was to make Docker more
  user friendly. The experience should be awesome. The more people we can throw
  at the problem the better.
  VL: Understanding what is off limits or at least where the limits are would
  be helpful. This is entirely driven by technology? We can collaborate "here" 
  but maybe we can compete "there".
  BG: It's absolutely right, there is a large number of projects competing with
  Docker Inc. and that's fine. Some folks want Docker to remain a packaging
  format. We should add a statement there on clarifying this.

Source (both as github gist and google doc):

  https://gist.github.com/inthecloud247/00eb3130f728b635db9e
  https://docs.google.com/document/d/1JfWNzfwptsMgSx82QyWH_Aj0DRKyZKxYQ1aursxNorg

Just today, the "Host management" proposal initiated by a Fig developer was closed (by him) without any discussion after he released the new Docker Machine project (which AFAIK was initiated without any public discussion or warning.) https://github.com/docker/docker/issues/8681

Benevolent dictator can work. But here Docker has said they want to make it possible for the project to be community led and run over time. Actions on the ground speak otherwise. Hard to restore trust once lost.

AWS started as an open-source community project boasting contributions from over 650 contributors?

Recent post by Docker CEO Golub:

  There are over 650 contributors (95% of whom do not work for Docker, Inc.).

http://blog.docker.com/2014/11/docker-governance-advisory-bo...

Many many solutions have been meeting that need just fine and many new ones are still in the works. Adding a new solution will not help things.

This is a completely new product and really should be differentiated as such via new packages:

  apt-get install docker.io-platform

Just don't put any of this stuff into the docker.io or lxc-docker packages and mostly things will be well in the world.

Agreed. I'm very confused about how the concerns from the advisory board are (not) being addressed in an open manner.

From reading the minutes, it seemed to me like one of the clear conclusions (based on concerns raised by big stakeholders) was that Docker Inc. was going to be firewalled from the Docker open-source project and that development work going forward was going to be done in a more open fashion.

With their (current) control of the main docker index and namespace, they could have stuck with a simple GitHub model (pay for private indexes) and would have been fine. With the community goodwill that they had, it would have been impossible to compete with them.

The reason more people aren't paying for private indexes at this point is because the service is still slow and unreliable, and the UI needs serious work. With their current funding levels, fixing those items shouldn't be a problem. For comparison, Quay.io was able to do it with just a few engineers.

They're just being greedy and overreaching at this point.

Systemd is a fair comparison.

Systemd is a modular system, and it's also possible to not install different components if you want. Most of the components aren't strictly necessary.

I think your community is begging you to continue to support alternatives. You just aren't listening.

If it's an open/pluggable system why the need for these 'partnerships'? Any OSS project can just quickly hack in support for Docker, just as they've been doing for almost 2 years now. This is getting ridiculous.

Docker Machine looks to be a revision/rebrand of your docker hosts proposal made in the last 1-2 days or so. Very confusing.

The proposal discussion: https://github.com/docker/docker/issues/8681

Rename "docker hosts" to "docker machines": https://github.com/bfirsh/docker/commit/e6abec4033f48d1cad31...

2 days ago, from you:

  I have now rebased the host management branch on top of #8265 and squashed it:
  https://github.com/bfirsh/docker/compare/host-management
  Any pull requests should now be based on top of that. The driver interface hasn't changed, so it should be a trivial matter to rebase any existing pull requests. The main thing which has changed is that drivers are expected to set up identity auth for communication with the host. See this commit for an example of how to do so.
  The old branch is here for reference.
  Full update and preview builds coming soon."

1 day ago, a message from tianon, core Docker maintainer:

  Has there been any progress on splitting the actual driver implementations out of the core binary?

And now this. Color me confused.

They sound like a closed-source vendor at this point. I'm surprised to see an open-source project mention "ecosystem partners":

  Each one is implemented with a “batteries included, but
  removable” approach which, thanks to our orchestration 
  APIs, means they may be swapped-out for alternative
  implementations from ecosystem partners designed for 
  particular use cases.

So if I have a startup working on an orchestration solution, what is the process to become an approved 'ecosystem partner'. Do I need to sign a NDA and pay for an approval process to get my stuff merged in?

These aren't new projects... just rebranded versions of half-baked feature proposals that I thought were still being reviewed/discussed. I guess somewhere a decision was made to move forward regardless of community concerns?

Baking these features into Docker is the beginning of the end of Docker's Enterprise story. Moving forward with these proposals guarantees the rise of Rocket and other Enterprise focused containers. Docker is forking its own community here.

I definitely wouldn't consider Mesos to be anything like an operating system for the datacenter. That's just marketing language and confuses things.

Mesos is basically an application scheduler. It doesn't manage the base operating systems or machine provisioning. Mesos is concerned with ensuring that one or multiple applications are launched and running on a cluster of machines.

The Saltstack framework is the only thing I know of that would provide all the primitives and control necessary for an operator to truly control a full 5-300,000 node datacenter from one station. It's the only thing out there that will allow for super low latency response to commands across the cluster. It also can do configuration management etc if you want, or a lot of people use it to just trigger existing chef/puppet jobs.

I've been using Saltstack in conjunction with Mesos to build out this full datacenter-as-OS stack. Works... :-)

I consider Twitter to be an open-source company. The amount of open-source work they sponsor and incubate surpasses that of most traditional vendors.

Their business model is also a lot better. They have a nice unrelated service they make money off of instead of trying to charge for enterprise support or licenses.

Ops (particularly in Enterprise) doesn't want batteries included by default. Principle #3 and #5 are incompatible IMO. Do one thing and do it well...

Seems to me that post-Docker 1.2, the Docker team has taken Ops concerns much less seriously and is focused almost exclusively on iterating Dev-friendly features.

Hope things change.

This is great news, particularly for Enterprise customers adopting containers. IMO, Docker's 'new' direction completely ignored the tremendous amount of support they had from the sysadmin and devops communities.

But crucially, they also crossed the business models of many startups (including CoreOS, Weave, Flocker, etc.) that rely on Docker maintaining an Open Platform. So this is an entirely logical response.

I'll be surprised if now Docker in response doesn't unveil an 'enterprise' Docker version that basically just strips away the unnecessary features and has more security by default. The enterprise market is just too valuable to let it just slip away like this. Your move...

Interesting... Didn't think that avahi would work across subnets. I hope it works with the OSX firewall turned on and set to secure settings. Will need to test, but there are workarounds I've had to use in similar situations.

Can you give some examples of TOML-based Rust package configurations that you think are good? My experience using TOML for config files has been very different.

IMO, using YAML for the config format is still the best option for most use-cases at this point. (Easy to support both json and YAML.) But I do think that it would be great to have a YAML successor that standardizes some of the cool things people are doing with custom YAML parses, and also removes outdated and unnecessarily complex parts of the original standard.

I'm currently using Consul in large production systems.

Started with Serf plus customizations and then migrated over to Consul. Happy to share technical details or answer questions.

Happened to find the pull request discussing the lxc-driver issue. It sounds like there was some interest in contributing upstream, but it didn't really go anywhere.

  https://github.com/docker/docker/pull/5797

From the thread, it seems like the concern was just that lxc-exec wasn't well maintained and the lxc interfaces weren't stable since it was undergoing heavy development. I think that's changed recently with both lxc and docker now post 1.0 release and 'production-ready'.

Docker still uses LXC if you want it to via the lxc-exec driver and the --lxc-conf option. It's just not the default, which probably makes sense since the lxc options mainly apply for advanced users.

So by default, Docker uses libcontainer for a simpler installation experience. But for advanced users, using the lxc driver is an option to look into.

IMO, Docker wins if it continues to play nicely with the other open-source projects people use with Docker, and also give credit where due.

It's great for those nasty legacy apps that only work on old unmaintained versions of Rails or old OS Versions etc.

Take all the nastiness and throw it into a box, without needing to contact Ops to reserve memory and provision a VM.

IMO, it's one of the major reasons why Enterprises get so excited about Docker. Legacy app dependency issues are horrible once you get past a certain scale.

VM's are expensive and non-self-service at most orgs since they tie up RAM and licenses.

Docker plugins ftw!

The Docker plugin proposal mentioned in the post:

  https://github.com/docker/docker/pull/8968

  > it's an objective fact that canonical decided to reimplement 
  > what Docker does rather than contribute to it.

What was the reason that Docker reimplemented much of LXC rather than contribute patches upstream? Latest LXC supports features that Docker's reimplementation doesn't, and it seems likely to get further ahead feature-wise now that Ubuntu is pouring more resources into LXC/LXD.

I'm confused...

Docker Inc CEO Ben Golub:

  Current meme gets it wrong. Ubuntu lxd is complementary, not 
  @docker replacement. Joyent brings Docker to SmartOS. 
  http://zd.net/1uB3Aly

https://twitter.com/golubbe/status/530475539262623744

From reading the release notes, it looks like the current 1.0.0 release doesn't work with the latest stable docker 1.3.1 due to incompatibilities with the tls layer?

  https://github.com/deis/deis/releases
  https://github.com/boot2docker/boot2docker/issues/571
  https://github.com/deis/deis/issues/2230

Why not wait until that is resolved before announcing production ready? :-(

In my experience, automation and devops 'best practices' are always an investment.

But there's a big recruiting and retention issue also... kinda hard to find and retain good IT/devops people these days if you do everything manually :-)