CoreOS (YC S13) Raises $12M to Bring Kubernetes to the Enterprise
techcrunch.com
techcrunch.com
EDIT I also continuously wonder how do things like Ansible fit into the picture. Are they competitors? Are they supplemental to the likes of Mesos/Kubernetes? Are they orthogonal?
It really depends on how you do deployment. Containers provide deployment (and more important rollback) that is better than other deployment tools like Puppet/Chef/... because they are atomic (they either work, or they fail, they don't get stuck in the middle) and they package up all of their dependencies within them, so that they don't have the "well it worked on my machine" problems.
Systems like Mesos and Kuberenetes, decouple applications from the individual machines (and the operating system on those machines), and are online systems with self-healing properties (so that they will fix themselves rather than waking you up in the middle of the night)
k8s and mesos turn containers into an API that spans an entire fleet of machines, and enables you to dynamically use (and re-use) the fleet of machines for multiple different applications. No more dedicated boxes for mysql, mongo, etc. This in turn enables you to have an easier ops experience, because every single machine in your fleet is homogenous (same OS, same patches, etc) OS management is abstracted away from Application management, so that they don't interfere with each other. Since things in the API are expressed in terms of applications, it's easy to add health checks and automatic restart to the system, and provide self-healing properties as well. Both kubernetes and Mesos also make replication a first-order primitive so that it is easy to scale in response to load.
Ansible is sort of orthogonal to systems like Kubernetes and Mesos. Kubernetes and Mesos are designed to be online, self-repairing systems. Ansible is a way to easily execute commands on a bunch of machines. I can see collaborative use cases, where you generally use Kubernetes for deployments, but use Ansible for querying some data while debugging, or somesuch.
Anyway, sorry for the extended response. There actually is way more that I could say about the topic ;)
You also mention the "no dedicated Mysql, Mongo boxes" etc. I am 100% for that. However, how can you really make that work with databases/systems like MySQL which were fundamentally designed to work on one machine, and scaling them across machines is usually very painful, or at least, let's say, not very "idiomatic" (if I can use that word here lol). I can see the auto scaling part working with distributed databases like Riak/Cassandra, but even there the solution is not clear-cut and "out of the box". It still feels like some manual orchestration work is needed - correct me if I'm wrong.
I can totally see the "online, self-repairing point" but only for application servers that were designed from the scratch to be easily scalable by just adding servers. Which is the case for most scripting languages PHP/Ruby/Python et al and for well designed JVM/CLR/native/Go applications as well. However, I would argue scaling the app boxes/containers is the EASY part. Again, you can always scale up by "just" copying machines (I know, it's always harder that that). The hard part comes with managing the database servers or your cluster of messaging queues, or some other stateful thing that has to persist data SAFELY. Does Kubernetes/Docker REALLY make my life easier with those kinds of things? Is the answer to use DynamoDB/BigTable/RDS/managed queues and forget about that hassle of managing a database or a queueing cluster? Looking for answers :) . Thanks!
There's two perspectives to this:
* From a user's perspective, ignoring performance implications, containers should be the same as VMs.
* From a hardware perspective however, containers are much more lightweight in terms of CPU, Ram and Disk space, since they all share at least the kernel.
Why should you care? Well if something is less resource consuming, it means that (a) you could run it for less than what you pay for VMs or (b) it can be sold to you for less. There's some additional benefits like fast "boots", strict decoupling of persistant and non-persistant storage (which I find an advantage, restarting a container cleans up whatever you don't care for), but at the end I think it comes down to money.
Is that a good thing? Granted, the AWS account I used to managed was 8-10 machines at most, which is probably well below the point where Kuberenetes makes sense, but I remember that it was useful to configure different kind of machines for MySQL than Apache (disk needs were vastly different, for instance) and that the MySQL instance for the web server had different needs than the MySQL instance for offline data processing.
To me, unless you have a big park of machines, these systems are a total overkill... but I guess that time will say.
I think that's an important point, and one which container vendors are not going to labour, as they want as many people as possible on their platform, even before they really need it. A lot of people are trying to use docker or coreos who really don't need to, and as they're not the focus of containerisation efforts, they'll suffer as they find out they're not really tailored to what they want to do, which is just get their small web service running reliably with the minimum of fuss, and be sure they can rebuild it or move it between providers easily.
If you have 1-10 machines which don't change much, use Ansible or similar to get predictable (re)deployments and don't worry about using containers.
If you have > say 10 machines, this sort of stuff becomes more useful, because you are herding cattle, and need the infrastructure necessary to keep that herd going, even if a few die off from time to time - then you can scale to hundreds easily as your business grows, you can manage lots of workers reliably on one VM in containers etc, etc.
For probably 90% of websites out there, with a sane setup that's never even going to become an issue and they could run easily on just a few servers.
you either heal it back or you shoot it. shooting is way faster and docker could help you by that.
Also 1-10 servers could be much, it really depends how much stuff you need.
Also docker adds "some" security. Docker isn't the perfect match, but on our site we run a match between ansible and docker (without coreos) and are very happy.
In our aws cloud we have another system which only uses fleet and coreos, the cluster upgrades itself which is a big plus, but doesn't work that good in our internal infrastructure with proxies, firewalls, etc..
We run 5 machines in aws.
And have like 6 vsphere esxi nodes.
- Abstract away elements of failover, disaster recovery, clustering, load balancing
- Abstract your application services cleanly from underlying servers
- Give you more consistency across and within development, test and production environments
- Allow for immutable deployments
- Allow for canary releasing, A/B releasing, rollback etc
- Better isolation of processes and services
- Move away from general purpose operating systems to lighter weight single purpose OS such as CoreOS
Configuration management still has a role to play in this stack, but somewhat smaller than the old world
I can only assume Tectonic and related tech are aimed at those developers who run hybrid or private clouds for the rest of their company.
I'm not sure how traditional PaaSes are limited in that sense. Heroku accepts a wide variety of software through the buildpacks mechanism. Cloud Foundry supports both buildpacks and docker images, with clean extension points to add further mechanisms -- for example, .NET-on-Windows deployments, which is under development.
Disclaimer: I've worked on Cloud Foundry and I'm employed by Pivotal.
For example if a service knows your container is serving a web application then it can sensibly provision load balancers, DNS, HTTPS, auto-scaling, static content caching, automatic QPS monitoring etc. to support your app with little explicit configuration. And with a commercial service you get an SLA for those things.
You can get much of this with a general purpose cluster as well, but of course you need to configure it yourself, and more importantly - debug it when it goes awry.
Disclaimer: I work for Google
At this point, I never pick PaaS because it's less work, I pick it simply because I've used it before and it's easy to get started with. Production apps are never hands-off if you're the one developing them ;-)
Docker popularized containers, and now we're seeing the supporting technologies that will help fulfill some of the promises of container-based deployment. Big companies have had this technology internally for years, and now it's starting to get productized.
I'm referring to things like schedulers, networking, monitoring, etcd/consul, and more. I personally don't think all of the pieces are there yet to make it as seamless and easy as it could be, but it's coming.
This statement rings pretty true as Kubernetes (also known as k8s) has some Google biases. Not all cloud providers will have such an easy time providing all the infrastructure necessary to run CoreOS + k8s smoothly.
For example, Kubernetes assigns each Pod (k8s unit of computation) an IP address, which is only simple to do if your cloud provider supplies something like a /24 private block to your nodes. CoreOS came up with the VXLAN-based Flannel project to make this model more portable[0], but Layer 2 over Layer 3 isn't something I'd like to throw haphazardly into my production environments. Google Compute Engine conveniently provides this setup as an option.
Another example of Google-favoritism is the strong preference of centralized storage--particularly GCEPersistentDisk. At first I was concerned about centralized storage by default as we know disk locality is a Good Thing (TM), but after reading a paper that claimed networking is improving faster than disks are[2], I felt somewhat better about this. However, it's still pretty obvious that a Google Persistent Disk is the way to go with k8s[3].
That said, I'm really happy that Google has open-sourced this project because it is indeed a functioning, tested, and easy-to-use distributed system. I'm sure that the devs aren't aggressively shutting out other cloud providers and that these biases are probably just a side-effect of their resource allocation process and the problems that they intend to solve (e.g. GCEPersistentDisk used to be a core type instead of a module--it has since gotten better). It's still important to evaluate a technology's biases and potential evolution before throwing your product on it.
[0] https://github.com/coreos/flannel
[1] https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...
[2] http://www.eecs.berkeley.edu/~ganesha/disk-irrelevant_hotos2...
[3] Do you see any other providers here? https://github.com/GoogleCloudPlatform/kubernetes/tree/maste...
I have found the Kubernetes team to be nothing other than extremely supportive of efforts to support AWS & non-Google platforms. It takes a little longer to translate some of the Google-isms to other platforms, but I'm happy for the thinking behind those decisions, vs just adopting lowest-common denominator.
https://github.com/GoogleCloudPlatform/kubernetes/tree/maste...
I'm mostly concerned about feature lag and vendor lock-in, so I'm happy to hear that this will be out so soon. I'm excited to try it out.
> I have found the Kubernetes team to be nothing other than extremely supportive of efforts to support AWS & non-Google platforms.
I don't doubt it one bit; in my experience, people on the Kubernetes IRC channel have been always really helpful and supportive. I just tend to be a little more pessimistic when it comes to resource allocation: a Google team probably prioritizes support for Google platforms, and that's no one's fault or foul play.
Thanks for your work!
However, I don't buy it for 2 reasons:
1. Highly available distributed services need to have geographical diversity, i.e. they should be "multihomed". This is true on AWS or in Google's internal data centers. That means you have WAN latency, in which case locality becomes again the primary design concern for performance.
Pre-Spanner, Google's solution was to use application-specific logic to be multihomed -- i.e. nearly rewrite your application, depending on how stateful it is. Spanner isn't a silver bullet either. You still have to solve latency problems, just within the ontology of Spanner rather than the application.
It's bad for your code to ignore latency within the data center, and then later add (incorrect) hacks to work around latency between data centers. If you pay attention to network boundaries from the beginning, it will be easier to multi-home.
2. A single machine is still your domain of failure. Even if it doesn't matter for performance, you still have think about machines to handle failures.
The interfaces between machines should be idempotent to handle failures gracefully. And many distributed storage services have complicated performance vs. durability knobs for how many machines/processes have accepted writes.
So I think Google does have a "single system image" bias, and you are right that Kubernetes has these Google-isms in its architecture.
Each pod has it's own IP address that is routeable anywhere in the cluster. This makes life much easier because you don't have to do port-forwarding onto the host node.
In all current k8s set-ups, each Minion/Worker node has a subnet that it allocates these Pod IP addresses out of. This isn't a hard requirement necessarily, but it tends to be much easier to make this work, since you only have O(Workers) routes to configure instead of O(Pods), but long term, I think we would rather do away with subnets per node, and simply allocate IP addresses for each Pod individually.
The paper seems to peg local disk bandwidth at 150Mbps, and then compare it to remote network disk access at... 150Mbps. NVMe is going to grant us 2.2GBps bandwidth and 450K IOPS (from a single consumer-grade product), so that paper is off by more than an order of magnitude. Local disk is non-volatile storage sitting a PCIe lane away from your CPU. I just don't see how disk locality is not going to be crucial for many workloads, for decades to come.
In 2020 a flash-only SAN isn't going to deliver 20Gbit/sec to each of 100 blades in the rack. A 4TB NVMe card on each blade will though...
Look at Intel's latest Xeon-D SOC, yeah, it's got Dual 10GBE, but you're not going to get 7.7GB/s over that... [1]
[1] http://www.intel.com/content/www/us/en/benchmarks/server/xeo...
You also mention blades, which goes into the next point of context which is that operations like Google and Facebook don't utilize blades like you might expect working at your average enterprise because they aren't leasing rack space or working with a limited amount of physical space. They don't need the same U to performance ratio so they can save money by using commodity hardware. Their applications also scale readily, so the loss of entire boxes is meaningless within a certain threshold.
CoreOS is currently working on adding support for Rocket[0] to Kubernetes. There is still a lot of work to do here, but things are progressing nicely. Kubernetes is a true opensource project and accepts contributions that makes sense. Supporting another container runtime engine makes sense in terms of choice for the community and forces clean separation between Kubernetes, the cluster manager, and the underlying container runtime engine.
When it comes to Rocket vs Docker I think the competition will be good for both projects and the community as a whole. At the end of the day, both projects and the communities behind them want people to be successful with containers.
[0] https://github.com/GoogleCloudPlatform/kubernetes/pull/5518
I think it's better said, Docker wants people to be successful with containers, so long as it's Docker containers.[1][2][3]
We've seen a lot of very open work coming from the CoreOS team with Rocket and the ACI... unfortunately we've only seen negativity and mud slinging coming from the other camp.[2][4][5][6][7]
Docker is more interested in having an "open" implementation, rather than an open specification. The things is... an Open Specification is far more important for industry changing technology like containers.
[1] https://news.ycombinator.com/item?id=8683705
[2] https://news.ycombinator.com/item?id=8938409
[3] https://news.ycombinator.com/item?id=8789181
[4] https://news.ycombinator.com/item?id=8938066
[5] https://news.ycombinator.com/item?id=8938176
From what I can tell, CoreOS was happy to let Docker, Inc control the container platform and build orchestration tools on top of Docker, since it's pretty clear that no one is willing to pay for the base containerization layer and the only money to be made is in orchestration. But then Docker released their own orchestration strategy and that started the current period we're in where both companies have come off as being very unprofessional and concerned only with their own future success.
Between Docker trying to build almost everything into the actual Docker executable (it's a container runner, API server, build tool, process monitor, cloud provisioner and orchestration tool...those last two are only not the case because they got called on the strategy) and CoreOS's pointless pushing of rkt (rkt itself isn't pointless, but the PR to add rkt support to Docker was, as was implementing Docker support in rkt). Both sides have foolishly put their own interests ahead of the container ecosystem.
To say that either of the two companies has been the bad actor in all the drama would be biased. They've both come across like spoiled little children trying to get their way. As someone who has been pushing my company (~8000 employees) to consider implementing a containerized deployment strategy, it's been hugely inconvenient to have to concede to people arguing against containerization all the points around how unprofessional the companies in the space have been, both in behavior and coding practices.
I love my Apple MacBook Pro and its soldered components, but I need my data center to be meaningfully upgradable.
Frankly, CoreOS started the ACI and Rocket because there was no open specification. Docker had started off with somewhat of a specification, then deleted it. A year later, CoreOS launches ACI as a public open spec, and suddenly Docker whips one of their own up over a weekend.
The entire point of ACI is nobody owns the specification. Docker wants to own the spec, govern the spec, and be the spec.
With something as critical as this technology -- we need an open specification -- far more than an open implementation.
After announcing Rocket, they embarked on a campaign of subtly trying to show people how uncooperative Docker, Inc was being, first by implementing Docker support in Rocket and then that sham of PR (the first ever PR PR) where they implemented Rocket support in Docker. It was so much code and, with no warning that it was coming, there was no chance that it would ever get merged. The only point of that was to try to argue that they'll support Docker and Docker won't support Rocket. It's so transparently passive aggressive to anyone that understands the underlying dynamic expressed in my earlier post.
> With something as critical as this technology -- we need an open specification
That's fine. But there's a right way and a wrong way to get there. They should have started by documenting the existing Docker implementation. It's open source and implemented by many Docker partners (Amazon, Google, etc). Once the community has accepted the specification, as the authors, they would have license to go to the community and push for changes they feel are necessary. Docker isn't fundamentally flawed, but it is flawed and I think even Docker, Inc admits that. There are changes that are necessary and they need to happen out in the open rather than behind closed doors at Docker, Inc.
Starting their own format was an attempted power move designed to fracture the community. And I see it as largely having failed. There are a lot of things that I like better about Rocket, but the ecosystem around it is way too nascent to even consider using it. There's no collection of images that I, as a developer, can leverage to avoid building everything from scratch. Docker Hub is a big part of the reason to use Docker. And there's no third-party integrations for Rocket. I can use Docker through AWS ECS and have the piece of mind to know that I can migrate to GCE and my Docker images will basically just work. With Rocket, I basically have to use CoreOS (a design I'm not thrilled with) or roll my own.
If either company seemed like it was trying to put the community first, I'd be 100% behind that company. But CoreOS's rhetoric around trying to position themselves that way is simply indicative of them not having first-mover advantage. Both companies need to grow up and realize that this is a nascent market and there is room for both to succeed if they're only willing to cooperate. But if they continue to squabble like little children fighting over a ball, the parents are going to take that ball away. Google and Amazon will roll their own proprietary containerization or, worse yet (for Docker and CoreOS), they'll cooperate on an open standard. But the community is what's important. If they can continue to grow the community, profits will follow. If they continue to try to corner the market and push each other out, the community will go elsewhere and both companies will fail.
> With Rocket, I basically have to use CoreOS (a design I'm not thrilled with) or roll my own.
Rocket works on any Linux platform. Other implementations are working on more and more platforms, including BSD's.
They all adhere to the ACI open specification.
ACI is an open specification that anyone/everyone can contribute to (including Docker if they wanted). To date, there are more than 4 completely different implementations of ACI (one of which is Rocket), all written by different organizations. Apache Mesos is working on a 5th, and more are coming.
That makes the ACI have 400% more implementations that Docker's "specification" which was written after-the-fact and is more-or-less a documentation of how Docker was implemented rather than an open specification in-which the community helped form.
ACI belongs to the community. It's being contributed to by a large userbase and is growing rapidly.
This was the entire driving force behind ACI from CoreOS -- to get a standardized specification that anyone could implement. CoreOS saw the need for there to be more than a single container technology -- but they must be inter-operable.
It's a lot like virtiualization before the OVF format came along. ie. If you made a VMware guest, you were stuck with VMware. After the OVF format, you could export your VM to an OVF image and import it into any other standardized virtualization platform -- you were not locked into a single for-profit company and carried at their whims.
You seem to think it's about a "money grab"... well, sure if users adopt Rocket maybe CoreOS gets support fees etc... but if users adopt one of the many other implementations of ACI, CoreOS doesn't get anything except a diverse ecosystem of inter-operable container platforms.
That's a win for everyone.
CoreOS still supports Docker -- and I've seen nothing but praise come from the CoreOS camp. That doesn't mean Docker is doing the right thing for the community at large -- instead, they are doing the right thing for Docker, the for-profit company.
I know exactly what the app container spec is...stop being patronizing. My point was that I have to use CoreOS FOR ORCHESTRATION because my normal options for orchestration don't (yet) work with Rocket or any of its implementations. I know CoreOS has publicly announced that they'll make Kubernetes work with Rocket, but that's not yet reality. And the orchestration option that we're currently pursuing, ECS, isn't likely to support Rocket any time soon.
And CoreOS doesn't exactly have a great history of being interoperable...the tools that make up CoreOS have hard dependencies on each other. For example, I find Consul to be better than etcd in almost every conceivable way and yet CoreOS forces me to run etcd. I'm not going to believe that CoreOS cares about interoperability until they're willing to take the same "batteries included but removable" stance that Docker has professed.
You seem to have a very pro-CoreOS bias. I'm not going to try to change that. Let me be clear on my biases. I like the design of Rocket far more than Docker. I don't want a single executable to build, run, monitor processes and be a REST API. But I also disagree with almost every decision that's gone into CoreOS. Fleet is cumbersome, systemd is a train wreck (the one thing an init system cannot be is non-deterministic and I've seen too many systemd boot sequences be that) and etcd is inferior to other options. So while I respect that Rocket can be used elsewhere, there's still little-to-no support outside of CoreOS when it comes to everything that goes into running containerization in a production environment.
My company has serious dev-test-prod workflow issues that can be solved by containerization. We're publicly tied to running in AWS. CoreOS is simply a non-starter. While I completely agree with all the negative things that have been said about Docker, I can still advocate Docker-based solutions in the meetings I attend on this subject because everything I need from the ecosystem is there, today, ready to use. Rocket simply isn't there yet and it could have been if they'd started with the existing Docker implementation as the basis for their specification. And you're not going to convince me that that NIH decision is in the best interests of the community.
It's not "their" specification. You really don't get what the ACI is. Nobody owns or controls the ACI -- it's a community specification being worked on and developed by a community, ie. no one entity dictates what it looks like.
> I find Consul to be better than etcd in almost every conceivable way and yet CoreOS forces me to run etcd
This is out of the scope of this discussion. Rocket runs on any Linux. ACI implementation run on any 'nix currently. There is not interoperability issues here. etcd is part of CoreOS the Operating System. However, you can use etcd outside of CoreOS the Operating System.
This complaint is like saying your don't like iptables in RHEL 6.x and since it's not easy to change out, you refuse to use Wayland. It makes no sense.
> I can still advocate Docker-based solutions in the meetings I attend on this subject because everything I need from the ecosystem is there, today, ready to use.
Perhaps this is a sign that contanerization is simply not ready for mainstream enterprise use yet. Sure, Docker stamped the big "1.0" to impress investors and Red Hat, but that doesn't mean it's actually really ready for the big time. Especially since other offerings are rapidly brewing. You've gone this long without containers, you can definitely wait another 6 months (or switch to BSD and use jail containers which have been mature for 15+ years).
> I'm not going to believe that CoreOS cares about interoperability until they're willing to take the same "batteries included but removable" stance that Docker has professed
Again -- you don't seem to understand what the ACI is. It's a community thing -- CoreOS doesn't own it. That guarantees interoperability. The ACI specifies what a container should look like to be "standards compliant". It's up to the specific implementation to make a system that works with that standard container format. The system can be implemented in any language, work any way they want, do anything they want, etc... so long as it can read and understand the standard container format. If that is not "interoperability", I don't know what is.
> You seem to have a very pro-CoreOS bias
No, I don't. I have a very pro-Open Standard bias.
The world does not need another vmdk.
We started building fleet from the ground up with plans to build a higher level tool on top, which would provide the application centric workflows you see in Kubernetes. But like all decisions at CoreOS, we only build stuff we have to build. When Kubernetes came out we quickly started evaluating it for our needs. It turns out it's a great fit, which is why we built Tectonic around it.
Kubernetes adds a lot of value to the container ecosystem and is grounded in the same roots as other CoreOS projects: opensource and highly collaborative. Both CoreOS and Kubernetes want to see our components leveraged outside of our commercial products, but we also feel that we can offer the best experience by combining them into focused offerings for people looking for a complete package.
However, fleet is designed to be relatively simple, and can be used to deploy 'services' (as in systemd services), while kubernetes is more oriented around deploying 'apps', and provides a much higher abstraction that makes discovery, etc easier.
Now it's all about execution.
CoreOS is now focused on balancing new features with maturity and stability. For many of our projects this focus started almost a year ago. Take etcd for example. CoreOS spent the last 9 months effectively rewriting etcd and it's raft implementation to meet the stability requirements we need going forward. Other projects have demonstrated great signs of stability such as CoreOS Linux, our Linux OS focused on containers. It's not perfect, but there are many people that run CoreOS Linux in production today. We expect this trend to continue.
I should also note that we have many more team members employed by CoreOS than we did a year ago. It takes time to build the right team, but so far I think we have done a fantastic job. I'm still amazed at the number of projects we actively maintain and ship with a team our size. Now we have the people power to make our projects/products solid.
Thanks for raising the question and hopefully we can continue to answer it with actions and shipping products that you will fall in love with -- or at least get the job done without the pager going off.
Yeah, I noticed that. It almost seems like it would have been better to just glom on to Zookeeper or some other directory service that was a bit more established, just for the sake of being able to work on other things for the last 9 months. Do you ever look back in retrospect on that?
In the mean time Zookeeper has shipped dynamic ensemble configuration which bring parity to the only thing etcd had any advantage in. (Not that it -really- mattered, most people that were running dynamic ensembles were already using Netflix's Exhibitor)
Zookeeper is also faster, has more features and you need it anyway if you are running SolrCloud, Mesos, Hadoop etc.
I'm not sure though that Zookeeper is universally faster than etcd (certainly as you add nodes it slows down... though most people don't need more than 3 or 5 nodes).
Dynamic ensemble configuration is definitely a place where Zookeeper was lacking before, but it isn't the only difference that matters. The big difference I remember with etcd was partition tolerance. Last I checked Zookeeper was all on the consistency side and it's partition tolerance basically just wrote off the smaller partition. Either choice can be a good thing or a bad thing for a directory/orchestration store, but particularly as you scale up to large numbers, partition tolerance semantics seem to better match the use cases.
So I don't think etcd is all inferior to Zookeeper, and I understand the desire to build a system where all components are fundamentally AP, but sometimes having a bit of impurity and some duct tape hackery in your model can help you get out the door faster and focus on more significant technical challenges that will help you build your community. Once you have a community, you have more resources to address that duct tape.
My opinion is that Kubernetes is closer in identity to Docker: tons of OSS excitement, many companies jumping on board, and really pluggable. As to whether its going to become the critical infrastructure piece that gets enterprise to pay $$$... that's a stretch.
Since Kubernetes schedules, I'm curious what makes you say this. Kubernetes without Mesos works perfectly well for the use cases that both CoreOS and Google envision.
And yes, as noted above I work for Mesosphere. YMMV.
Don't get me wrong, I really like mesos. I think it's a fantastic piece of technology and I really want to see it do extremely well. I just think Kubernetes is something a little more accessible to the wider developer community. Plus we get to see it grow from infancy and help contribute to that growth. AND its written in Go, which is a major plus.
Mesos and Mesosphere talk a lot about scale, with the implication that the technology you're modeling powers the entire Google fleet in one big harmonious system. It doesn't. There's a fairly hard upper limit on the system you mentioned in terms of scale, which is one of the motivations for Omega. The system you mentioned also fits into an ecosystem from a DLM[0] to a RPC system to a build system and on up. Pieces of that are gradually being open-sourced, too, like Bazel. Google is clearly becoming aware that their infrastructure being generally available will benefit the industry enormously, and regardless of my personal feelings on their business model and exploitation of user data, that is a good thing. If you're telling me that Mesos can do better than all of that expertise, better than John Wilkes, better than all of that, to the point where you don't even consider Kubernetes on the same path as Mesos, I don't know what to tell you other than come back to reality. This seems to be a rut with Apache projects that model Google systems in some way: Zookeeper, Mesos, Hadoop, Flume. All of them look like if I described the systems in question over a very blurry fax, and the open-source community did its best. That's not outright criticism, it's just having the unique experience of having seen their Google counterparts; it's a good thing they exist, but there are absolutely opportunities to do better, Mesos included.
I get what you're trying to do with Mesos, but I strongly disagree with your characterization of Mesosphere as a datacenter operating system. It's disingenuous, and positioning Mesosphere as the only company who can do hyperscale in containers is a tough sell. Last time I used Mesos, in 2012, it could barely handle 100 machines and I had to write shell scripts to even get a simple static application scheduled. I then threw 10,000 jobs at it and the scheduler hung and took down the entire cluster. Clearly you have come a long way since then, but it's apparently easy for you to forget that given the age of Kubernetes.
[0]: You'll pardon me if the thought of Zookeeper doesn't excite me as the backbone of the entire system.
I'm biased, of course. I work for Pivotal, we're the lead builders of Cloud Foundry. Our current generation of software can spin up hundreds to thousands of machines without too much fuss. Our next generation[0] is ... better.
And again, this is my bias showing, but I think our design for container workload scheduling is better than Mesos and Kubernetes.
We don't get much buzz on HN because we sell a complete PaaS to F500s. It's not very approachable on a personal level.
Lattice is a selection of Cloud Foundry components, principally the Diego runtime system, Loggregator log streaming system and Gorouter HTTP routing system.
If how software is written matters to you, then you might like the way we work. Cloud Foundry is Pivotal Labs DNA, scaled up. 100% pairing, every new line of code is test-driven.
We dogfood everything by running Pivotal Web Services as a public PaaS, which is always within a week or two of HEAD across the board.
Where we follow is that Kubernetes is extracted from real experience and Mesos has a head start on implementation.
In terms of design, I like ours best. Tasks and long-running processes are distributed using an auction mechanism, which means that there's much less global resource-status chatter required to make the scheme work. Diego is "demand-driven", meaning activity occurs when new demands are made. Mesos is "supply-driven", meaning state-affecting activity occurs when resources become available. They both solve a critical problem by pushing intelligence out to the edges, but to different edges.
I am much less familiar with Kubernetes, so I will avoid being any wronger than usual.
Disclosure: I work for "the Kubernetes company", Kismatic. http://www.kismatic.com/
Expect CoreOS to shift resources towards proprietary revenue streams while allowing the community to pick up slack on open projects.
(etcd dev here. We've got a long roadmap ahead.)
To your second question, Kelsey said it best: https://news.ycombinator.com/item?id=9330398
So far the approach seems to be, work on a special project in secret for a couple months that aims to compete with existing solutions and then hype a partial release of the incomplete project on a Monday.
I can tell you first hand the majority of our team members work on our opensource projects. It's one of the main reasons I came to work for CoreOS over a year ago. But the concern is still valid -- will CoreOS really keep funding our opensource projects or will CoreOS pull a fast one and go open core?
So far our actions prove we are committed to opensource and that continues with todays announcement. We wanted it to be really clear that CoreOS is all about opensource projects and collaboration, even if that means our competitors can compete with us, that we took major effort in keeping the two brands separate. There is coreos.com for "Open Source Projects for Linux Containers", and tectonic.com that combines those projects in a commercial offering. There are some non-open bits in the commercial offering, but they don't conflict with the opensource projects.
You are right that we will allocate more resources towards making money, which is required if we want to continue the good work we are doing on the opensource front. You are also right in that the community will pick up the slack, but please keep in mind that CoreOS is part of that community and we plan to continue leading the way.
fleet will exist indefinitely, but kubernetes essentially replaces it with something (promised to be) better.
Tectonic would be closer to RedHat's Openshift project/product, but that's not quite right either since Tectonic is not a full blown PaaS solution.
Granted, but let's be serious for a moment.
Nobody with widely-recognised components is going to just stop moving up the value chain to selling turnkey systems. That's where the serious bucks are waiting.
I expect that before too long, my employers' sales team will be seeing CoreOS and Docker sales folk as well as the existing competition from Apprenda, Salesforce/Heroku, IBM and Red Hat.
(And, again, I don't speak for my employers).