Intel Pulls Out of OpenStack Effort It Founded with Rackspace
fortune.com
fortune.com
The big one is that OpenStack is about building and operating your own Rackspace; this is not something IT organizations can even hire for let alone carry off. The idea that they could/would/should was purely aspirational.
The others are that it's basically a mess - no two OS deployments will look the same - and that it has neither an operational advantage nor too much of a pricing advantage (50% for RHAT) over VMware - which is better integrated and much, much better from a admin experience and debuggability standpoint.
I looked hard at leveraging OpenStack for the service we were building but the underlying code was often cringetastic and somewhat naive. At some level if you run software as a service you can work around poor code quality - something friends from AWS have emphasized - but you can't throw it over the wall and have other people without large engineering staffs there to help run it - which doesn't work in the small. VMware is the opposite approach - the entire model and practices evolved out of arm's reach software sales and support. So naturally it works better.
It wasn't _just_ that Openstack was unreleased and talking about foundations and making grand promises without any production-grade code. It was just that we could never work out which version Rackspace was actually running for their actual customers, and the details of how to actually _deploy_ it and make it work seemed enormously fiddly compared to VMWare, which was where they were taking aim. I expected them to say "now running on Openstack, 100%!"
Also the design seemed to mirror AWS, as if the only answer to their dominance was to ... copy every facility they were producing exactly?
We had a more definite vision of where VMs should go, and thought it was a bad plan to aim at "Amazon, but smaller!". In particular we really really wanted live migrations to be a part of our platform - where we really really cared about uptime of individual VMs, and wanted people to be able to upgrade them on the fly. Plus, y'know, for a hosting company, being in control of, and having opinions of our hosting platform was what people paid us for!
So we designed our in-house platform BigV instead (now Bytemark Cloud Servers) -> https://blog.bytemark.co.uk/wp-content/uploads/2012/12/Desig... [pdf] And even though we (re)invented an NBD server to make all the live migration stuff work https://github.com/BytemarkHosting/flexnbd-c I believe we've ended up with something that does a small number of things far better.
I see people asking whether Openstack is actually running anywhere very public, and have read some pained war stories. So it still feels like the right decision, even before this announcement. Plus it seems like live migration in Openstack is still an exotic and difficult option, whereas that's been our standard practice for years.
There are plenty of problems running your own hosting stack, but I like that we can solve our own problems on our own terms, and tie software decisions to hardware or data centre decisions that we make at the same time.
I eye up Kubernetes & Ganeti every few months, but they keep reminding me that if you've got the expertise, and a specific purpose in mind, you can usually build something that fits your purpose more closely.
(though the second half of that sentence will probably be written on my self-carved tombstone)
I think the thing people - and especially the OpenStack guys - don't appreciate is how terrible it is to (a) lose a workload or (b) require someone visit the datacenter (which may actually be a colo in another state or hours drive away). Having to fall back to some sort of terrible insecure management like IPMI or a dedicated mandatory management network (of questionable security), etc. is just not viable.
Systems and infrastructure architectures need to address a few things that really matter - error handling, continuous self monitoring, state compression and linear, systematic self-recovery - and while some OpenStack components handle this (ceph is pretty great except for a few things around access control and security) the whole doesn't handle them well at all. It's not enough to log a message (or worse, just log an exception stack).
We will hopefully upgrade soon to the Newton release which will enable us to provide SaaS (Murano) and CaaS (Magnum) to the users. We will probably also evaluate Openshift on top of Openstack.
The biggest pain points is the integration with our existing HPC infrastructure (specifically the parallel filesystem/storage). We haven't really looked much at the code except of debugging some authentication issues, when we tried to integrate keystone with our core IT's AD, so we can't comment on that.
Additionally, the official openstack documentation (the commercially supported one is quite superficial) is often very confusing and it's very hard to find proper CHANGELOGS for each of the openstack components. Also we really don't like (or rather despise) openstack's issue tracker (Launchpad). It is very hard to find any existing issues, when we encounter some problems.
We have also undergone upgrade from Mitaka to Newton with zero-downtime and completed within a day (I understand that zero-downtime upgrade is a capability specific to Sardina FishOS). We will be upgrading to Ocata in the coming months.
We have found that in cloud, it is critical to separate the Operator role vs Consumer role. OpenStack is primarily Operator-facing, who are in turn providing the capabilities to serve certain Consumers. OpenStack enables the Operator to flexibly provide virtualized compute environment requested by the Consumer, while the Consumer does not care how the compute environment is provided by the Operator. With this view, lots of things become clear. Granted, not everyone in the OpenStack world have this view, but we are sure glad that we do from the early on!
Our deployment is not the largest, but we aren't small either (several hundred hypervisors). We are an organization of several 10's of thousand people, though the user base at the moment is still in the hundreds.
A number of comments here mentioned that they had live migration problems. We have not had these problems. It worked from the very beginning for us.
Our view was that we would work with a vendor on OpenStack, rather than trying to string together the parts, in the same way that we don't string together the parts that make up a Linux distribution. FishOS makes it simple for us -- and that's before looking at operational tools in the product.
I have not been involved in the commercial terms, but my understanding is that commercially we have TCO that would be hard to match otherwise.
People try and deploy OpenStack from sources, or even from Ubuntu's etc packaging.
This isn't how you should do it.. You need an opnionated vendor who meets your needs.
Is this related to some endemic problem with the technology, big egos within the project, dominance by another outside player or what? Cursory searching doesn't reveal any "big thing". Maybe it's just dying a death by a thousand cuts.
1. We thought that private clouds were generally valuable. In 2010 it seemed like everyone was going to have a private cloud. It turns out if you are a midsize business, having a rack of "cloud" doesn't really offer you much benefit over having a rack of managed vms. You still need to pay someone to manage that rack. The vast majority of the benefit of cloud comes from having a ton of workloads, which means a public cloud or a huge service-provider sized business.
2. We focused on community building by supporting all use cases. The best way to build an open-source community is to bring everyone in. The community grows fastest that way. Unfortunately, it also means that the product quickly becomes an unfocused frankenstein that is decent at everything and good at nothing. Projects like docker and mesos are suffering from the same problem.
The end result of these mistakes OpenStack is good for certain use-cases. It does really well in large companies that need a public-cloud like environment to manage their infrastructure and can hire a team of people to manage it (e.g. comcast, verizon, e-bay, wal-mart).
I don't know that the exodus is due to endemic problems so much as the market finally waking up and realizing that public cloud is the future.
You folks popularized a misconception and hoped to monetize it in perpetuity and were eventually corrected by reality. So goes all big $$ software.
Get ready for the next big thing and first step that...ad nauseam.
The problem with Openstack is it had too many moving parts, and no standard/reference implementation. It was destroyed by it's own grandiose goals. The project wanted to say 'yes' to every permutation of software. Making it a disaster from an operational standpoint. Everything promised, but nothing verified.
Visible also in things like the installation process (which for ages didn't have an "official" answer, so you had pay one of the big vendors to do it if you didn't want to try automate it yourself)
It's a PaaS for IaaS foundation. Unless you have the resources to line up and support Openstack, I recommend just build on AWS/Google/Azure/Rackspace/Heroku/DigitalOcean, you name one.
I wonder if big customers such as Bloomberg ever regret / think of moving to public/private IaaS.
Instead it was a vehicle for hardware vendors to sell compatible product and consultants to build something they'd have to service forever afterwards. (Yes this is a cynical and half joking take, but it's also serious)
So what went wrong? Welll, If you're not going to build a product that teams can actually figure out how to use, then you need to have Cisco or Oracle level sales talent behind it. OpenStack didn't have that, and maybe couldn't have it because there wasn't one boss company behind the product.
Like I mentioned, there is a trade-off between the usability and required Sales Power. As far as usability goes, even consultancies that were directed to use and sell OpenStack solutions couldn't get their people to figure it out. A consultancy based technology has to be coherent enough that a junior associate can reasonably start billing hours for it. OpenStack will stymie a close-knit team of veteran "devops" people— And I'm talking about people who actually know systems administration and development, not a buzz word ninja.
Everybody got that completely wrong. Open source works, but you still have to understand how to engineer it. So far, companies that succeeded are mostly following an "open core" model where they control the core too (Atlassian, Automatic). Red Hat is the exception, and with Red Hat putting a lot of effort on upstream OpenStack development no one could pull off open core for OpenStack. Mirantis has plenty of non-upstream code and last I heard wants to sell SaaS to work around that.
Plus, no one devoted the right amount of engineering to the lower levels of the stack (at least QEMU and libvirt). You cannot seriously expect to partner with vendors for guest operating systems without knowing all of the stack---this was something else that Mirantis and others missed.
So the problem was not that OpenStack was bad, it was that everybody thought open source is magic. But it isn't magic, and if it looks like magic, it means you don't understand it.
IMO, OpenStack was sort of a consortium effort to compete with VMWare, AWS and Salesforce, but it's operational model is closest to AWS. Nearly all the big enterprise IT companies tied their rafts together hoping to stave the flow of customers. It doesn't appear to have worked.
RackSpace is advertising AWS migration and consulting services on podcasts I listen to. Not exactly a stellar testament for their own cloud product, but I think they still participate.
HP's strategy is a bit haphazard given the recent corporate splitup. On the plus side, the HP/HPE split seems to have allowed HPE to ditch all the weird www# URLs. They retired their public cloud, but still sell hardware that can run OpenStack at least.
Dell bought EMC, and proceeded to shut down their OpenStack offerings in favor of VMWare, which they own a substantial fraction of.
IBM is in a continual process of downsizing its hardware divisions in pursuit of higher earnings per share. It seems their big push is cloudfoundry, which seems to be more about containers and k8s.
So why are they bailing? I'm guessing:
- it's way harder to hire support staff for OpenStack, than VMWare - AWS reduces capital costs and upfront investments - Customers on existing solutions aren't prepared to take advantage of new opportunities - Many of these companies have existing product lines they don't wish to disrupt
Most of IBM, Intel, HPE etc have thrown in the towel and now offer their own services on top of Red Hat OpenStack.
OpenStack has now found itself beyond enterprise, and now being the de-facto platform for NFV running mobile networks, and I guess Red Hat are becoming the winner here as they are so used to supporting an OpenStack 'type of' infrastructure for large bodies such as banking, telco, health etc. When you consider Red Hat are already large well established contributors to all of the layers of the OpenStack 'stack' such as KVM/QEMU , libvirt, the kernel itself, + overlay networking tech such as OVS, and now DPDK, you can see why they are well positioned to support and run OpenStack clouds.
[1] https://www.theregister.co.uk/2017/03/28/red_hat_cloud_quart...
Given all of the inter-dependencies between OpenStack services and lower level Linux constructs and software (OVS, Ceph, QEMU, etc.) those who were already used to distributing and managing these already turned out to be the ones who were best able to make the combination work.
That said at least some folks who were working on OpenStack and related technologies at Canonical are among those now looking for new gigs:
https://twitter.com/zulcss/status/852542117566189568
Thus while cloud remains on their priority list it is not immediately clear how big a part OpenStack will play in that going forward for Canonical versus solutions that run on the reigning public clouds.
Canonical seriously lacks expertise in KVM (+QEMU+libvirt), and you cannot take that for granted when you have a customer with VMs crashing that is asking for a fix.
FWIW, I recently had to explicitly inform the Canonical Virt maintainers (of which there seem to be very few) about which patches that ought to be backported to fix a bug in one of their libvirt packages that was seriously affecting (i.e. preventing from patches being merged) OpenStack upstream CI environment.
The said bug had fixes already available upstream, and are straight backports with no conflicts. No one bothered to do the "unsexy" work of backporting & cutting a quick stable build.
Only after pointing out the commits (with help from one of the lead upstream libvirt maintainers), and posting them on a LaunchPad bug, did the Ubuntu maintainers stepped in to backport the said fixes.
Cloud Foundry is built on technologies that predate k8s, but yes, containerisation is the unit of operational currency.
Insofar as Dell EMC has a multi-play, it also includes Pivotal, which is the leading contributor to Cloud Foundry. Unsurprisingly it runs smoothly on VMWare, as well as AWS, GCP, Azure, OpenStack and I forget what else.
Disclosure: I work on Cloud Foundry on behalf of Pivotal.
Just conjecture on my part, but I would have serious reservations about that part of their service catalog right now if I was vendor shopping.
Their own cloud isn't mentioned on it. They've obviously been putting their eggs in the consultancy basket for 6 months now,
Hell, even if you go to cloud in the menu, they're only advertising other clouds: https://www.rackspace.com/cloud You have to go through the 'Infrastructure' menu to find their own cloud offerings.
I'm wondering how long before we get notice we need to migrate away from them due to impending shutdowns.
Devstack works. Sort of. Poorly. Everything else is pay-a-consultant territory.
OpenStack refers to a number of different projects (Nova, Neutron, Horizon, etc). I would advise getting a high level understanding of how Nova, Neutron, and Cinder/Swift interoperate. Then drill in to Neutron. As with every other cloud platform I've worked with, most of the IaaS complexity lies in Networking, in my opinion. Neutron, and SDN in general, is a fairly complex topic depending on your experience with network architecture and administration.
So for example a development environment I use is a docker-compose environment building Monasca. That environment is a work in progress, so half the time I'm fighting that environment just to make a proper change.
Then from there, I don't have Horizon, so I have to use command line apis directly to set everything that I wish to use.
The whole thing has just been a challenge, every part I've had to learn in some respect. The actual code I've changed in Monasca, has been the simplest part so far. Validating it's correct, and testing it has been the far harder challenge.
Monasca also being a bit of a challenge since it's several different repos of services working together, instead of a large one akin to Nova.
I'll get it, but it's been a struggle so far.
Take a look at the sponsors for KubeCon and Dockercon and consider that many of these same companies were those originally trying to catch the OpenStack wave, and then ponder how many of them will be extracting themselves from those projects in 5 years (or at least lowering their investment, which as we see here is generally perceived as extraction).
Prominent folks in the Kubernetes community will tell you that it's different but the recent conferences sure have a similar vibe to OpenStack summits circa ~2013 when the marketing really started to overtake what the technology could deliver.
I believe the decline of OpenStack is inherent in the type of technology. I don't believe that type of "infrastructure plumbing" software lends itself to high-speed high-quality innovation in the "open source" model. Some reasons:
1) it's boring infrastructure software: Because it is software for "data center" infrastructure, it's not glamorous and interesting enough to attract programmers who don't interact with that world. Other successful open source projects like LAME MP3, ffmpeg, Linux kernel, etc have programmers that can experiment on laptops and ordinary users that can download and play with it.
2) no abundant source of high-quality code contributions: The enhancements you want contributed into OpenStack should be provided companies running complex private cloud operations. They would battle-test the code and that knowledge would trickle down into the open source contributions.
However, a company that's facing the prospects of a complex private cloud would more likely choose Amazon AWS or Google Compute platforms instead of tackling OpenStack.[1] This removes a subset of potential OpenStack contributers. OpenStack is so complex to run/customize/maintain that it requires building up an internal staff with skills similar to AWS/Google/Facebook's datacenter engineers.
If the company does decide to run OpenStack, it's likely they won't contribute back any source code. (Same as Linux situation where most users of it don't contribute source code.) This removes another subset of potential contributors.
Probably the best "private cloud operating systems" are developed by Google/Facebook/Amazon/MS for their own datacenters and those 4 companies are not contributing to OpenStack. Wal-mart has sophisticated datacenter operations and they may be the largest Openstack user but their contributions to OpenStack do not match the new innovations of AWS mentioned at re:Invent[2] or Google CloudNext[3]. Yes, Red Hat is a sponsor of OpenStack but their internal cloud operations are not stressed like AWS/Azure/GCP with customer-facing challenges nor do they manage IT complexity like Wal-Mart.
Sometimes, commercial versions of plumbing software do lose to open source projects. One example would be Microsoft's Dryad "distributed computing" losing to open source Hadoop even though MS had a 7 year headstart. However, Hadoop+HDFS is still not as fast or reliable as Google's internal distributed mapreduce. (Maybe the idea of buying a hundred expensive Windows Server licenses to satisfy a 100-node Dryad cluster contributed to the market ignoring it.)
Based on history of how certain open source projects fall behind proprietary counterparts (such as OpenCL not as cutting edge performance as NVIDIA CUDA or OpenGL not matching MS DirectX features[4]), OpenStack seems on to on a similar trajectory of "not as good as the proprietary alternative".
At this point in time, the "private cloud OS" space is moving very fast (possibly faster than flavor-of-the-month Javascript frameworks!) and the open source development model is not able to keep up.
[1] http://www.computerworlduk.com/cloud-computing/guardian-goes...
[2] https://aws.amazon.com/new/reinvent/
[3] https://blog.google/topics/google-cloud/100-announcements-go...
[4] https://softwareengineering.stackexchange.com/questions/6054...
From my perspective, companies that decide to run Openstack do contribute code back, but the process is so long and arduous to get code accepted that it's not hard to see how groups would just give up and just merge upstream to their own.
Core reviewers have to accept your code, and they are on no timeline to do so. Months will go by without a review.
I think that's a slightly different reason because OpenCL is just a spec with no real teeth. NVIDIA/MS have a pretty tight grip over the market both in h/w and s/w, and creating a CUDA like spec does nothing meaningful if there is no alternative source for hardware and software. Everything including drivers, libraries, compilers etc. is closed, whether it's CUDA or OpenCL. So aside from some vague promise of portability, OpenCL doesn't really open anything.
With the slowdown of single threaded scaling Intel knows that performance improvements for consumers in the near future will be increasingly anemic. Servers and ML are two spaces where increasing core counts still makes sense.
Id' be game if you can find a less spammy site. But when I submitted this an hour ago, there wasn't any.
Open governance by individuals is much more stable. That stability benefits both individuals (whose stake is not vaporized when they change employers) and businesses (who are more insulated from impulsive moves by big players).
I think the mistakes made were:
1) the so-called big tent approach 2) too much complexity 3) core projects not listening to end-users, and focusing too much on the plug-in model
It is a shame because there were so many smart and hard working folks involved. It really felt like a community experience.
Unlike Openstack, it has a proper scheduler, and lets you rebalance VMs across hypervisors efficiently. Also unlike Openstack, it can restart VMs if it (or the hypervisor) dies, if you've enabled that.
And completely orthogonal to Openstack, it has very strong consistency guarantees. It's not made to start 1000s of VMs in seconds, since each master node has to agree on all decisions, and each operation typically "locks" the involved hypervisor. On the other hand, I haven't been able to break it once in over six years.
Note that it really just exposes an API and comes with a superb command-line client. Some assembly required.
Source: deployed Ganeti with great success at a billion-euro company, moved on to promising "cloud" project which insisted on using Openstack and promptly quit after a year of fighting obscure bugs (and naive colleagues who did not want to try anything else :)).
(If you'd like help deploying it, I'm available!)
It's been super stable and great for us. Plus #smartos has been incredibly responsive and helpful with any problems we run into.
There is also Proxmox and Cloudstack.
I wanted to dive into Joyent but their wiki was skeletal.
To make my comment more useful: I'm hearing good things about Proxmox and will likely lab it up soon. r/homelab has some feedback on it as well.
https://docs.joyent.com/private-cloud
Anything missing or questionable could usually be resolved by looking in the docs folder of the corresponding service's github repo.
We believe OpenNebula is a very good alternative to OpenStack. It's a completely open source product, with many strengths. Very easy to setup and to maintain, without the big mess that is required by OpenStack. Plus, users don't get caught in all the politics (lots of vendors pulling in their direction).
I certainly recommend you check it out.
It's more a competitor to VMware clusters though, or was last time I played with it (a few years ago).
It has a decent rep.
Please. You seem to be making the same mistake, that many make on blogs and news websites, of exclaiming "containers are the future!" while both VMs and containers have their place and use-cases.
I'd rather suggest you read the below. (Since I'm linking to a blog, quick background about the author: Rich is one of the long-time contributors of KVM-based Virtualization stack and the lead author / maintainer of libguestfs.org)
https://rwmj.wordpress.com/2013/06/19/the-boring-truth-full-...
The thing is, though, they are at odds. Enterprises don't have nearly the dynamic workload set that people thought but where they legitimately have some is in devops style development. And if you have container management, which you do, then highly dynamic VM management is much less valuable.
Most of the customer sites I interact with on the high end just end up baking their own system images tied to a SSO solution and just manage it with VMWare/Ansible/something homegrown. On the low end people just don't deal with OpenStack at all and just do it the hard way with a VMWare or KVM style solution. For either side of that equation containerization is a faster value-add.
From the other side what people ask me about cloud integration is more feature-driven of particular clouds than anything else. I'm just not seeing the momentum in the field.
From that perspective, container solutions are delivering a better developer proposition than OpenStack has yet managed. There are ways now to build container clusters that you can ship in parallel to AWS and Azure with very little code difference.
In that earlier discussion I was skeptical of OpenStack precisely because of its focus on infrastructure first. Without the buy in of being a clone for a specific cloud structure (AWS compatibility over anything else, for instance) or the backing of traditional datacenter/server vendors (IBM who eventually started into BlueMix; Microsoft whose "on premises Azure" is now firing on most cylinders but was announced as a plan early in OpenStack's history), OpenStack didn't seem to have an obvious niche in the infrastructure world. The closest to a niche it might have had in its early life was the promise of application portability between clouds and that never quite seemed to be delivered.
I can tell it frustrates infrastructure folks to hear that containers have been eating OpenStack's lunch, but that is the very real case from the developer perspective. As a developer today, I go for containers and OpenStack is no longer relevant on my radar. Sure I can run containers on OpenStack, but containers abstract away more of the infrastructure and I have less and less care what cloud(s) is underneath the container cluster. When I asked OpenStack people what OpenStack might deliver to me that vision of application portability was tantalizing but never seemed quite finished; container technologies have actually delivered that.
> There are ways now to build container clusters that you can ship in parallel to AWS and Azure with very little code difference.
OpenStack is an IaaS, so just like AWS. Focus on the context. Do you want your own private cloud? Yes or not?
If no, then this discussion can end, because AWS and Azure run on their proprietary IaaS code. As a customer, you request resources from the IaaS layer, and you build your server/platform from that point.
So arguing that container clusters can ship to other clouds with very little change (almost likely writing the APIs to create a container) is unfair in the context of why one would choose container over OpenStack. The purpose is different.
If your answer is yes I am building a private cloud, how are you going to do that with Docker alone? Can you build a software-defined network with Docker? Absolutely not with Docker since Docker is a host-based deployment solution.
What you are looking for is ability to create OpenStack the same way CloudFoundry / Kubernetes are created. You write up a manifest, and the necessary databases and services are deployed to some EC2 machines. In the case of CloudFoundry, you write up a manifest file, describes number of instances, types, credentials, what not, then call Bosh to create Cloud Foundry (will create a pool of app machines, router servers, UAA, etcd etc). Machines are created based on a stemcell, basically an image. You want to create your IaaS based on images. You want to be able to script up a manifest and deploy your IaaS. You want a lift and drop IaaS infrastructure. Container can do that, but it cannot be done simply with Docker. You need that infrastructure abstraction layer to cover up. That's probably what Docker Enterprise Edition might do, but I have not really dig into it yet.
As a software developer, do I ever want a private cloud? No. Does my employer? Maybe. Is it my job to tell them how to invest their infrastructure dollars? Quite possibly no, because software development and infrastructure are typically held at arms length. But even when they are not in a "proper" DevOps shop, the ballgame of which cloud is then subservient to developer convenience and how easy it is to deploy software to a cloud and how productive developers are writing software for that cloud.
So yes, the purpose of OpenStack and Container technologies are very different and I appreciate that technically. In terms of real world value to me as a software developer, however, I have platform problems not infrastructure problems. I don't care what the infrastructure is under the service so long as it provides a stable, reliable platform for me to build upon. Containers abstract that for me in a way that solves real platform problems that OpenStack was only ever relevant to me in so far as its ability to once hint at a possible solution to. That's not fair and that was expecting too much from OpenStack at the time, but that's life.
Of course, I would advise against running a private cloud unless there is a dedicated team of at least a dozen or so. I applaud Digital Ocean for able to survive and make good business from their private cloud. As a developer I totally agree I just want my code to be deployed and that all the appendices are deployed and configured.
OpenStack is at the same layer as AWS, GCP or Azure. It's not an abstraction over them -- that's a PaaS. I'm not sure OpenStack was ever seriously intended to work at that level.
I work on a platform, Cloud Foundry, which provides the kind of portability you're talking about. Red Hat have OpenShift, which is another such platform.
Disclosure: I work on Cloud Foundry on behalf of Pivotal.
Not that asking AWS+GCP+Azure to commoditize themselves in such a way makes much economic sense, but that's the proposition.
Cloud Foundry gets around this by using BOSH for deployment; it essentially creates the kind of layer you're describing.
There was a hint at some possibility that it might, especially in some of the hype circles early on. It is of course unfair to blame OpenStack for never actually fulfilling that dream, when that was not its intention per se, but that is part of why today people continue to compare what OpenStack promised versus what containers delivered.
It's frustrating to read people comparing container technologies or worse container schedulers with OpenStack. Apples and oranges.
If you are planning on running containers on premise you need something from managing the IAAS side of things. OpenStack is excellent at IAAS and is the foundation for running higher level abstractions and services on top.
I disagree with you that you need an IaaS solution to run containers on-prem, but realistically, yes you do need an IaaS solution
A few years ago, I leveraged a bunch of methods in OpenStack to build a thing which would launch instances based on Bitcoin payments to certain preloaded addresses. These addresses allowed "templates" to be associated with cluster capabilities and code to be deployed. Payments to that address in Bitcoin would immediately net you an instance, of a certain type, on someone's infrastructure.
The general idea behind this creation was a way to abstract hardware components into a system by which applications could launch themselves and utilize the resources provided in a fair, secure and trustworthy way. It is my belief this model was WAY ahead of it's time, a precursor to the hybrid models we see emerging today, and lead directly to my personal realization that federation of all systems will be a basic requirement for implementing trusted computing in the future. We're going to need it with AI. Not sure how I know that, but there it is, irrationality and all.
Unfortunately, standardizing deployment methodologies doesn't net you federation. Standardization itself doesn't net you trust, unless everyone can agree on the standard, which is out of necessity done in an non-trustworthy way. Votes of a board, for example, aren't represented with fair consensus, unless there's an algorithm behind the votes. Otherwise, votes end up being slightly irrational, because people behind the votes are slightly irrational. Or very irrational, depending on the company they work for.
And yes, federation can be implemented in a trustworthy way by a corporation, such as Google or Amazon, but that requires all people in that federation (a discrete group) agree they will use this thing to implement trust, even though the thing may not actually have trust implemented in a rational way (i.e. by algorithm).
At the end of the day, OpenStack was doomed not because of container tech, or the structure of the board, or who ran the biggest cluster, or because it was overly complicated for simple use cases.
It failed because it did not implement the basic requirement of delivering trusted infrastructure in a scalable and trustworthy way across a broad range of infrastructure, in a wide range of locations, and do so in a way that separates the use of the infras from the irrationality of humans running the infra.
Until something does that, and does it well, we're stuck with Google, Amazon and other provider's solutions. This is also a good rationalization for the continued increase of cloud services by companies and the continued emergence of hybrid models in the future.
The most likely upgrade path for us right now is VMWare 6.5.
I always wonder if one of Openstack's weaknesses was also the same thing that was it's strength, it's openness and need to support everything, as well as remaining agnostic to users. Nearly every company I know needs to register VMs on a local DB of some sort, and has patches on Openstack to do that. But every time I went on IRC and asked if we should think about a nicer way of providing this hook, I'd either be redirected to a different project, or get some rude replies.
I also wish there were some more modern companies behind the project; a lot of the developers were probably under pressure from penny pinchers in the management.
That said, it was a terrific project to start my career on.
Frankly, we're doing a piss poor job of it. Migrations don't happen even if manually ran, the compute nodes are already at capacity, even with the meagre VMs we use.
I have kept pushing for Mesos myself. I see it as a good way forward. Openstack may be a technical good body of code... But in the end its too convoluted codebase and many moving parts to be effective in most deployments.
Like Openstack, Mesos brings its own set of new complications.
Unlike Openstack, Mesos almost always increases efficiency of operations once it's in use. Openstack always seems to result in more and more effort spent on just operating Openstack...
But looking at it, there are also less moving parts with Mesos compared to OpenStack. As a test, I set up a Mesos Cluster in 3 VM's. It wasn't terribly hard at all.
But at work, we had an alarm thrown in nagios about critical ram consumption on a node. So I go and do the migrate. It works.... And it doesn't. It migrated the VM to the same node. I had to do 2 extra flags to tell OpenStack to migrate the node properly. That's just ridiculous, and dumb.
And not to mention, but Mesos supports proper handling of VMs and migration. It frustrates me to no end the amount of hand-holding I have to do with anything OpenStack based. Whereas Mesos was "Install, configure, run".. With a large amount of work on Configure.
And then you get to restart everything. Yay.
Whereas if Kubernetes is on something else (OpenStack/Mesos) you at least have control to restart Kubernetes when it shits its brick.
For control we are using CoreOS with kube-aws. It actually works quite nicely as we can scale and upgrade running clusters. The only issue with it right now is upgrading and managing etcd.
I think the whole issue with OpenStack is the overambitious goal to be a one stop from bare metal servers and switches to applications. Kubernetes with Flannel or Calico can take care of the software level networking and provisioning, while there are a lot of other ways to provision VMs for running Mesos or Kubernetes depending on where you are deploying.
In other news: http://i0.kym-cdn.com/photos/images/facebook/001/044/247/297...
Spare the drama dude. Just because something doesn't work for you doesn't mean it doesn't work very well for others.
When you say VMware 6.5, are you referring to ESXi aka vSphere Hypervisor? That is not a 1 for 1 replacement of OpenStack.
No, it's not a 1:1 replacement. It's a migration. It would also be a migration if we decided to upgrade OpenStack, because we'd be going Havana -> Ocata.
However, there are defined upgrade paths for ESX, we already have ESXi/vSphere in our environment, and we can hire someone relatively inexpensively to help us with vSphere if we need to augment our staff. Folks who understand Openstack are not anywhere near as easy to find and pay.
The team was essentially dissolved several months later, as I knew it would be.
But for the short period I was there (about 2 years), it was a great place to work, and a high point in my total career. Prior to that position, I had been doing web development in PHP pretty much exclusively. Doing server automation was a completely new space to me.
Soon after starting, our team was tasked with a migration to OpenStack. Since our current infrastructure front-end was already based on PHP, I got tasked with looking into how or if we could use PHP OpenCloud to work with OpenStack. It seemed workable to me; I was able to extend the classes in such a way using namespaces and other techniques so that we could add additional capabilities to the interface that weren't already supported (and there were a lot of holes to fill!), but wouldn't break things if/when we had to upgrade OpenCloud (this was ultimately tested a couple times I was there, and the changes proved flawless - at least on the PHP side).
Ultimately, we had a nice stable front-end that worked well with both our original system (some VM architecture that I forget) and the new "cloud" infrastructure based around OpenStack. In effect, a user could deploy either an actual server (if available), provision a VM (if a server was available), or build a cloud server "system" from a myriad of parts (we tried to support and provide as much access to the OpenStack stuff as we could). We also had a RESTful API for clients to also used (some of our clients resold our services under their own names). Some of the backend stuff was a bit "messy" in how it worked (I won't go into details, but I "authored" a fake "O'Reilly" "book" (really just a front "cover" mainly) whose mascot was dickbutt) - but despite the mess, overall it worked well, considering all the moving parts (where it would tend to fall down - not always, but enough - was when an upgrade to OpenStack was performed).
In short - we were also one of the few companies running OpenStack in production. Our owners ended up selling to LeaseWeb, I left - but the idea was that LeaseWeb wanted to transition things to their API and system, and I honestly don't know what happened with all the work and such I was involved in on the PHP side of things (there was also a point where me and a coworker had to quickly ramp up and learn GoLang to make an interface from Rancher/Docker over to the Nobis API - that was a fun and interesting experience). I imagine that some portion is still running, but who knows.
I personally think that in the right hands and with the right infrastructure OpenStack can be a very workable and working technology. It seemed to work well for the systems we used while I was at Nobis. I don't know honestly whether you could use it to scale up to anything like AWS or Google's offerings, but I think for medium-sized stuff like we were doing (or like Digital Ocean does - who at the time was our direct competition), it can work well - at least as I experienced things.
When the previous engineering team was shown the door, the entire system had no maintenance and no path forward. It's an epic management fail, but ... let's just color me cynically and sarcastically surprised that I used "management" and "fail" in the same sentence.
That said, OpenStack is an ambitious product. It essentially seeks to give you a lot of the functionality of AWS, except with on-premise infrastructure. That is a tall order. AWS is a multi-billion dollar effort that I assume has has far more engineers than the OpenStack project.
OpenStack is a complex solution to a very complex problem. Having worked with other commercial IaaS solutions, I will say I hands down prefer OpenStack to the others I've worked with.
If there is a private cloud solution that delivers the same/similar functionality of OpenStack with less admin overhead and more simplicity, then I want to hear about it.
I worked as a dc tech during the .com bust recovery period and the level of incompetence and the personalities I encountered led me to leave < 6 months.
Any decent SA from my generation can code a one-off virtualization base without too much problem using kvm/qemu. LXC is a clean in-fit to that. Sounds like this is standard RS procedure. Fleece the idiots, use the willing, and promote the owlshit.