The Guardian goes all-in on AWS public cloud after OpenStack 'disaster’
computerworlduk.com
computerworlduk.com
X to Y Y to X A to B B to C
MongoDB to this MySQL to that Why we switched Why we switched back
blah blah
It's one of the oldest themes in computers - man, this software is crap - that new software will fix all our problems! Until you find the problems with the new.
Precisely the same age is the new software vendor harnessing all that negative energy about the competitor and megaphoning the "man, the new will fix the old, we're so excited!"
Having said that, I have worked with the compute servers of all the major cloud vendors and Amazon must be credited for the quality and consistency of its AWS systems, and Google and Microsoft's cloud computing is in almost all respects equally good - there is as far as I can tell absolutely no reason to choose Amazon over Microsoft Azure (yes, even for Linux systems) or Google Compute Engine. In some areas of functionality GCE and Azure I found to work much more easily and smoothly than Amazon. When I went to work with OpenStack I immediately found the lack of completeness, inconsistency and lack of polish characteristic of many open source projects - trying to do simple things instantly became hard problems.
This way of thinking isn't exclusive to software.
Anyhow its especially prevalent in software. Just search Google for "why we switched from".
If anyone is thinking of writing yet another "why we switched from X to Y" blog post, I can save you the effort of a long blog post explaining the reasons. It's because "the old software was shit and the new is fricking awesome!" You don't need to explain the details.
I mean everything. Almost literally everything.
The most recent thing I can think of is seating arrangements.
At my current workplace seating arrangements are mostly random. Someone here had the idea that we should group people by seniority so that people with related issues will be in close proximity.
However, at my previous workplace we grouped by seniority, and someone had the idea that we should mix it up so that project teams would be in close proximity.
[edit]: And lets not forget outsourcing vs. insourcing, grouping departments by geography vs. grouping departments by business type, etc., etc.
i often wonder if they actually believe what they are saying, or are just ... going with the flow and doing their best to look busy
They probably believe it. I fully expect that I've actually done it before without even realising.
Is it good or bad? It means that things are changing, and they are not going back to the same situation as ten years ago. It is a matter of evolving. There are new insights, new technologies, new people.
Sure, it's a universal theme that people can get disillusioned with the promises of software (or any technology) but I don't think that criticism is relevant to this particular story.
What I read in the article is that a few years ago, their hardware was end-of-life (we can speculate that as end-of-lease, or end-of-maintenance, whatever). It forced them into a big fork-in-the-road type of decision: do we attempt to build out an internal cloud or go with AWS? They decided to invest in their datacenter and do it themselves. It turned out that their internal engineering capability could not match the innovations of AWS. The Guardian isn't hoping for "magic software". Any CIO choosing AWS will know they'll still have "gaps" in functionality. (Netflix's OSS portfolio built on top of AWS with its staggering array of "housekeeping" software is a good example of highlighting those gaps.[0]) Instead, their multi-year "experiment" told them that their home grown team could not keep up with AWS.[1]
This doesn't seem so strange since very few companies could hire and maintain the IT competencies to match AWS features even with OpenStack as a foundation. Facebook Inc and Google Inc have the hard core staff to innovate on their proprietary cloud stacks but The Guardian is a newspaper and not a technology firm.
Maybe a lesson here is that OpenStack requires a commitment to supplemental software engineering that's beyond the reach of non-tech companies that treat IT as a "cost center". WalMart could be one of the few non-tech companies that would be able to innovate on top of OpenStack.[2]
[0]https://www.youtube.com/watch?v=R2kKmMyqTfc&feature=youtu.be...
[1]from article: “We didn't manage to deliver self-service, we didn't manage to deliver decent load-balancing or autoscaling and actually all the benefits we get from AWS we simply did not get inside of the cloud that we were building internally,” he continued.
[2]http://www.infoworld.com/article/2890873/cloud-computing/wal...
That they're having problems getting load balancing working properly is a real sign that their ops team just didn't know what they're doing.
Second that.
Maybe there are only ten companies in the world large enough to have positive ROI to build an internal AWS, like Google, eBay, Facebook, ... who don't want to depend on Amazon.
Ultimately you can always diagnose failures as "they didn't know what they were doing". But usually there are more nuanced explanations for wrong directions. For example it might be that OpenStack seemed to be heading in a better direction in 2012. Maybe it was reasoned that it's worth it to hedge against AWS lock-in and you can always easily fall back to Amazon, or even mix and match AWS and OpenStack. Etc.
If OpenStack worked well I don't see any inherent reason why you'd have to be bigger than Guardian to use it for autoscaling and load balancing.
My issue with their description is that they seems to have tried to essentially emulate what is a very complex platform with the help of OpenStack, which is a very complex platform in a situation where they almost certainly didn't actually need more than a fraction of the functionality.
> If OpenStack worked well I don't see any inherent reason why you'd have to be bigger than Guardian to use it for autoscaling and load balancing.
On the other hand, if that was all they were using it for, OpenStack is massive overkill and there were plenty of simpler options.
I've been in a similar situation in a small research group, but AWS doesn't fly for privacy and regulatory reasons. Managing hardware and infrastructure is a huge amount of overhead that doesn't directly relate to the core mission.
Yes, but that's a false equivalence. You don't need an AWS work-a-like unless you plan on competing with AWS. So much of AWS is dedicated to doing things you don't need to do in most single tenant systems.
But if managing hardware and infrastructure is a huge amount of overhead, what in the world are you doing? If it's more than sliding a server into a rack, hooking up network and power, and leaving it to itself until/unless there are hardware failures or SMART alerts, then you've missed out on automation steps somewhere.
For server environments I set up, I spend perhaps on average 30m per server in a datacentre per year, including travel. On top of that we estimate on average perhaps 10m of remote hands per server per year.
As for load balancing I'd believe they couldn't get it working within Openstack. We're trying to deploy OpenStack at work and the the number of things that don't quite work are a pain.
Both components here (bootstrap script + resource allocation) can be small. The most recent bootstrap script I wrote was 60 lines.
For resource allocation, you can go insanely complex (in which case you should probably consider AWS or installing OpenStack or similar after all), or very, very simple. I've run off the shelf solutions for it, and I've written my own in a couple of hundred lines - it very much depends on your needs for a specific platform.
What almost nobody needs, is the complexity of an AWS-level platform when setting up a single-tenant cloud where you know and control the requirements. Public cloud providers need the complexity because they need to be able to support a vast number of different customer requirements. When you don't have that issue, a ton of complexity falls away.
> then they would have not been able to keep up with companies that did go with AWS.
They're a media company, not a cloud hosting provider. They need to be able to support the functionality they need internally, nothing more.
> As for load balancing I'd believe they couldn't get it working within Openstack.
I'm not surprised. Which raises the question of why the picked Openstack in the first place. Clearly it was not evaluated very well.
Yeah, I am sure it is quite easy to make them agree on the directions OS should take...
A couple of years ago I experimented with and compared some open source cloud platforms. Admittedly my knowledge is old, but alredy then I was wondering the hype and visibility of the OpenStack project. While it has many big-name supporters that guarantee its lucrativeness for the enterprise, its feature set and flexibility was seriously behind other open source alternatives.
Back then I fell for OpenNebula because even though it had its warts it already delivered many features (especially concerning hybrid clouds and heterogenous virutalization environments) that OpenStack still had on its future roadmap.
In a former life, I was frustrated by the futility of trying to be involved with the Openstack project. My next phase was to warn people away.
Now I wonder why I bothered. Years later it's apparent that I was right, but what benefit do I see from that? Better to aggressively ignore bad technology and spend effort on the stuff that looks promising.
Networking which he also mentions in passing is also being rewritten (https://www.openstack.org/summit/vancouver-2015/summit-video...).
I generally agree with his final points - OpenStack really needs to work for the end users, not for the many competing vendors working on it.
Dell tells the story. February 5, 2013: Dell announces plans to go private. May 20, 2013: Dell abandons Openstack.
IMO OpenStack is effectively where 80s/90s Unix was. Either Linux/BSD will emerge from it, or the private cloud vendors will Windows it.
1. "We have switched some components in our software stack" without much info is not an interesting story to read.
2. I have friends who work on OpenStack in RH who would I'm sure be very interested to know the sort of troubles that The Guardian had so if there is a usability or functionality issue it can be addressed.
I mean geesh, people have been building small clouds since there were servers. That's the way mom and dad did it, and by gummity it oughta be good enough for you. The "new" stuff was autoscaling, PaaS, and so forth.
So build out a few servers for content management and publishing, then write very small amount of code to push what you have out to a CDN. If you want realtime data capture, capture it using AWS (or whatnot) and pull it back locally.
I'm not saying that's optimum for every solution, just that the all-or-nothing kind of thinking is probably what lured them into building their own cloud in the first place. You need a cloud for some stuff, so use a cloud. But you don't need a cloud for every freaking thing the company does. Its assets in the form of text content, internal docs, and branding are probably extremely small in modern terms.
There are so many open projects in the cloud space that one would think it's a solved problem.
Here's a list off the top of my head:
1. Docker - Container implementation.
2. Kubernets - Manage a cluster of Linux containers.
3. Mesos - Manage a cluster of resources (not just containers, I'm guessing there are some feature overlap with Kubernets).
4. CoreOS - An OS specialized in running containers.
5. RethinkDB, CouchDB, Cassandra - Distributed databases
6. Ceph, GlusterFS - Distributed file systems
7. RabbitMQ, NSQ, NATS - Distributed queue systems.
8. Manage VM's in a Data center ?? - I don't know any projects in this space.
What's missing is an interface to manage all these together. Maybe this is the direction OpenStack should be heading?
It's as if you were saying that to compete with Amazon.com (retail) you simply have to deploy Magento or Open Cart...
1. Apache CloudStack ( https://cloudstack.apache.org/ )
2. oVirt + Foreman ( http://www.ovirt.org/Home , http://theforeman.org/ )
3. OpenNebula ( http://opennebula.org/ )
4. Roll your own with clusterlabs + PXE + puppet
Many newspapers (at least in UK) did. Guardian is one that stuck around to digital age. It makes them money.
Other publishers who have gone down the "We're a tech company!" route have been forced to give up on that strategy[1]. I wonder whether the Guardian will be able to make that breakthrough, or whether they'll end up migrating to Wordpress or something similar.
1: http://digiday.com/publishers/gawkers-kinja-retreat-shows-fa...
If you want to see a newspaper who are really doing something interesting with technology, check out the Daily Mail, last I heard they were going all-in on Scala. Newspapers that are still in print are fascinating IT places - because come hell or high water, the paper has to come out the next morning. They are the definition of mission critical. Websites are easy in comparison.
Would love to discuss over a pint sometime. Do you attend HNLondon?
The "let Amazon run it" standard response is I think problematic: the cloud market is extremely concentrated. This is not a good thing in term of competition and diversification. We have already seen many times a bad software update shutting down a whole cloud service. Having more concentration is only going to make our infrastructures less resilient.
Resizing, stopping VMs, while admittedly being a rather trivial tasks, point to this usage of OpenStack. When I would mourn the loss of individual VMs on OpenStack (or public clouds for that matter), I would turn gray soon.
Then again, I just tend to say "hosted on our own hardware" or just simply "self-hosted".
Edit: apparently they sourced it from Wikimedia Commons, which has a rather... shoddy replica.