Why we are not leaving the cloud
about.gitlab.com
about.gitlab.com
Does anyone else feel a bit weird seeing off hand comments getting quoted in the explanation for a business decision? I guess we should all get more accustomed to our public input carrying weight in the zeitgeist.
Gitlab's "develop in the open" nature really shows through here. I am not saying that's bad, it's just so different than most startups and established businesses.
On a more general note: It's got to be incredibly hard to do what GitLab does with their extreme transparency. I feel like we have to be careful about reading too deeply into things and nitpicking their culture or process. HN is full of "expert" advice, much of it being terrible. They weighed their options, invited feedback, then made a decision.
I appreciate what GitLab does in being so transparent. None of us are owed explanations or insight into how they operate, yet they go out of their way to provide it. Kudos to syste and his team!
So I disagree with you, because you look at this very black and white while the reality is grey.
I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy.
The lack of automation, especially around validating db backups, failover (not having the failover process scripted and tested is _begging_ to have a nightmare at 2am where you're reading documentation on how to fail over a db), etc.
The simple thing of having your hostname / $PS1 say the machine's purpose could have stopped this. All prod machines have a bright red PS1 and a clear name of <type>-<service>-<prod/dev/etc>-<region>-<dc>.corpnet in my setups.
All of this is reflected in their discussion style, meeting style, etc. Ad-hoc, not very carefully designed, random off-hand comments. Obviously a young team with a lot to learn. Nothing wrong with that, but a lot of customers are relying on their skills! Learn quick!
I do agree with you it reflects poorly on GitLab for only having a primary and replica, with broken backups.
BUT, they are a startup, and they need to be laser focused on growing the company and securing funding for the next quarter so they can keep the lights on.
This kind of pants on fire growth in a startup often (always?) comes at the cost of redundancy and best practices. If you stop to make your platform bulletproof instead of the new features you promised customers/investors, you die.
I'm not saying this is an excuse for them to permanently shrug off making their platform more redundant and engaging in best practices. But, as with many startups, they're focused on delivering features as fast as possible to grow their user base.
I think, and hope, that their recent outage has been the experience they needed to prioritize their shift toward more redundancy and best practices.
Having run and sold a few startups and turnkey operations, let me tell you - if you don't focus on your core product first and foremost and demonstrate the utmost competency in it, you're just pissing money away and are likely to fail.
I don't have direct experience running/selling a startup, but I have worked for several as an employee.
In my experience, the product is important, but not necessarily the deciding factor in whether the company succeeds. Most of the success is down to the management team (e.g. CEO, CTO, CFO) being able to sell the company (and product) to investors during funding rounds.
Doesn't matter how good the product is, if management can't pitch it to get funding you're dead.
Obviously it is important to have a good product, but just look at Uber for example. Losing billions of dollars per year, with no clear path to profitability, and yet they're valued at something like $60B. Great sales/marketing by their management to investors!
Good products literally fund themselves and don't need outside investors. Just like drugs sell themselves, a good product will sell itself without the need for anything like marketing or investors. The money will naturally come.
This to me calls into question the entire idea of crowd-sourcing decisions and plans like this. It was a single person who moved the needle, and now there is retroactive agreement(and quoting) with the people who were against it from the start. If it went the other way there would be plenty of quotes to pull from of people supporting the move to dedicated.
Pretend for a second you plucked someone from a royal family that never prepared their own food, and they asked you how to make food. Are you going to recommend they purchase land, start a farm, grown their own vegetables, and raise their own cattle? Hell no. You tell them to go to the grocery store (cloud) and if they are desperate, they can purchase some pre-made food at restaurants (SaaS providers).
When you don't know anything about a person's exact needs or abilities, making the safest recommendation is easiest, and therefore it proliferates.
I hate the cloud but I know how to do what I want on my own without it. You don't know that about other people.
Rather, the cloud is like leasing the land from an owner. You still have to till, plant, maintain, and harvest, but you have to pay an annual rent to do so. There are a few things that the landowner is responsible to provide, but most of the day-to-day maintenance responsibility is on the lessee. People generally recognize that while renting is sometimes necessary, it's much nicer to be an owner.
We really need to stop this belief that the cloud is a magic thing that can fix your problems. It splits out the responsibility for actual, real-life hardware maintenance to a hosting company, but other than that, the responsibilities, including the responsibility to take backups, set up security, and design a fault-tolerant application, remain in the cloud user's court.
Cloud is just a way to rent a virtual server; you still have to make the server work correctly.
I rent dedicated servers from a large-scale provider. They're really good at hardware and networking, but that's all they do (well, that and some OpenStack S3 equivalent). I have to do my own Icinga monitoring, Ansible automation, etc. For the type of product I run, it's fairly simple. It's maybe not as cheap as owning the hardware, but I think it's a good compromise.
Not to mention that when everyone is using that same tool from Amazon, and at some point it gets big and clunky and its development stalls because people rely on it never changing.. then other options out there eventually leap frog. (I strongly dislike the google-cloud UI, I waste so much time when I have to use it for one of my clients that has stuff there)
DB services is a different class of problem than bare metal v. cloud application servers, and cloud DB services can be used whether you run your application on bare metal or cloud instances (but, for the record, the same type of things are mostly applicable here, in that the user still has to do their own security and configuration).
I still try to research elsewhere too, just to cover my bases, but HN is quite helpful as an archive of knowledge. Especially knowledge about running something in production.
What I mean is, instead of writing about 8TB disks vs 2TB disks, get hard numbers for those and calculate costs for different versions to the costs in the cloud.
Instead of saying that engineers are expensive, look at market data to determine how much you have to pay to hire these people.
Instead of writing about the potential of full cabinets, get quotes from data center providers of what they can provide and how they would deal with this. I'm sure Equinix has solutions for that kind of problem.
And in addition to that, it seems that managed dedicated wasn't considered at all. Why hire someone to swap disks for you? Why not just renting capacity for servers and have the data center provider taking care of hardware issues? They operate at bigger scale and can do this with little overhead costs. You're still much cheaper than the cloud but don't need to drive somewhere to swap a server (you just switch to a spare one until the data center fixed the problem). With Gitlab's scale, it shouldn't be too hard to get a decent bespoke quote from companies like OVH.
Regarding managed dedicated.. I had very bad experience with rackspace in that regard, which is why we moved AWS. At another job we had our own DC suite but had some company taking care of initial cabling as well as remote hands from the datacenter. If I ever do bare metal again, that's why I would do.
-- Robert A. Heinlein
This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this.
Physical environments DO work well, but they do require experience to run them. I have used AWS for as long as its been around, but nothing beats physical environments for "known workloads", as long as you run them EXACTLY like you would run a cloud environment. This is why the hybrid approach is such a great thing. Run what you know well in physical and reap great savings, run what you dont know well cloud ( as well as take advantage of the analytics products), and reap the speed ;)
- This means thinking of physical servers are individual units. - This means redundancy at every level. - This means architecting for failure ( servers and switches do gie ). - This means no shared storage for performance critical parts ( shared storage as below the OS level ). - This means objects stores/sharding/etc as your storage layer. - This means real engineering - And this means exactly the same whether you're physical or not.
I've managed environments of 50 vm's and environments of 50000 physical servers. The methodology is always the same.
and yes, this means you can save some monthly cost, and apply it to staff, that can do a lot more than just maintain this infrastructure.
PS: for those that think that showing up to a datacenter is required, you're doing it wrong. Pretty much any datacenter has hot-hands service, and with the right redundancy, hardware replacement is something you can do at a slower pace.
PPS: im sure someone will nitpick some of my points. The reality is, there is real money savings here. For example, Snapchat spends MORE in cloud infrastructure in 2016/2017 per year, than ALL OF GOOGLE did in 2012... think about that for a second... even netflix runs openconnect to push bits..
Your other points and experience are well taken, but this is definitely too harsh. Think about the nature of GitLab as an open company building dev tools and paying below-market salaries. They have a huge amount of developer mindshare relative to their internal head count. They are not very competitive for hiring people with your particular skillset.
Given these facts, I think an open decision making process serves them better than trying to make decisions behind closed doors. It may come off a little ham-fisted, but in actuality I think it's far less ham-fisted than a huge number (maybe even majority?) of engineering decisions made every day in companies that just don't have the expertise to make the right decision and often don't even realize they are getting burned by what they don't know they don't know.
There is plenty of ops talent willing to work at a high-profile startup. I have experience building and running PB-scale Ceph clusters, private and public clouds, HA infrastructure covering everything from DB failover to BGP, and interviewed at Gitlab when they announced going on-prem. I asked for half the market rate (because I was more interested in solving the scaling problem than money), but did not get it.
The HR person at the first (non-technical!) interview asked a bunch of programming related questions from a script which caught me completely off guard. One question went along the lines of "tell me about a difficult problem that you had a particularily elegant solution for". I think they've learned by now that ops problems usually can't be solved by a simple algorithm.
In retrospect I'm glad I didn't get it, but they seem to be having issues with more than infrastructure.
As a side note, if there are other startups wanting to move to own hardware but struggling to find the right people, I would love to know where to find you. Non-AWS ops jobs are far and few between these days.
It seems more common with more mature "startups" that have done the cost / benefit analysis and realized that the only way to achieve economies of scale for their business is to exit AWS. Dropbox, GitHub, Uber and so forth. Or businesses that simply must operate their own hardware, like CDNs.
There are some smaller places that see value in building a low-cloud service from the beginning. One of those companies is where I work, e-mail in profile.
Not 2012 but in 2013, Google spent 7.3 Billion dollars on data centers [1]. Snap's has agreed to spend a minimum of 400 Million dollars per year on Google cloud according to their S1 [2]. Where did you get your figures?
[1]: http://www.datacenterknowledge.com/archives/2014/02/03/googl...
[2]: https://www.sec.gov/Archives/edgar/data/1564408/000119312517...
As the credits run out the architecture reverts to the natural state - architectures tend to be the derivative of organizational structure, code base, and processes of a company.
Through this lens these decisions start making more sense. It is a reactionary process. Sometimes when you start small you bounce around like that and are lucky enough to grow for years and years until you hit a wall.
Two such companies made the mistake of doing that with OpenStack, which is terrible and should die in a fire.
But one later switched to DC/OS and containers and it has worked really well. They've been migrating apps running on EC2 instances into docker containers than can run on marathon in our local data center and the savings is pretty substantial (even adding in the cost of the teams needed to maintain our own platforms).
Managed solutions are great for startups. There is a lot of value in not having to setup, maintain and manager your own hardware .. but that does reach a limit and companies need to be prepared for that transition and avoid lockins.
Not only it does take a huge amount of time, but it takes so much time that it require multiple dedicated highly-skilled people.
Last but not least, the last article about Gitlab was about a major outage where they had as much as 6 out of 6 backups unusable! The fact that they still exist today is to the sheer luck that there was an unexpected copy made manually before the disaster.
That speaks at length on how that 100 people company is struggling to have the resources and the qualifications it needs to execute (what startup isn't?). Let's not send them on a suicide mission to manage their own hardware on top of everything they already have to achieve. ;)
Dedicated server providers will manage your hardware, that's it.
I know it's more than a handful of deployment recipes, but it's not like they don't manage their cloud instances either...
This article is a false dichotomy with a seemingly rushed decision.
You'll still need people managing servers, but no one has to live close to a datacenter and you can emulate at least some cloud-like features (e.g. spinning up a new server if one server fails and dealing with the failing server later).
The only companies where that doesn't work is when you need physical security for your data (locked cabinets that no one from another company can enter), but that's no concern if you already had your data in the cloud before.
- Up to thousands of Amazon EC2 instances, from multiple AZs, can connect concurrently to a file system.
- Data is stored redundantly across multiple AZs.
- Low, consistent latency.
- Multiple GBs per second.
1. It doesn't scale to the size we need for all repositories, so we still need sharding.
2. It is expensive, $300/TB-month
3. You're still sending all traffic over the network, we prefer the latency of local storage we can achieve by developing Gitaly https://gitlab.com/gitlab-org/gitaly
Bare metal doesn't mean colocation. It also doesn't mean going bare metal only, you could go with a hybrid approach and save the base costs while maintaining the cloud scalability.
Setting up a hybrid cloud like this is not overly complicated and AWS and other providers will offer you a direct connect network connection in many carrier neutral colocation datacenters so you can sit on your cloud vpc with a physical connection in your datacenter of choice.
Get the best of both worlds. Providers like ovh can get very close to the cost of buying your own metal with a tco break-even at over 18 months, not counting network and power equipment costs. Using ec2 or GCE for a large number of persistent vms without a heavy discount seems... Wasteful.
Why not go hybrid? Do your baseline load with bare-metal and your spike load with AWS, GCE or Azure Cloud.
Fundamentally the performance problems they had could be solved by either software or hardware, and I have more faith in the team's ability to learn the distributed computing / caching techniques required than to learn how to run their own metal.
Sure it's some work, the setup is sometimes not ideal, but it's a huge gain imho.
In addition, many people bring the argument that infrastructure is not their core skill. It is true, but do they need to get to the level of public clouds to run their own infrastructure ? I mean, even with sub-optimal choices on hardware but smart choice on how you handle your architecture, you still can have huge savings and not so many risks by doing it in-house.
https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...
https://gitlab.com/gitlab-com/www-gitlab-com/commit/8f7d618d...
Post about "We use X and is great" will usually attract lots of "X is bad, use Y" comments.
I really hope that GitLab took a well informed decision based on arguments and counter-arguments.
All the original internet companies run their own hardware. They rent out excess production capacity to us peons in the form of cloud services.
These companies that all run their own hardware exclusively are telling everyone that it's stupid to run your own hardware... Why are we listening?
Look to the newer tech giants as a prequel of what's to come. Netflix for example, is completely at the mercy of Amazon. They might as well be "Netflix brought to you by Amazon". Their edge is in software alone, nothing that causes a huge barrier to entry like custom hardware. This makes them much easier to dethrone. Hilariously, they rent all their hardware from a direct competitor who has access to all their software secrets. Does anyone else see something wrong with this?
The cloud as a money saving venture is and always has been a damn lie.
It's the same tactic as when automotive companies paid off local governments to destroy Americas public transport many years ago. All the big tech companies have their hands in the cookie jar besides Facebook, who remains mostly silent but runs their own hardware as well.
The major tech companies have every incentive to make you think running your own servers is nigh impossible. Don't drink the fucking koolaid. If the industry continues to consolidate rapidly AmaGooSoft will be the ONLY places you can have web servers in ten years
Really, what software dev hasn't put together a desktop PC and plugged in some Ethernet jacks? That's all you need to Colocate.
Your response echos what cloud providers want everyone to think. Hardware is too hard for us, let's pay someone to do it and make 50% profit margins on us
It's absolutely hard to run your own hardware. There's a reason that companies which run their own hardware have entire teams devoted to it.
You seem to be thinking like an engineer, not a business person. For most businesses, cloud providers offer a substantial cost savings.
The echo chamber needs a devil's advocate and sometimes the resulting discussion gets interesting enough that I question my original beliefs.
In this case I believe owning my hardware is cheaper for me because I already do. I might be a rarity though, or just old fashioned and wrong
There are some scenarios where it makes sense to run bare metal, but they're few and far between. If you've evaluated the (capital, operational, and human) costs of cloud vs. bare metal and bare metal came out on top, that's fine. But don't assume that other mature businesses haven't also done so.
For companies with AWS bills less than $10k/m, it'd be ludicrous to even consider bare metal.
I've been experimenting with "cloud over" where I only use cloud servers for spare capacity. With docker it's getting easier. My baseline CoLo has the same performance as a super high end AWS instance and costs a third as much after my fixed investment.
I still have the cloud if my box goes down or takes a big traffic hit, but I'm not paying for a bunch of cloud servers all the time either.
It's ludicrous that you suggest that anyone who is not willing to dive head-first into 100% cloud everything should be fired. The cloud has a place, but there's no reason that cloud should be the only option, as you appear to suggest.
If you're spending millions on infrastructure, you should absolutely do a rigorous analysis of whether the cloud is better or worse.
I've seen way too many companies with full-time sysadmins maintaining only a few dozen servers. The significant cost savings come from eliminating those positions.
If you have a bunch of idle sysadmins because you thought you'd have to replace a system's drive every week and hired 4 extra hardware-only sysadmins, you can probably fire them without moving to the cloud and save even more money, right?
Significant savings is not something you get from the cloud unless you were massively over-provisioned with your hardware. In nearly every case I've seen, moving an equivalent workload to the cloud has resulted in more costs for the same size workload. But the company still made the decision to do it because it was less stuff to worry about and it made expanding that much quicker.
You pay a premium for the flexibility of the cloud.
Those "old-school sysadmins..." get hired by large cloud companies. Amazon has plenty. The cloud is just somebody else's computers.
Plugging in 150 machines is significantly more challenging. Providing disaster response, backups, stable power, and VLANs to secure services, quickly ratchets up the cost. And nobody wants to pay one engineer per machine to babysit one machine, x 150. That wouldn't even be a good idea, because engineers get bored and start mucking with stuff that needs to be left alone.
Data center administration is a first class professional grouping. Software engineers who don't know that are a real hazard to any business.
Till you're the guy in your 20 people startup who has to do it and crawl around the racks with 100kg of equipment to assemble, cable and power up everything.
When you're a business, you have to pay people to handle hardware no matter what. The question is not whether you need to pay someone to administer the hardware -- the question is who you are going to pay. It can be an employee, contractor, or third-party service company like AWS or remote hands at a colo.
I've seen multiple companies move to cloud merely because they were annoyed at having to go down to the rack to deal with hardware, often making oblique justifications about difficulty when they really meant that they don't like driving down to the colocation center.
Those companies have spent millions more paying AWS than they would've if they just hired a couple of hardware jockeys.
The argument basically becomes a question of whether you know how to hire someone who knows how to deal with hardware. If you don't, it's better to go with Amazon even though they're going to rake you over the coals cost-wise. If you do, then practically speaking, the differences should be small. Your hardware people will automatically fix the hardware issues just like Amazon's hardware people.
The bulk of the work of running servers is at the OS admin level, which is still the customer's responsibility with cloud servers. These arguments about difficulty of administration would work for something like Heroku, but they don't work for Amazon.
I'm in agreement, after the hardware deploy, you're pretty much left with what you would have in a cloud environment: a bunch of VMs (or in this case OS instances) that need care and feeding, and only rarely, much rarely than many would think, do you have to replace a disk, or recable because requirements changed.
Yes that's a loaded statement but so is comparing "the cloud" to "a desktop".
I've had degraded instances screw my customers websites many many times in AWS. On our VMware cluster I never have any such issues because I make sure not to touch anything running production sites
Except that most cloud providers offer replication and load balancing, integrated directly into their other products already. They spent a lot of time making sure that it works so you don't have to.
Yes, shit happens. But shit happens everywhere, and it is far more likely to happen when you run your own bare metal if you don't pump sufficient time and resources into it, than at the company that has hired thousands of engineers and ops people specifically to provide these services.
They have stuff that will allow you to make your stuff redundant if you know how to architect, configure, and deploy it, but you don't get these benefits just by becoming an AWS customer. You have to know how to set up an ELB just the same as you have to know how to set up haproxy on bare metal.
Is it easier to configure an ELB than a haproxy instance? Probably, but it's not that much harder, and configuration is mostly a one-time cost. Pay someone on staff to spend a couple of hours testing and configuring haproxy or pay 3x more than you should every month to rent VMs on Amazon? Which one is cheaper over "the long term"?
All you're doing is renting some space in a server. The configuration is still mostly up to the user. Hardware requires maintenance, yes, but people are really hamming up both the frequency and difficulty of hardware maintenance.
Have you ever used EC2 with traditional apps hoisted onto the cloud? Just because you can click a button for ELB doesn't mean anything when the system wasn't horizontally scalable to begin with. EC2 is just VMs, it doesn't provide any high availability at the VM level.
With local solutions (e.g. vmware/kvm/openstack), you can live migrate these annoying (but critical) pets around as you phase out hardware with very limited traffic interruption. With EC2, the thing eventually just shits the bed and you get a notice saying the instance will be retired even though it's already in a worthless state.
Glad to hear the vmware cluster is working out for you. Anecdotal but I've seen people having issues without touching vmware too. One personal experience doesn't diminish the cloud. Maybe different business needs? One sysadmin maintaining customer websites vs multi-person team managing a web app?
I'm not against running your own servers. Bare metal is fun, interesting, and possible. I'm just against blanket statements that "it's bare metal or nothing!".
The question is, does it make sense for businesses with a handful of servers running their office environment, to a few thousand servers running production for a moderately popular web service, to pay consultants to drive out and tend the servers in their office closets/colo racks, rather than piggy-back on someone else's (mostly) well-oiled machine with economies of scale.
If you can build an in-house equivalent of the AWS team, do it. If you're going to pay someone to do it for you, AWS might be a better deal than your neighborhood business IT firm.
Using Netflix as an example again, if they're pitted against a company that runs their own hardware but is otherwise equal, they will lose. A portion of their profit is siphoned off as Amazon's profit. What's already happened is Netflix is directly funding development of Amazon Prime Video.
Meanwhile, the most successful hardware vendors are all farming production out to third party manufacturers...
The only clear differentiator then is actually owning a datacenter. And the Total Cost of Ownership of that isn't easy. For someone who doesn't want to be in the datacenter-ownership business, opting in to cloud would make sense. Concerns for privacy and government intervention are a valid concern though.
Correct.
Traditional product companies very rarely own all their own factories.
Do you honestly think Netflix management is incompetent? They've run the numbers on this countless times. They have way more insight into their spend and operations than a random anonymous internet commentator.
so, you were saying?
Or do you think facebook,google,aol,tripadvisor and pretty much anyone beside HotNewStartup are incompetent?
> Or do you think facebook,google,aol,tripadvisor and pretty much anyone beside HotNewStartup are incompetent?
Far from it, and that's precisely my point. At Facebook scale, it certainly makes sense to run your own data centers. At HotNewStartup, it obviously doesn't. Competence is finding the solution which works best for your scale and organization, not following some generic rule.
The video is the most important part of their product, so keeping it at your ISP or a local exchange not only lowers latency, but most likely also gives better throughput since it only has to travel through your ISP's network or local exchange links without having to go through the rest of the internet backbone.
Latency ( 30ms vs 300ms) doesn't matter much in video playback, only throughput. Once you hit play, it isn't bi-directional.
But you're just nitpicking. This IS their core business, and they realized its "too core" to give to someone else.
So yes, you do "outgrow the cloud", and well, part of "outgrowing" can be cost structure. Gitlab offers a free service, their main competitor, github, runs their own datacenter. Why do you think that is?
Put another way, if there was a cloud DC in each major city and Netflix hadn't already started using their own edge nodes, then there's nothing stopping them from spinning them up in each cloud DC as opposed to their own hardware. There's nothing inherently special about their setup except location.
Nothing about this situation says that Netflix "realized its "too core" to give to someone else." I don't quite understand how you're making that leap. Instead, they had a very specific requirement (that most companies don't have, mind you) that current cloud hosts can't provide. Nothing else.
Besides, here is a list of companies using AWS:
https://aws.amazon.com/solutions/case-studies/
You think Google and Microsoft is incompetent when they offer cloud services to their customers? There are a lot of companies that do know want to own any datacenter related infrastructure because it is irrelevant to their business and there is no on-board expertise. While there are companies that can afford to hire staff for running large DC operations. I thought this is pretty obvious to everybody.
Most companies at even 1k servers will get enormous cost savings from O&O.
A former employer, then in the 600-ish box range, costed out moving to ec2. Our ops actually brought up our internal hadoop cluster on ec2 as well as ran parallel workloads for a week. Amazon at the time offered price deals to attempt to get them as a marquee customer, as well as a bunch of engineering time. Even so, ec2 cost 3-4x as much for 1/3 the throughput. ec2's prices have since fallen, but so has the price of bare hardware.
People also overestimate the cost of hands-on; this company contracted for 4 hours / week with a local person. If you get serious about puppet/chef you can successfully run lots of servers in a remote datacenter.
Or a traditional business renting a storefront? Something that happens all the time. Even huge companies like Apple use OEMs. Samsung manufactured Apple chips for years despite putting up a direct competitor to apple in the Galaxy S.
Systems like this allow people who do not have capital to scale up. Just like developers build malls and lease space to retailers or OEMs build factories and lease them to companies to make equipment.
As soon as you can confidently predict sufficient demand, the economically rational decision is to hire a good ops team and run real hardware. But to do so, you need to either have money or get money - and the costs of cloud service are zooming up, potentially leaving you without the margin to invest.
The question is what that level of sufficient demand is.
What do you mean by this, because it sounds like a straight up lie?
Amazon and Google are in an ongoing price war and cut costs regularly.
Obviously if you're growing your costs will grow, but that's basically a tautology. Bare metal costs also rise as you grow.
Except you're gonna need between 10 and 100 people to recreate the service in-house, not just one.
Trying to re-do something from scratch in-house is always more expensive than just buying the product.
while true; do
curl https://rails.app/do/the/thing
sleep 3600
done
I don't mean to say that shell scripts always can replace more complex systems, but sometimes they're enough.
There is no economy of scale with the cloud. As you scale up your service, Amazon/Google happily scale your bill right up with it. When you roll your own datacenter, things get much cheaper per unit (bandwidth, storage, etc) as you get larger.
I feel this is one of those "easier said than done" statements...
An organization of even medium size? Not a chance.
Sure, paying extra for a dynamically scalable service is inefficient if your resource needs are flat 24/7/365, but is that realistically the case?
(OTOH, one must not forget that there is a cost to engineer systems to take advantage of dynamic scaling to minimize costs while meeting requirements.)
Yes, the vast majority of apps are not 12 factor architectures designed for scaling so nodes can't be brought online and killed at the flip of a switch.
This also explains why older tech companies disproportionately use their own hardware, even those that aren't all that big. They all made that decision at a time when it made sense and the switching cost is sufficiently high to prevent them from turning back now.
I doubt there are many rational cases now for newer companies to use anything but the cloud. Even at the scale of Snapchat (>3B$ infrastructure spend over next 5 years) and Netflix, it appears the public cloud is still the rational choice.
So while it may be worthwhile if you're huge enough to get a big discount, that doesn't mean it applies to small or medium size companies
If it requires no or a smaller ops team then it seems like it might be.
Could be solved by more staffing, I suppose...
If you needed a team to manage it before, you still need Ops with "the cloud"
Most tactics in business are money-saving tactics, and cloud is clearly one: dynamic scaling means only spending now for what you need now rather than spending now based on anticipated future need, and someone-else's-server means economies of scale in ops.
> The cloud as a money saving venture is and always has been a damn lie.
I personally have conducted migration savings assessments for companies going from data centers (owned/lease hardware) to cloud services. I can tell you with 100% certainty that the savings are there and have seen financial proof of the savings.
It's either that, or I'm a liar.
/s
My attempted joke wouldn't make sense outside of this context
On paper, the cost differences between renting/colocating traditional dedicated servers and using cloud service with similar performance/capacity is huge, so it's pretty unusual to actually save money by migrating to cloud. I'd love to hear where the savings actually come from.
At a small scale, you're paying the cloud less than 2 or 3 competent admins with datacenter and networking experience cost for salary. In that situation, you're saving money, because you're saving the hardware ops team. You're paying more dollars per iop, core and GB of ram, but the alternate method of obtaining these resources has more surrounding costs than "ze cloud".
On the other hand, once you're shoveling several hundred kilo-dollars per month to a cloud provider, it makes sense to throw a million or 10 at dell and hire those operators, because it will save money within a year or two. This only makes sense if you need the resources, but if you need those resources, it helps being more efficient.
But if the problem is in the software stack, one would still need competent sysadmins/system engineers with skills to diagnose and resolve the issue. When (just a random example) oom-killer wreaks havoc and free(1) insists there's more than half of physical memory still available, it doesn't matter whenever one's in the cloud or not.
I believe, skilled system engineers are still a requirement for any large project, be it in cloud or not.
And that's why cloud will win every single time, forever, on all metrics.
Because the cloud requires less engineers to achieve the same work, as it takes care of the low level hardware work.
That used to work well before the "cloud" era, and still does.
If you use some VM-based cloud infrastructure you will still need a bunch of good system/linux admins to make your application work on the linux in the cloud. Without these, you're toast.
Bare metal requires you to have both good system and linux admins, and on top of that, good hardware admins, networks and datacenter hands. These are different skill sets.
So again: If you do the bare metal thing right at the scale it pays for itself, you'll probably end up paying both the skillset for the cloud deployment, and add the hardware skillset on top.
I believe, usually, "bare metal" hosting means you order the hardware, get it installed, but networking/cooling/power supply/etc are done by the datacenter people, not your own staff (your own staff may be not even permitted to enter the server room). No less-common specialized hardware to deal with, either.
System engineers must known OS internals well. If they do, it's unlikely they can debug, say, a kernel memory leak (which can be a thing in a VM), but not a lockup in a network card driver's interrupt handler (which is close to impossible in a cloud VM, but I saw this on a bare metal). Maybe I'm wrong, but that would be, like, too specialized and just weird sort of specialization. Am I wrong?
To go to Amazon EC2 let's see... 64 GB of RAM for $0.862 should do the job there, but just for that I'm at $620/mo. Then let's add bandwidth, 10TB out at $0.01/GB = $100/mo. Now for disk I have to use EBS which runs (let's say st1 will do the job) 0.045/GB/mo*3TB = $135/mo. And that's before any IO.
So to move from bare metal to amazon I go from $1.5k upfront + $90/mo to a total of $855/mo on Amazon.
I can buy a new server every few months for the same cost of hosting that at Amazon - unless I'm severely missing something. Maybe it's worth if it you're a very small installation or one with very unpredictable traffic? But I just don't see it.
Also, most businesses don't have demand that steady. Nor is it advisable in a cloud architecture to rely on a single large server.
Most of it does need to remain online at all times including several large databases. Just running my logstash server alone eats 8 GB of RAM and 200+ GB of disk.
In bandwidth and storage alone I'm past the cost of the server - so even if I could split this up and keep it mostly offline it still wouldn't be close to worth it unless I'm missing something more significant?
> You're not including the time costs of operating that server.
Do you mean swapping the drives about once every 3 years with free DC remote hands? Operational costs there are almost nothing - maybe 10 hours a year of my time. Surely much less than it would take to get everything running on a cloud architecture. And that cloud architecture would still require some level of maintenance and monitoring I'm sure.
Nope, you pay more for a single large machine than multiple small ones.
If your service is truly so stable that you only need to spend 10 hours a year maintaining it (and demand isn't growing), then maybe the cloud isn't right for you. But that's not true for most companies.
As for personnel, if I had to, I'd hire mostly devops people or pay freelancers for jobs as needed until hitting a limit where you have to dedicate a large portion of someone's time to it and then repeat the math - I fully suspect that this would mean sticking with bare metal. I suspect a 2 or 3 person group dedicating their time to managing only hardware and basic infrastructure for something like Kubernetes could probably handle hundreds of servers without issue. Even if that averages out to an additional cost of $1000 per server per year it'd only be approximately a $80 per server month, nothing like the increase of going to EC2. And renting entire racks gets cheaper than the individual colocation I'm paying for at the moment. It'd surely be interesting
I have no idea where people came from when they moved to AWS/etc but I always have (and still do) picked up the phone and got someone on the line who usually fixes whatever the issue is with me on the phone.
As far as demand spikes - even smaller hosting companies will have hardware they can spin up pretty quick.
1. Reserved instances. If you've got a running business and are expecting to be around for 1-3 years, you get 40-60% discount on that 4xlarge.
2. Do you want to move the service exactly as it is? Maybe you don't need the large ebs? Maybe you can rewrite the storage to S3 instead which is much cheaper? Do you have heavy, sporadic tasks that you can move out of your main system and into lambda+queue and use a smaller instance?
3. How much do you spend on people to monitor the hardware, source and replace the disks, do firmware updates, etc. ? How much on external services to monitor your box which could be replaced with integrated AWS solutions at free tier?
Basically what I'm trying to say is: if you just lift your current system and move it to AWS, you're ignoring lots of opportunities. You need to consider much more than a 1:1 hardware requirements migration.
It's really amazing that, when faced with clear evidence of the expense involved in moving to AWS, you suggest that he prepay for a year of service up-front and redesign his application just so Amazon's bill doesn't look so egregious anymore.
My experience aligns with his. We moved from racks with 20-something boxes to EC2 with 100+ instances. Our monthly bill was 80% of the cost of the hardware in our data center.
How did we solve this problem? Why, move to Docker and Kubernetes of course! Over a year of manpower has been devoted to that task. What kind of savage would ever return to bare metal in this enlightened age of expending millions of manhours redoing stuff that was already working perfectly well?
If you want to autoscale, autoscale on the cloud and keep your primary nodes on bare metal. There's no need to start forking over millions of extra dollars to cloud providers to host all of your infrastructure.
Cloud has some unique benefits, but it should be used for those unique benefits only. There's no reason everything has to be moved there.
I'm not advocating to rewrite just to give Amazon money - you're creating straw man arguments here.
I'm just saying that if we're comparing different ways of hosting, we shouldn't pretend they work the same and have the same tradeoffs. Maybe one server approach is optimal. Maybe the cost would be smaller if the service would be split into various components. That's all part of a proper comparison. Including the cost of getting from one arch to the other.
Specific example:
> How badly will moving to cloud storage via S3 affect his performance v. having the files on a local disk?
Depends what they do with the data and how the users interact with it. Maybe there's no data that could be migrated (all records need to be available in memory for the web app), maybe it would be slower but have a trade-off of using smaller instance, maybe it would improve the performance considerably because file-like blobs are not stored in the database anymore and users can get them quicker via local cloudfront caches. There's no generic answer!
As for the outage... S3 was down for a few hours. And for many people it happened when they were sleeping. If your single server goes down in a data centre - what's your cost and time of recovery? S3 going down once a year still gives many companies better availability than they could ever achieve on their own.
If the need to reassess the architecture doesn't convince you this way, think about migrating from AWS to own hardware. If you were using S3, sqs, lambda and other services, are you going to plan for standing up highly available replacements of them on separate physical hosts? (Omg, we need so much hardware!) Or will you consider if it can be all replaced with just redis and cron if you have relatively little data?
You could make the same argument going the other way though - why should GitLab spend money recruiting/hiring people, leasing space, setting up monitoring, etc, if their solution today works? Why should anyone re-architect their system to give $COMPANY money? When your bare metal's RAID controller craps out and you have to order another, will his customers be understanding of his decision to move to bare metal?
It doesn't makes sense to factor in fixed costs of such a migration.
Well, that reason would be, at a minimum, a halving of their hosting expenses.
It also doesn't necessarily take much refactoring to move either way. Even if you're heavily dependent on cloud storage, etc., you can access that from an external application server.
The person I replied to was suggesting that the parent refactor in order to make AWS costs less egregious without articulating any particular reason that the original commenter should do so.
> When your bare metal's RAID controller craps out and you have to order another, will his customers be understanding of his decision to move to bare metal?
His customers won't need to know, assuming he has a standby that can take over. Even if he doesn't, he can rush down and install one and move the disks over. With an AWS outage, you can't do anything but say "I hope Amazon fixes it soon". The bare metal equivalent is a power or connectivity loss at the DC, which is much rarer than AWS outages.
So, pay for the privilege of being vendor-locked to Amazon. What a wonderful idea.
(There are multiple services providing S3 interfaces BTW)
So ~$900/month for bandwidth alone.
The disk and CPU costs for me are worth it, much better to pay $600/month and let that be someone else's headache. But the bandwidth makes this totally a nonstarter.
Out of curiosity, what are you moving?
I'm also not sure how these smaller datacenters somehow charge so much less for bandwidth.
https://media.netflix.com/en/company-blog/how-netflix-works-...
What are some examples of companies that started in the cloud then switched to their own metal? Surely there are some blog posts about costs before & after?
https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...
"Netflix operates its own content delivery network (CDN) called Open Connect. Netflix manages Open Connect from Amazon, but the storage boxes holding videos that stream to your house or mobile device are all located in data centers within Internet service providers' networks or at Internet exchange points, facilities where major network operators exchange traffic. Netflix distributes traffic directly to Comcast, Verizon, AT&T, and other big network operators at these exchange points."
Everything else - such as the Netflix UI, your account management, billing, etc - is on Amazon. But Netflix has an enormous amount of hardware distributed around the world. If you wanted to compete with Netflix, you would need to do a lot more than uses Amazon's cloud services.
Yes you'd need multi-million dollar contracts with all of the major studios. Netflix isn't as much a technical problem as it is a business problem.
HBO has a far worse network, UI, and technology in general but its still competitive simply because it has Game of Thrones and other shows.
After pioneering streaming and realizing that the studios were asking themselves why they don't just load up their own computers with movies and get paid to stream them out, Netflix was forced to start its own content division to keep subscribers onboard and prevent the studios from poaching all of their customers by failing to renew contracts for content.
We have a similar problem in online services. It's impossible to compete with Facebook because the law allows them to hold data hostage.
We badly need copyright reform. The current incarnation is impractical in the digital age.
Copyright should be adjusted so that people can make money off of their creations without allowing the culture to be held hostage to corporations. If the copyright term was shortened to, say, 10 years, a Netflix-with-only-content-from-10-years-ago would be totally viable without having to ask for anyone's permission.
Copyright is a government-granted monopoly. It's gotten totally out of control and severely restricts competition in this type of space.
Right now, competition is hamstrung because people are prevented from entering the market with anything that the consumer would find desirable, barring permission from the rightsholders.
Competition is not real competition if the same data cannot be accessed. For example, Chrome and Firefox compete because they both access the same internet, and whichever browser provides the better experience for that internet will win out. That kind of thing is not possible on walled gardens like Facebook because the law allows them to obliterate anyone who would create an alternate "Facebook browser", and it's not possible with things whose primary offerings fall under copyright protection, like Netflix.
We need to open these things up so that people can compete and provide the consumer with the best experience without needing anyone else's permission. It's true that most people will want access to new releases, but if copyright expired after 10 years, you'd at least have a secondary service that people would actually want to use, which would compete with Netflix et al for viewership on a lot of the content.
The movie industry has Long spent its years fighting piracy instead of providing a solution to a problem. Now there are solutions to the problem and the industry is refusing to adapt and continues to hold the content for themselves instead of licensing it to the likes of Netflix.
We know the studios have the right to license content because that's exactly what they do when they sell licenses over seas to broadcasters in other countries to air content.
This isn't about copyright. It's about the studios lack of ability to license content.
This is why in 2017 piracy is still a major issue.
This isn't even getting into exclusivity licensing that the studios seem to be real keen on doing with Netflix, Amazon, Crackle, VUDU and a jillion other streaming services. You know those shows that are branded "Netflix Original" on Netflix? Those aren't literally made by Netflix, they've just paid for some/all of the production and licensed them exclusively for some period of time. They're still produced by all the same studios that have been producing shows and movies for years.
Actually, they are a media company. Their "edge" is in content creation and licencing deals. They might have been first-to-market for mass-scale streaming, but streaming video technology is a commodity, and they know that they will live or die by the desirability of their media catalog.
> Netflix for example, is completely at the mercy of Amazon.
Amazon and Netflix were symbiotic for years, as Netflix was a headline corporate customer, and I'm sure they received preferential pricing. And given the quality of Netflix's engineering team, I would be very surprised if they don't have a plan to migrate by the expiration of their contract if necessary.
That's only partly true. But if you considering a lot of other "media companies" haven't been able to compete with Netflix's ease of use, then you have to ask the question how is Netflix different from any other media company. And the answer is because they are also a software company, where their successes is primarily attributed to making DRM video in browser easy and seamless for average folk.
The original success of Netflix depended on being the first company with a fixed-price-no-ads streaming business model in addition to really good user experience across a wide variety of platforms. I remember being astonished at watching a Netflix movie on my iPhone (when I was still an iPhone user) while riding a train in 2011'ish. It was miles ahead of any other competitor in providing solid, usable clients on multiple platforms combined with reliable streaming in most cases and networks. This was as much a triumph of technology and product design as it was of the content available. If it was only about the content, it would be Hulu (at least in US) instead of Netflix who would be in pole position today.
Actually, the original success involved putting DVDs in mailboxes.
> I remember being astonished at watching a Netflix movie on my iPhone (when I was still an iPhone user) while riding a train in 2011'ish.
Right, but you're going to be equally astonished in 2017 if you don't get the same level of service from Prime Video, Hulu, HBO GO, YouTube Red, et. al. And in 2019 a YC company will offer a one-click media publishing company that will allow any random joe on the planet to publish a DRM-protected movie available through a world-wide CDN with a pay-per-view payment system.
In 2011 tech was a major differentiator, today not so much. Netflix is moving with the times, that's why they are offering original content instead of being content to provide a good platform to stream other people's IP.
> This answer is in the same category as "Actually, Google is an ads company". That is to say, it makes a plausible case but ultimately wrong.
https://qz.com/607378/were-live-charting-googles-first-alpha... (tl;dr: the overwhelming majority of google's revenue comes from ads)
> Prime Video, Hulu, HBO GO, YouTube Red,
So Netflix has a six year lead and attendant network effects in it's favor. If you don't believe me, I created a facebook clone in PHP last weekend which works just as well (or better) than the original Facebook. Do you want to join it?
I completely agree with you. I personally have a Netflix account, and do not subscribe to other services.
My point is that right now, today, this very instant in time, Netflix is not primarily differentiated on technology, but on having content that their subscribers want; that online video distribution is now a commodity.
This is where we disagree. I don't even know what content Amazon or HBO or Hulu has at this point. Netflix has already made me a lifelong customer with their product and user-experience in the past 10 years or so that it'll take an earth-shaking difference in content for me to switch (or add) a second monthly bill for content. Netflix is part of social fabric (at least in the US). All of this was won by relentless focus on an excellent product and user experience... and perhaps only a middling content catalog.
One of the big points of the cloud is that is gives one the ability to do things that one would trouble doing without the cloud. A small business with a single part-time developer can deploy an app to Heroku, and not need an ops person. That sure beats the heck out of the small business not being able to deploy software at all. As a former small biz owner, I did not want to have to manage a bare metal server in a colocation rack with a not-cheap data pipe. I just wanted a CRUD app with a minimal feature set.
When Heroku went down, I sat around wishing it would come back up, and eventually it did. That was way better than having to service a physical box when it failed.
Heroku cost me a few hundred a month, and freed me up to earn much more than that. In my eyes, it paid for itself many times over.
Well. AWS went down (part of it), but Netflix stayed alive.
They employ their ops team and their ops team has equity stake in the company's success as well as their career riding on the line.
If a chunk of Amazon goes down and takes out your website, all you can do is wait and hope Amazon is prioritizing the fix.
>When Heroku went down, I sat around wishing it would come back up, and eventually it did.
This method of recovery just doesn't cut it in businesses with lots of transactions 24/7. Just sitting around can be flushing thousands per minute down the drain. In those deployments, you need to build across multiple cloud providers; and once you've reached that abstraction, you might as well have your own hardware as well and just burst to providers.
A company like Netflix can get by because they have their own hardware for streaming so an outage only affects people browsing for a movie at the time. And on top of that, their only interactive transaction is signing up for the service, which is extremely infrequent compared to the rest of the traffic. So essentially it's safe for Netflix to use AWS because they don't need high reliability/control of their own destiny.
>A small business with a single part-time developer can deploy an app to Heroku, and not need an ops person.
This use case I understand. However, anything at the scale that is frequently discussed in these contexts (gitlab, etc), they need ops people.
The cloud offers convenience that all the `original internet companies` wish they had when they were starting out. There may be value in switching when you hit a certain scale, companies like Netflix will have already done cost-benefit analysis to that degree. But it's a massive undertaking that introduces substantial risk for long periods of time. Many companies are just not in a position to do that.
Their original proposal on bare metal was for 64 servers. They need an ops team regardless of whether they control the hardware or not.
When you are in business, saving money is not always the most important thing. In reality, you usually spend 95% of your revenue. The question is not "How much money can we save", but "What investment will bring the greatest return".
It's a bit like buying the building that you are working in rather than renting it. Or hiring full time cleaning staff rather than hiring a firm to take care of it. The second choice is more expensive, but requires less capital/time up front. If you can use that money and time to better effect then it is a no brainer. As companies get bigger, they have less opportunities to make large returns on their money/time, so it often makes sense to start focusing on those smaller returns.
As an aside, this is also why you shouldn't go to your CTO and say, "I can save $X by refactoring/rewriting/testing our code". The CTO can usually always find things that appear to be better ROI opportunities and so you get into a situation where refactoring/rewriting/testing becomes forbidden. Rather you should always attempt to use appropriate techniques to maximise throughput in development and to keep your stake holders happy.
Exactly and most of the companies are not original internet companies, and they use the cloud.
Instead this is about gitlab's definition or rather level of transparency and something they seem to be proud of and are even being appreciated very often here at HN. I feel that gitlab have taken the transparency thing to an extreme, to the extent that they even named the engineer that caused a recent downtime. I don't think it matters that the engineer didnt object to it. Then there are posts about why they are choosing some UI framework and why they are moving away from cloud and now this. It's fine to be transparent but in a professional world it's also important to be private about certain things. Trying to please people by being transparent for the sake of it will take a toll on the team eventually. It's much more effective to just get things done instead of constantly being transparent to the external crowd, about every single detail