We run on bare metal and to move to GCP would triple our hosting cost, not including bandwidth (which is free* for us).
Do people get discounts for doing PR? I’m really curious.
- you don't have to support your own low level services like Pub/Sub, Cloud Storage, Spanner, etc.
- you don't have to worry as much about things like HIPAA compliance
- developers don't have to wait for you to allocate more hardware, greatly slowing down the software development process
- it's much easier to test your applications, since you can spin up a full copy using rented infrastructure, and then throw it all away once your tests are complete
- many other benefits that I'm forgetting
And of course if you buy enough from them, you do get additional discounts.
For many (most? I don't have data) companies, the actual cost of the infrastructure and bandwidth itself is tiny compared to all the other "hidden costs".
- Bandwidth is VERY expensive in AWS compared to what you can buy in Datacenters. A 1 GBPS connection is so "cheap" you can get it and a 42U rack at HE.net for $400/mo ( http://he.net/colocation.html ). That same 1GBPS is roughly 300TB/month. Assuming it's all outbound - that's roughly $21k in AWS.
So bandwidth is a small cost IF you're not moving large amounts of data out of AWS. If you stream media - it can be a HUGE portion of your bill.
- HIPAA Compliance - That and other compliance aspects assumes that you need those compliance frameworks. In enterprise world that might be the case (SOC2 and ISO27k1 are very common requirements for Enterprise SaaS clients). However in the B2C? Users don't care.
- Waiting for hardware allocation. Once you get to a certain size you have hardware spare, typically. Developers can use that for their tests. You can also have your datacenter/operations group work with your developers during project planning so the code and HW can be ready at the same time.
That being said, I don't disagree there are advantages to the cloud. I'm AWS certified, I drink the coolaid. But saying that there are "so many other benefits" is ignoring the fact that there are "so many benefits" to on prem too. There are pro's and cons to both sides.
Requisitions for non-cloud server resources can takes hours, days, and in some places weeks; even when there is availability depending on the org. At best it takes close to an hour compared to a few seconds in the cloud via a web dashboard.
On prem delivery/provisioning is not slower when executed properly.
Furthermore, If you have spiking workloads or hyper growth and can't do proper capacity planning - that's exactly the original AWS value prop, run a hybrid private/cloud environment and extend to the cloud when needed.
That being said, not every organization is as dysfunctional.
I think the best approach would be a hybrid one, where developers could spin up instances in the "cloud", and ops would migrate them into the main infrastructure when it made sense to do so.
This works great for gaming companies who can see 10x loads during the first few weeks/months of a new video game release.
I don't understand this argument. Startups usually do not have money to have physical servers sitting around just waiting to be utilized, not to mention the IT resources to stand up those servers just in case someone might need them later. You're saying that everyone has a ton of servers sitting in the rack just waiting to have containers deployed to them? Who?
From k8s webpage (https://kubernetes.io/): "Automatically places containers based on their resource requirements and other constraints, while not sacrificing availability. Mix critical and best-effort workloads in order to drive up utilization and save even more resources."
Furthermore, running k8s to pack containers into AWS instances that are are themselves actually full blown VMs running on a hypervisor is of course one way to do computing. It is perhaps worth considering that there are others, more direct ways that can be achieved more economically.
I work for a Fortune 500 where we have to wait weeks for hardware. This is not unusual at non-tech companies.
Here's a full list of our certifications. We're rapidly adding more. Feel free to email me at bookman@google.com if you have any questions: https://cloud.google.com/security/compliance
The Ops people there were pretty proud of their infrastructure, but they took it to a pretty extreme level. I think they were racking their own stuff and even running their own routers etc.
We actually haven’t had a hardware failure (apart from HDD that have all been replaced with SSD) in something like 7 years.
We don’t even pay big bucks for Dell, HP etc. Just cheap SuperMicro boxes.
There is nothing "extreme" about any of that.
If you own the hardware or use debt it appears on side of the balance sheet where as if you just pay a provider then it is a cost of business like any other software license so it makes the company look finally better even if the overall actual cash cost is 10-20% more.
Also Etsy has had all Senior Management replaced so new technology leader could just be leading the charge as Etsy was a bit of an anomaly running their own infrastructure because that isn't really a competitive advantage for them from the business customer side.
I'm oversimplifying - there are a bunch of other considerations, like guarantees about future prices and capacity availability and whether having more flexibility makes it worthwhile for Etsy to pay more, but fundamentally Etsy looked at their current costs and asked Google to make an offer. (I would assume they actually had all three major players bid, and Google won.)
Also to your point, the announcement specifically called out Google's AI and big data capabilities. Those are workloads that can involve a lot of bursting, so the cloud might be especially attractive. Imagine trying to implement BigQuery in bare metal - you can do it, but you have to have a lot of extra servers lying around just in case someone tries to run an analytics query. The same goes even more strongly for something like a model training workload. For bursty enough workloads, running in the cloud can be cheaper than bare metal even given the cloud provider's profit margin.
Depending on how much model training Etsy wants to do, something like TPUs might have also given Google a fundamental edge in costs over bare metal or other providers. I don't know enough to know exactly what factors went into Etsy's decision, but I think these are reasonable ideas.
They might not bid in the strictest sense of the word, but when the vendors are mostly 3 players (Google, Amazon, and Microsoft) you just ask each one for a quote. I don't see how functionally that is different from bidding.
Btw, I know there are other hosting providers out there (Rackspace comes to mind).
I can't think of any one of them that would be able to deal with Netflix scale without shitting the bed, missing features, or an almost unacceptable lead time as they build out infrastructure.
Generally you need to have a certain amount of usage before companies will engage these discussions.
Still a lot of issues on GCP for low latency DB deployments though (no permanent NVME, higher latency on the network backed block storage, etc).
Presumably you start looking at Spanner but that’s it’s own world of hurt (higher latency than MySQL/Postgres, a reeaaallyy basic query planner, the list goes on). You have to change the way you think about data with it and pretty much re-architect your data access.
Agreed re machine learning etc.
If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud.
If you're a non-technical company with a small IT department that wants to have an online presence and/or build internal online tools without having to build out DC's everywhere, you would go with cloud.
I could come up with dozens and dozens more. Cloud isn't optimal for everyone, but it is VERY helpful for certain organizations.
I agree, but I don't think that's the market either.
I think the market is in massive big-corps, moving their IT infrastructure to the cloud.
Like regular, non-tech companies.
Small startup bills might be hundreds, thousands, possibly 100's of K but probably not millions. Big corp contracts may be millions - and stable.
(and before you trot out Netflix, the vast majority of their bits are served out of OpenConnect appliances colocated at eyeball ISPs and peering points)
The rationalization was that in the long run it was cheaper to maintain the status quo, and not have to worry about networking, firewall, and replication issues by hand. I'm pretty sure MS also gave them a large discount at that scale.
My hunch is you'll see a lot of move back on-prem during the next downturn when hardware providers need to move a lot of hardware under duress while businesses look at cutting their opex (you have to pay for cloud compute/storage forever, whereas you can run on paid gear for the cost of power, network, cooling, space, and the tech people you need anyway).
To quote Warren Buffet, “You only find out who is swimming naked when the tide goes out."
Don't assume this.
I worked at a 'big tech company' that made a very popular device.
Our IT was a disaster.
We made one of the world's most popular gadgety things - we couldn't make our email work.
Doing things the right way is extremely expensive sometimes, you have to pay zillions for consultants and they can screw it up.
If 'IT' is not a massive cost - often - the consideration is moot - move it to the cloud where things will work, and there is low overhead.
I'll bet $100 that Amazon is earning those margins easily.
Another example: why do companies use Windows instead of Linux which is free? Well, because Linux in a regular corp environment would be a huge nightmare. $50/employee to get something that works? That's dog-food heap. It's hardly a consideration, easy, just buy it. Maybe MS margins are huge. Who cares, because the value they created is monster.
Sometimes 'point and click' IT is worth a lot given all the risk factors etc. etc..
If you think about it- why on earth did companies start having massive servers in the first place?
That was an anomoly.
But that's exactly what Etsy isn't. They've been around for years. It's a stable, mature business (at least by the standards of the sort of business they're in). What possible advantage could they gain from switching to this cloud?
Oh, you meant advantages for the company? Never mind.
Let's say AWS has 50 services today. Etsy won't have to duplicate any of those by building & maintaining their own versions. It's an extraordinarily time consuming and expensive process to do so. The capabilities AWS can enable will get greater, while the expense to replicate on your own what AWS can do, will also climb. The extrapolated result, is that AWS & the others will perpetually take market share for the next 10+ years from do it yourself approaches. More for the software duplication expense (time & money), than the hardware/bandwidth cost difference.
Why on earth would a startup waste money like that? Just rent a few servers, most places will have it up in a few minutes, guaranteed, any large provider have automated that long ago. Much, much cheaper than the cloud. It seems, even in this thread, that there are two choices: cloud or colo. Hell no to either. Rent dedicated -- yes.
The only reason one would go with the cloud is if you plan to use their services ie not EC2 but the many others. However, those can have a runaway cost too and it's not clear whether a new startup should take on handcuffs like that -- once you decided you are depending on SQS etc it'll be hard to move to elsewhere.
Because the cost of hosting a typical startup, even on GCP, is trivial compared to a single engineer's salary.
We actually make an effort to cut our hosting costs while expanding capacity pretty much every year now because Intel keeps making better chips.
* Well designed security / encryption may cost a lot.
* Compliance may be a pain.
* multi-DC and HA
* Surge resource usage over holidays.
* "Sleeping" resources at night or wasted.
I've been in public cloud pitches that sold their security measures as a golden standard that made my managers believe there's no need for a security team anymore. No need for protocols or proper architecture as x cloud has enough flashy certifications.
Yes, it's still mega expensive. But allows us to scale when we need it and not have to worry about the hardware, cost, or management of it.
There's an entire other side of building a web server that devs, thankfully, never have to see. OpenCompute [1] aims to standardize what that process looks like. To think that you can run bare metal, and do better than industry standards is naive IMO.
In the end, if you're worrying more about anything than business logic, you're not really delivering value -- you're solving engineering problems. You can sacrifice that control and solving problems for greater abstractions, i.e. serverless, fargate, etc.
Unless you have some extreme security requirements (which most don't) you probably don't have a good reason to run bare metal.
Of course, if you don't need comparable specs to an entire physical server (I do a lot of personal stuff on a t2.micro!) or don't need the same capacity 24/7, the math is very different. The cloud is definitely cost-effective for elastic workloads.
I would disagree here. You can run the metrics for your stack, but it's likely cheaper on AWS. https://awstcocalculator.com/
If you don't need to compute something but twice a day, or just certain times -- you can pay pennies on the dollar for a serverless function. There's no need for an entire server for things like this.
We run about 3 cages (~4k pieces of hardware total) for about a tenth of the cost of what we've priced out as AWS/GCP. This includes DC staff to swap/provision hardware, leases, capex, and our bandwidth contracts.
I worked for a company that had bare metal. They were always shipping parts around, lots of movement back and forth between the data center, lots of night deploys for ops team which meant a burden on the HR department. There was a general anxiety about security.
It might be "cheaper" on paper to run bare metal, but it means your managers and executives have to make a lot of dinky little decisions all the time. I'd happily pay more to "push and forget".
E.g. WhatsApp have been on SoftLayer for a long time and are still there many years post-acquisition.
What kind of price differences are you thinking about? How much hardware and how many people are employed to take care of this? Does the tripling of cost take salaries into account?
Having to hire a team that will maintain hardware, deal with scalability and give guaranteed uptime isn't cheap and requires a lot of management. I'm not saying it never makes sense, but offloading all that work on to someone like AWS and Google who have teams doing this for you reliably can be well worth the cost.
(Disclamer: I'm a Cloud Solutions Engineer at Google Cloud)
also most cost savings come at night, if you are not active on the whole globe you probably only need half of your resources at night. even if your global the chances are high that at some time you still need way less resources. etsy is a marketplace on most marketplaces there are times where demand is really high and demand is certainly not so high. (christmas, black firday, regressions, etc..)
also if you self host, high availability can and is a pain. monitoring your database is a pain, it takes a huge amount of time, which most companies do not have.
p.s. working at a small company and services like RDS/gcloud postgres! (thanks google) google buckets (not sure how they are called their name is not as cool as s3)/S3 are a huge win for us.
Aside: strangely enough googling GCS gives me Glasgow Coma Scale, you'd think by now google would have personalized the result for me
* elastic demand, either in the sense that your global traffic has peaks and troughs, or your global traffic is steady but its global distribution changes such that the balance of where you're deployed would ideally change over time; in either case you can benefit enormously from automatic scaling
* workloads that can take advantage of the spot market (or whatever GCP's equivalent is)
It's easy to say you can buy a huge server for a few thousand but the "cloud" provides incredible agility, security, and basically removes operational overhead. When compared to buying, maintaining, and decommissioning servers and software along with the time and personnel costs, it becomes a much closer race. Include global scale, elastic capacity, managed services, and support policies, and it's closer still.
Rarely are companies paying 3x IT spend just to run in the cloud, especially for large companies like Etsy.
I think you can imagine a scenario under which the end goal to have this headline was someone’s primary goal for the past couple of years.