How much is Spotify paying Google Cloud?
medium.com
medium.com
The blog post says "We also added queuing and batching in the client. (This wasn’t first time that we needed to roll our sleeves up and reimplement a Google Cloud API client: in another project we implemented a high performance client for the Datastore API.)"
> Batched calls are accounted for as multiple operations.
"Pub/Sub passed the test with flying colours. We published 2M messages without any service degradation and received almost no server errors from the Pub/Sub backend. Enabling batching and compression on the Event Service machines resulted in ~1Gbps of network traffic towards Pub/Sub."
I have 64 1k messages? Ok, I am going to put them inside a single 64k message and reduce my request count by 64x, and profit!
If you look at the API, the publish API allows you to push multiple messages in a single API call, and if that is the batching - publishing multiple messages - then yes, you and others are absolutely correct that there is no cost benefit.
But rather than publishing
{ "messages" : [ { message 1 }, { message 2 } ] }
You publish
{ "messages" : [ { container message containing both, still less than 64k } ] }
Then you've reduced your operations count by half. Cloud services like this have a per message overhead, for things like auditing and delivery tracking, which is why the pricing model is what it is.
If you are ok with having to unpack and process multiple messages at once as I've described, then it's a way to optimize against the pricing model.
Even if practically that number is much lower (around 100 per batch), the cost ends up becoming around $3,000 for the messages and $9,000 for the network, so roughly $12,000 per month.
64 1k messages for Events, sent individually or in a batch
1 64k message for BatchedEventi.e.
# serialized: 1 message, 1 op
publish("['foo', 'bar']")
# instead of:
# a) 2 messages, 2 ops
publish("foo")
publish("bar")
# b) 2 messages, 1 call, 2 ops
publish("foo", "bar")A software engineer at Google earns, on average, $127k per year [1]. We multiply that by 1.4 [2] to obtain a cost per employee of $177k.
This means that according to this estimation, Spotify pays for 20 Google software engineers.
[1] https://www.glassdoor.com/Salary/Google-Salaries-E9079.htm [2] http://web.mit.edu/e-club/hadzima/how-much-does-an-employee-...
On the other hand, this blog only estimates the cost for Pub/Sub, when Spotify is using a whole slew of other Google Cloud services as well.
That ignores RSUs, signing bonus, and other bonuses, which make up the majority of the compensation package for senior engineers.
Even at entry level, total comp (ignoring insurance, cost of office space, free food, and other things that aren't directly paid to the employee) is over $127k/yr. A baseline new grad offer last fall included $180k of stock, or $45k/yr, on top of a six figure salary plus a $50k signing bonus. This probably goes without saying, but new grad offers can go significantly higher if the new grad has strong competing offers, and senior offers start out even higher and have a much larger range.
Dedicated servers create a different kind of lock-in: Dynamically scaling up dedicated servers is like laying down the tracks while the train is moving. Acceptable to some, but not to a company like Spotify.
Dynamically scaling load is only really interesting if the load is extremely variable. You could just buy 5x the dedicated hardware, which will likely be able to serve 10x the traffic and may still end up paying less. Who cares if it sits idle most of the time.
Now Spotify, like Netflix probably has extreme variance in load, much more so than a normal company. So maybe the cloud works for them. But that's not a common situation at all.
Our customers like it because their existing ops staff can greatly reduce their workload and allow development staff to move much faster.
The thing with them is if they go down you have Google to worry just as much as you (if not more) for bringing them back up.
You don't have that with your own servers.
Googlers might be more skilled than your ops team, yeah, but I would never bet on huge companies to worry about the fate of something that is not their main revenue generator.
I think cloud services are appealing for a couple of reasons from a top-brass perspective.
a) You free the capital, and the balance sheet looks nicer.
b) It's less complicated on a PowerPoint level. Just a box. (probably not the reason Spotify opted for google though.)
c) You don't think it's your core competence to keep servers safe and running, and rather would like to spend the time and effort on developing services while enjoying reason a) although you know it won't be cheaper.
d) Maybe it's hard to find the right people
> Add a second for failover.
That's not how it works. How does the second server pick up where the first server left off, when the requests are stateful? How does it know when to take over? (What detects that and how?) Or if it's a human, how does that human make the judgment call?
Cloud services are absolutely appealing from an engineering perspective. I am capable of building an application and running it on a fleet of machines such that the infrastructure takes care of itself completely and 100% by itself. Essentially no failure mode can involve me, unless either (A) my application has a defect in its logic, in which case it's unavoidable under any solution (B) probably a lot of customers are similarly affected, and someone is working on it. The data stores that I use are hosted and will not be impacted by any typical failure mode. The application that I've designed is deployed onto machines automatically, and if any of them encounter a failure that takes them offline, they'll be replaced automatically in a way that's a non-event. Furthermore, I have a blueprint for such systems such that building a new application in this way requires no more effort than filling out a single web form, after which an automation layer sets everything up, from the source control to the continuous build and deployment onwards.
Not all systems are simple enough to fit into this model, but if you account for it while designing new software, quite a lot of it can be. With the right building blocks you can operate very operations-free infrastructure at the application layer even at large scale. These problems might start to look different once you consider challenges that require tens, hundreds, thousands, or more individual machines to solve.
Another challenge is ensuring that new systems are built in a way from the beginning that can scale and that will be easy for other people to maintain. You don't want some random new-hire at your company setting up the "opaque box from hell" that has all the data on its disk (and, ahem, a "second for failover", that through a mysterious and unknown mechanism somehow takes over for the first) and that no one else will be able to maintain during failure. You also don't want to be surprised a year or two down the line when you're scaling up for the Super Bowl or Tax Day or Black Friday, or whatever, and your software falls over because "Bob the Intern, who thought every software system can be solved with a single box and 100 lines of assembly code" didn't anticipate that the business would be larger one day. If you standardize onto a certain set of building blocks, you can build software that, in addition to being operations-free, can scale to a large degree.
Note: I'm speaking hyperbolically. No software is truly operations-free. It's all a matter of degree. I have worked however with teams that are expected to hand-manage their own machines, and teams that use effective cloud automation, and the pattern I've seen is a huge difference in their operational load from routine engineering tasks. I have also never seen systems that completely 100% automatically scale themselves, but there is still a huge difference between systems where you simply say "I want 10 machines" / "I want 10 requests per second to my data store", and a system like, "I guess it's time to tell our DBA to figure out how to shard our MySQL server".
I also think that a large company could achieve a lot of the benefits provided by the cloud in-house, with its own data centers, if it had really strong internal automation and APIs, and strong teams who operate shared services for other teams (e.g., managed data layers). The really strong software I'm aware of for solving these problems has been kept as a competitive advantage, not released open source. So the practical reality is that few companies have through-and-through automation of the quality and degree that you can get from top cloud providers.
> a) You free the capital, and the balance sheet looks nicer.
Yes, I agree that this is a genuine reason for some companies to prefer the cloud, but these are largely not companies who have "top brass". Companies that are large enough to be "top brass" are typically not so capital-constrained as that this would majorly affect their purchasing decisions. The companies that care about this, rather, tend to be smaller and more capital-constrained.
I think the more mundane reason why companies prefer the cloud, which is frequently overlooked, is that individual business units can purchase cloud services with more autonomy than they could traditionally order IT services in-house. Cloud budgets often fit within what an executive can expense within their own authority on a month-to-month basis, where as the same capital purchase would require more review and approval. So some companies tend toward the cloud because it makes them more agile by providing decisions they can make that are smaller and more easily fit within the scope of less-senior people.
Yes, this is also probably true, as hardware costs are typically very noticeable as they are large but few. The old trick of splitting the bill works fine :-)
Anyway, I'm a control freak. I would very much want my teams to keep control of everything that is important to our core product and services, including servers and infrastructure - and everything that have to be ordered from another party - internal or not, actually more so if it's inhouse, is a potential delay or outage that is not fixed as soon as it is possible.
But then again, everyone should know their limits... :-)
outsourcing allot of those things saves money it's also a different expense of you buy infrastructure it has value which means depreciation overtime which affects how your accounting is done.
Cloud makes sense for allot of companies not just startups the biggest reason why companies with eatablished infrastructure don't migrate is the initial cost and regulation.
Not to mention that often it can takes weeks for new hardware order to be approved / delivered / provisioned.
Cloud? Just dial it up... Well, if you are at Netflix scale, you probably couldn't just dial it up, and would have to work with the vendor to ensure adequate capacity was available for your use.
As for hardware failures in general, you can afford to have some complete extra servers on hand doing nothing most of the time with the amount of money you'd save on renting bare metal instead of paying for cloud. Or if engineered properly, you'd be using all the machines at once but be able to afford to lose half of them, and failover automatically.
Yes, a cloud (bunch of VMs on a cluster) is going to be more reliable than a single machine, as you get hardware redundancy. But a cluster of machines with hardware redundancy is going to be much more reliable than a bunch of VMs on a cluster, as there's less complexity involved without the cloud stack, and thus fewer points of failure.
Public clouds are an efficiency gain for both the datacenter and the end user. The bottom line to me is more engineers are focused on operating and improving the public cloud than the state of the art managed dedicated racks from 2014.
You'd still save a lot of money compared to cloud servers, need little admin work (the same compared to compute instances – amazon's EC2 servers), and have a reliable server. Providing the servers doesn't take forever, either. Some bare metal servers are up and running within 120sec, though, you can't just shut them down and stop paying like cloud servers.
Companies that offer these services are OVH.com and softlayer, though softlayer always seemed to be a bit expensive. I'm sure you'll be able to find similar companies though.
I can see that this makes little sense to startups because the admin work is way more expensive but for bigger companies, this should not be the case (I don't consider Spotify and Netflix a startup anymore).
Isn't that what Amazon does? (even though they are virtual servers)?
You'd still save a lot of money compared to cloud servers,
Can you post some numbers? A 24 core 48GB Dell server with 3x120GB SSD's costs around $11K, or around $300/month for a 36 month lease.
An 8 core 15GB 160GB AWS c3.2xlarge costs around $200 each per month with a 3 year reserved instance. 3 of them will fit in one of the Dells, so it would cost around $600/month, or about $300 more than the raw purchase price for the hardware alone.
But also with that price you get network, power, cooling, rack space, maintenance, etc, so it's not clear that buying and running that server is significantly cheaper.
Some bare metal servers are up and running within 120sec
How long does it take if you want 500 of them? And 2 hours later you want to turn them off. Then 2 hours after that you want to bring them all back online again.
you can't just shut them down and stop paying like cloud servers.
But that's the big value-add in Cloud servers, it's capacity on demand so you only pay for what you us -- need to spin up 500 servers to run month-end processing? You only need to pay for them for the few hours or days a month when you need them.
Further, you can add multi-datacenter or multi-region redundancy at very little additional cost - you don't need to keep those redundant servers running when you're not using them.
See http://www.pythian.com/blog/virtual-cpus-with-amazon-web-ser... for more details.
I have a feeling that a lot of faults that in the google infrastructure result in 2 minutes down time, or even just a raised eyebrow in the ops centre would put most SME's (or event big corporates) on there arse for a week. Do you have a spare generator? If main power goes how long do the UPS keep you going for ? If you have a dc fire what happens... These are things that you can plan for but need real capability to deal with, capability that needs scale.
Take for example something old school and established (ZzZzZ) - accounting.
Say you are an accounting firm with an established IT infrastructure and you now want to launch a new service that will allow new smaller clients to perform certain tasks even a single one (say fixed assets devaluation for deductibles). The core of the service is a messaging API as it's an accounting application it's usage wont be fixed and the product team estimates that on a normal week it will handle 20,000 requests, on a end or the quarter week it will handle 500,000 and at the end of the financial year it will handle 2,000,000 how do you scale this?
Say a single server will 20,000 request so you have here a 1 to 100 scaling between peak and normal usage which means that if you own your own infrastructure you are paying for 99 servers to idle during most of the time. Even if you can some how manage to do some inner-organizational thin provisioning you will still be wasting allot of hardware because as it turns out most of your other internal services also follow a similar usage trend so that might not be an option to begin with.
Now you can rent VPS/IAAS from say Amazon or Rackspace but then you still have a huge overhead sure you'll pay less for usage because those 99 instances will be down but you still need to pay for other resources e.g. reserved IP addresses and storage, you need to practice and test the machines every couple of weeks/months and you need an live-ops/sys-admins with balls of steel to be confident enough to agree to spin up 99 instances at the very last moment and hope that everything will work out smoothly.
But even that doesn't really solve it because it doesn't allow you to be flexible with onboarding new costumers, it can't predict various trends like a change in legislation or a holiday weekend prompting people to use your service earlier / later than expected.
But if you say just use a PAAS/SAAS messaging service all of this goes away, you don't worry about scaling as long as the service can handle your peak traffic, you don't worry about waste because you pay what you use so if you've only sent 18796 messages in week 37 you pay for that exact amount, you don't need to worry about some promotion that the marketing and product dev team was running that caused an influx of new customers without telling you, and you don't have to really worry about down time, backups, and even security (as long as you get sufficient assurance from your service provider) because as long as their service is up you are up and running.
(btw -- I disagree with your math there on the actual cost -- I agree it could ultimately be cheaper to run things in house, there's no way it's going to go from $3.5 million to $20k annually. It'd be more like $3.5 million to $2 million annually. At that point, it's a question of whether or not you want the opportunity cost of having your tech staff engaged in deploying/operating/maintaining server infrastructure as opposed to building new features/etc)
In short, it's insurance.
And if Spotify wants to change strategy tomorrow, they can just end their contract with Google. Winding down internal operations is way more complicated.
If hardware were free, sure.
thanks for the post @dmytton.
not sure if it's accurate It'd be lovely for Spotify or Google to chime in.
http://arstechnica.co.uk/information-technology/2016/02/spot...
If you want to further refine you could probably estimate a large chunk of early morning US/EU time where traffic will be very light. maybe reduce the estimate by 1/3 or 1/4th to account for this.
> Currently our production load peaks at around 700K events per second. To account for the future growth and possible disaster recovery scenarios, we settled on a test load of 2M events per second.
In the absence of any other numbers, using the 700k number seems reasonable otherwise you would start to get into modelling without any real data to back up your assumptions. This at least gives you a maximum they would be paying under current production conditions.
Thanks for the great articles David, keep them coming! :)
Assuming max as the norm is not too right. (just to make the number look high)
Having run global services myself I can tell you that the shape is usually no standard distribution. Peak/trough depends on the exact usage demographics, but the one I have on hand has a trough of ~1/2 peak, and an average of ~3/4.
It's really not fair to tell the world that their costs are based on average of 700k rps, when that is their peak traffic through the system.
I am pretty sure it is also not unusual for average customers on AWS to get some discount, with some reserved instance committment). One such example is probably Redshift which is extremely expensive.
Would be nice to get an estimate quote for AWS but just for SQS I don't expect to be too expensive.
The biggest way in which all such assumptions are likely to be wrong is that they overlook the likelihood that Spotify has been offered a potentially substantial discount in exchange for attempting to win their business, and to be an early adopter. Not that there's anything wrong with that, but it's something to take into account while evaluating what you think Spotify might be paying.
Many cloud service providers offer a discount to, for example, qualified startups participating in certain accelerators. In the same fashion, it would not be surprising for a provider to entice a major potential user and hopeful net promoter with a significant price reduction as well as other perks, such as more access to senior people for operations support, as well as influence on the roadmap.
So while I think this blog post could plausibly estimate what a typical customer with Spotify's usage pattern might pay, assuming the estimates are correct (I haven't checked the details), one cannot draw conclusions about what they're likely to actually be paying.
[Edit: I see now that the article recognizes this at the bottom. At the time I made my comment, it seemed to have been overlooked in the discussion here, where only one comment acknowledged it and the rest seemed to discuss the numbers as factual and face-value: https://news.ycombinator.com/item?id=11242166 ]
>>> Is Spotify really paying $290,518/month?
>>> Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts. Cost is important but using public cloud and especially managed services like pub/sub, the biggest benefit is in ease of deployment and not having to manage/scale everything.
> Is Spotify really paying $290,518/month? Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts.
> For example, in one month, our server costs were well over $4M on Google App Engine after spiking 5M users in the course of two weeks. We scrambled and optimized, but the cost was real[1].
P.S. Howdy David!