Ways to reduce the costs of an HTTP(S) API on AWS
gameanalytics.com
gameanalytics.com
The response from it is about as small as you could make it, however at that volume it is about 150gb a day.
If I hosted this on AWS, the bandwidth alone without any compute would cost $900 a month. Prohibitively expensive for a service I just made for fun.
The cost of just sending the HTTP response headers alone is the majority of that cost to. There is no way to shrink it.
It is currently hosted on a single $40 linode instance and can easily keep up with the ~2400 sustained QPS. I think it can get up to about 50% more traffic before I have to scale it. And linode includes enough bandwidth with that compute to support the service without extra costs.
I don't see how anyone pays the bandwidth ransom that GCP and AWS charge.
Furthermore, if you are doing something for fun like you are, the bandwidth ransom definitely comes into play for elastic cloud environments, but anyone doing anything significant on AWS/GCP has definitely already negotiated down their bandwidth spend with their AWS/GCP account management team.
Netflix is a great example. They run most of their services on AWS. But they also run their own CDN with real hardware in data centers because serving it from Amazon would be a deal breaker.
There are reasons to use AWS and GCP. But when I start a project, I don't start there. It's too expensive one way or another, and the "free" tier gets blown out extremely quickly.
A smaller provider will provide what you need, normally be cheaper, and has no lock in. If you later decide that you really want autoscaling or managed databases then you can move easily. And if you do switch, you'll at least know what your product even wants to be, and it's projected growth.
Sure, it looks like they saved about $2000/month, but consider that those savings probably won't even pay for more than a quarter of a one of their developers.
Even though their service is free (their parent company gets business value from the aggregate analytics they obtain through their service), it very possible that there's something they could have done to bring more value to their parent company than the money they saved here.
Maybe it's unreasonable to expect a company to talk about that in a blog post, but it left me wondering.
- They mention that the initial savings of $1500/mo from omitting unnecessary headers was 12% of their egress cost (so the total before this was $12500)
- Then they got an additional 8% of savings by increasing the ALB idle connection timeout to 10 minutes (down to $10120)
- Finally they said they saved $200 per day by switching to a lighter TLS certificate chain ($6000/mo, so down to $4120)
None of those steps seem to have required any meaningful amount of development work. Let's say this took a developer one week? The return on that effort would be $100k a year, or $2500/hour for the first year alone.
Give and take is an open source attitude. It doesn’t always have to be about source code, sometimes it can be about cost savings techniques such as this.
Although I never run a business, I do believe this kind of optimization is quite meaningful even though they will never be the top priority of a business.
Those optimizations lower operational cost while being mostly maintainance free (except the one that switches off from AWS certificate manager, which may increase some effort when renewing), risk free (unlike refactoring a large legacy system) and requiring little engineering effort (Maybe 10 engineering days from investigation to writing the blog post?)
In addition this blog post itself brings intangible benefit on their branding, website ranking and hiring.
What matters is the return on investment, and as you state, one of the great things about cost optimization is that its returns come largely risk free. By my math the optimizations described here return $100k a year. On a risk-adjusted basis, what task could this developer have performed that would have returned more?
I'd bet the optimization and subsequent write up in a blog post didn't take more than a week to get done from start to finish.
Your server bill should be $100/mo - $200/mo max for a "small" business. I've ran a multi-tenant SaaS platform on a $200/mo DigitalOcean budget (server for Postgres, server for Redis, server for node.js apps) that brought in $30k/mo. If you're spending that much a month on cloud hosting, consider yourself got by the marketing of "serverless".
I don't know what DigitalOcean would charge for servers at that scale.
So you're assuming that configuring nginx properly, once, takes 3 months, every year? If it takes the developer (or sysadmin) less long than that, you're already saving money.
Johnson, you’re fired! I just saved myself 10 grand a month!
Narrator: where do I sign up to work for that guy...........
Another suggestion:
Terminate somewhere else.
If you fit inside of the CloudFlare T&Cs, you can probably save a much larger amount terminating there and having them peer with you using the same TLS every time, or failing that, try someone like BunnyCDN.
I've found that while AWS CloudFront is easy to instrument, it's neither very performant (lots of cache misses even when well configured), or cost effective (very high per byte cost).
Can you elaborate for someone who isn't that familiar with networking? How does this work?
We went through something similar a couple of years ago, when TLS wasn't as pervasive as it is today and at first focused mostly on minimising the response size – we were already using 204 No Content, but just like the OP we had headers we didn't need to send. In the end we deployed a custom compiled nginx that responded with "204 B" instead of "204 No Content" to shave off a few more bytes. It turned out none of the clients we tested with cared about the string part of the status, just that there was a string part.
When TLS started to become more common we realised the same thing as the OP, that the certificates we had were unnecessarily large and costed us a lot, so we switched to another vendor. When ACM came we were initially excited for the convenience it offered, but took a quick look but decided it would be too expensive to use for that part of our product.
Nice deep dive into the S of HTTPS anyway.
https://nilsmagnus.github.io/post/proto-json-sizes/
GRPC streaming is almost as efficient as just using a raw TCP stream, but saves you having to write the protocol glue code. There are already clients and servers that work, and you can just write your protocol definition in the form of a protocol buffer. Worth a look for this use case.
(Also, the clients know how to do load balancing, so you don't have to pay Amazon to do it for you. Unlike browsers, most language's GRPC clients are happy to take a list of IP addresses from DNS and only send requests to the healthy endpoints. Browsers, if you're lucky, try opening a TCP connection but will happy keep the same IP address even if it 503s on every request. Chrome, Firefox, and Safari all do different things.)
that is of course true, but they won't be able to ommit not working/failed/overloaded nodes whereas a load balancer might be able to do so. On the other hand the client might be programmed to just use another IP from the list and resend request if one node fails to answer, but this would increase the total time required by the client to do a successful connection. I also realise that non-responsing nodes might be rare enough for this to be a negligible problem - just playing devils advocate here.
When you control the client, you don't have to resort to L3 hacks to distribute load. You can just tell the client which replicas are healthy. (And both ends can report back to give the central load balancer information on whether or not the supposed healthy endpoints actually are.)
L3 load balancing actually works somewhat poorly for HTTP/2 and gRPC anyway. They only balance TCP connections, but you really want to balance requests. That is why people have proxies like Envoy in the middle; the client isn't smart enough to be able to do that, but it is. But if you control the client, you can skip all that and do the right thing with very little resources.
https://developer.apple.com/library/ios/documentation/Genera...
If you're going on that path It's probably best to avoid revocation altogether, since it doesn't really work, and go the let's encrypt way, certificates with lower lifespans.
On that scale a 15 days cert on rotation is probably fine.
Having just ruled out RSA on an embedded project for exactly this reason, definitely the first thing that came to mind.
If they’re getting down to the byte differences, under their additional options, they really should have had binary serialized data instead of JSON. Something like CBOR “can” near immediate conversion to JSON but it would mean an update to all of their end points and they might not be feasible but could be worked in for new projects over time.
If you could reasonably deploy a website that doesn't offer anything else for https, you'd instantly fix many session establishment-based CPU DoS attacks. It's multiple times faster than what you usually allow your server to negotiate.
Interesting idea from the post: "it could be a selling point for a Certificate Authority to use URLs that are as short as possible"
At my last job we had a fun and exciting outage when AWS simply didn't auto-renew our certificate. We were given no warning that anything was broken, and it apparently began the internal renewal process at the exact instant the cert expired (rather than 30 days in advance as is common with ACME-based renewal). Ultimately the root cause was that some DNS record in Route 53 went missing, and that silently prevents certificate renewal.
We switched TLS termination from the load balancer to Envoy + cert-manager and the results were much better. You also get HTTP/2 out of the deal. We also wrote a thing that fetches every https host and makes sure the certificate works, and fed the expiration times in prometheus to actually be alerted when rotation is broken. Both are features Amazon should support out of the box for the $20/month + $$/gigabyte you pay them for a TLS-terminating load balancer. Both are features Amazon says "you'll pay us anyway" to, and they're right.
Was this some time ago?
The FAQ for ACM (https://aws.amazon.com/certificate-manager/faqs/ ) says:
> Q: When does ACM renew certificates? > > ACM begins the renewal process up to 60 days prior to the certificate’s expiration date. The validity period for ACM certificates is currently 13 months. Refer to the ACM User Guide for more information about managed renewal.
> We switched TLS termination from the load balancer to Envoy + cert-manager and the results were much better. You also get HTTP/2 out of the deal. We also wrote a thing that fetches every https host and makes sure the certificate works, and fed the expiration times in prometheus to actually be alerted when rotation is broken. Both are features Amazon should support out of the box for the $20/month + $$/gigabyte you pay them for a TLS-terminating load balancer.
You're implying that AWS doesn't support HTTP/2 on any load-balancers they offer, but ALB has supported HTTP/2 since launch ( https://aws.amazon.com/blogs/aws/new-aws-application-load-ba... ) 3 years ago.
I don't see any current load-balancer priced at $20/month (ALB, NLB and Classic ELB are all ~ $8/month), so I can't guess which one you were using here ...
"up to 60 days before" includes "five minutes after". What it excludes is the renewal starting 61 days before the cert expires, and, as documented, it sure didn't do that.
Stuff went wrong and we had no observability. That is the AWS way.
https://serverfault.com/questions/836568/terminate-http-2-on...
Digicert is pretty expensive otherwise... always a shock when I look up prices... There is let's encrypt, but I never tested it with anything hosted on AWS.
Still, the article has great tips. And even if your app is some B2B service with <200 users, it still wouldn't hurt to implement the measures. Even if the product owner doesn't care if the solution costs 20$ or 200$ a month. Some of these tips are pretty low effort. Saves energy at least.
Of course it's true that they don't use all AWS services, either because they don't need them or because they had something built in house earlier which works for them.
If you serve a latency-critical service, tickets are a must.
They do talk about it, SSL tickets and TLS session resumption are referring to the same thing.
Encrypt and sign the data via NaCL or similar, send via UDP duplicated 5-10 times, no response at all from the server (it's analytics, it doesn't matter if very few events are lost and you can even estimate the rate).
As for the REST API, deprecate it and if still needed place it on whatever 3 VPS services have the lowest costs, and use low TTL DNS round-robin with something removing from DNS hosts that are down.
It's not surprising that it's related to the gaming industry. Some of the best AWS re:Invent videos I've seen are in the GAM (gaming) track. Even though I've never worked in that field, the problems they get hit with and are solving often are very relevant to any high-traffic site. Because of the extreme volume and spikiness of gaming workloads, they tend to find a lot of edge cases, gotchas, and what I'll call anti-best practices (situations where the "best practice" turns out to be an anti-pattern for one reason or another, typically cost).
So for 10k HTTPS requests, the price is 0.01 $. If you serve 5 billion per day, that is 5000$ a day. With such high traffic I believe it is needed to handle it using performant webservers (Go, Erlang?) to keep costs reasonable, and probably terminating SSL at load balancer is the way to go
Also if they own some/all of the SDK's that are used for hitting their API they could bake in brotli compression at that level.
popular unmetered options: he.net, ovh, hetzner - You generally lose a lot of the "cloud" capability with these options however.
cloud options: digital ocean egress is $0.01/GB ($0.005/GB if you buy it via droplets), linode is $0.02/GB, vultr is $0.01/GB, etc.
[1] https://www.techempower.com/benchmarks/#section=data-r18&hw=...
10 Gbps fully saturated delivers about 3300TB for that $1-2k/mo, versus the $22k/mo you'd pay AWS for the same.
I'm absolutely not talking about the "unlimited bandwdith" bullshit that discount hosts offer.
[1] https://www.cloudflare.com/case-studies/di-fm-eliminates-egr...
Bandwidth is the killer thing with aws. It's designed to make you move services inside the boundary.
Also, the article clearly articulates that the answer is, yes.
But CPU usage is increased for decompression and CPU is the only real bottleneck.
Just because you don't pay for the compression electricity doesn't mean you get away with it.
This ties back to my previous comment on the User-Agent subject yesterday, remove all headers except "Host" from all HTTP traffic is the solution.
HTTPS is a complete waste of energy. Security should not be overarching, it should be precision.
WebSockets are also bad, since they don't work well with memory latency. Use "Transfer-Encoding: chunked" on a separate pull connection instead.
A harmless meme in the US might get you executed in North Korea. Optimizing for energy usage (which is already pretty minor on modern hardware for HTTPS these days) over security is odd.
Just like wind-power actually competes with nuclear because it take 30 days to wind down a nuclear power plant.
Also data can be compressed with more efficient hardware on the backbone without you having to deal with it.
The biggest cost of the internet is idle things and synchronized CPUs, async. never made it unfortunately.
The only thing we need more of is cores and we can't have that because memory is too slow.
What do you think happens to the value of the dollar when the physical supply of energy becomes unstable in the coming years?
Money is energy because debt needs energy to either have been spent in the past or energy being promised to be spent in the future.
The proportion between these is what makes debt money trustworthy or not. When the states around the world privatizes houses that is old energy (so far since the energy to heat the house is marginal compared to building it), but when the stock market goes up that is a promise for new energy.
Globally all money for old energy is saturated (negative interest rates) and now all liquidity is being injected into the stock market that promises that the future will be rich with energy.
The stock market (and all companies) is a promise to spend energy we don't have.
The only energy that is added to earth is sunlight, the only way to capture that energy are trees and plants.
All jobs are now meaningless because of the energy that we are wasting. And people now depend on wasting energy to have a job.
Do you see why energy is more important than money?