Serving 39M Requests for $370/Month
trackchanges.postlight.com
trackchanges.postlight.com
A lot of the comments here are comparing bare metal/virtual servers to a combination of two+ products with literally dozens of built in capabilities, security controls, built in scaling, monitoring, logging, deployment mechanisms, and built in high availability. These are not apples to apples comparisons in the least.
Fwiw I've run my own metal at some fairly large websites(can go find me on Linkedin) and I can honestly say that to discredit the amount of time you spend on solving the above capabilities WELL is to probably toss more than 120% of the cost out the window. People time to manage what goes on the hardware is almost always higher than the cost of the hardware. I've spent 5+ years working with everyone from startups up through massive enterprises and only the top 10% does any of this well when doing it themselves.
Taking some of the other examples here, let's say you do try and solve this with 2 virtual/physical servers from a traditional hosting provider. What now do you do for: high availability, nearly instant scalability, monitoring, logging, alarming, deployment and security controls. Those last few all scale on their own too (oh you've never filled a harddrive with logs before?). Are you going to run all of your software on these 2-3 hosts? What happens when they go down? How quickly are you recovering? What impact is there to your customers during this time? What value do you place on that downtime? How many of your tools just failed? Lost data?
How about patching the OS, the software you run on it, your tools, your languages/packages? In busier environments where I ran my own logging, monitoring, metrics, alarming, backups, software, firewalls, etc + the operating systems those things ran on + the modules/packages most developers use these days in languages such as node.js and python, you are talking about whole days in a week to track, test, and understand these changes and the risks associated.
I'm a #serverless zealot by trade sure, and I am happy to be called out on that, but these comparisons of a very high level platform that essentially removes most of the day to day care and feeding of your own "iron" can not be compared to running your own iron at just that iron's cost.
I think the confusion stems from the fact that we don't really know how expensive a "request" is. Many applications can do 1000's of requests per second per machine (say 15K/sec).
Extrapolating the 15 requests per second = $370/month in the article, my hypothetical one machine application will cost $370K/month on lambda. For that, I can rent colo space, hire people to run all the services you mentioned, and gold plate the racks while I'm at it. That is crazy overhead for one machine worth of workload.
The GB*sec of a request in my extrapolated workload is certainly wrong, but I suspect a lot of people ran this calculation in their heads.
I don't really know what a request does in this context, except that version N-1 was 100x more expensive than the current one.
I think the right takeaway from the article is that spending some time to optimize workloads that are costing O(one engineer salary) to operate is often worth the effort, and, for low traffic stuff, it is hard to beat one-size-fits-all infrastructure like lambda.
On per request pricing this is something you can do with Lambda now semi-trivially. You can figure out average Lambda execution cost based on duration of execution and memory you have configured + API-GW related request costs. Depending on your DB you can potentially tie this back as well (harder with some DBs vs. others)
I'd say that while we see cost savings for most serverless platform customers, like an overwhelming amount honestly, we also see companies adopting it for the benefits of time to market, reduced operation burden, and the overall capabilities that they don't need to write themselves. Hard to ROI some of that. While this article did take on a cost based view I'd love to see more folks talk about how these "softer" benefits impact them.
Thanks for your reply!
- Chris
I'm not here to pick a fight against AWS. You guys have built a powerful ecosystem and it's definitely applicable to certain business cases.
Your points about building the same ecosystem as AWS on traditional hardware are absolutely correct, this is very difficult to get right. A lot of places simply don't. I have always said that AWS is great for loads where you need to start be able to easily scale up and down, something which is prohibitively expensive to do with your own servers.
But what you might not see in your position inside Amazon, is that for most companies it's much easier to hire developers/sysadmins for "traditional systems" than it is to find people with deep AWS knowledge.
Because AWS is such a large ecosystem the market of individuals with the domain specific AWS knowledge that your business needs is much smaller than regular old sysadmin/developers.
This means either it's very hard to find the right people, or you have to pay them multiples of what you would pay a standard developer/sysadmin. It's the difficulty of finding people to fill the role that vexes me most.
> What now do you do for: high availability, nearly instant scalability, monitoring, logging, alarming, deployment and security controls.
Ah, but you are assuming people who are using AWS are adhering to best practices and have all these set up, tested, and working in production. I've worked at a few companies with on site and AWS presence, and I can say that none of them followed best practices all the time...
In my experience from pre-AWS days hiring good Ops people for systems, networking, etc is hard no matter what. I'd argue that the developer's role/day to day changes little in the move from traditional "on-prem" type infrastructure to cloud, even with serverless platforms in play. Generally speaking though I'd aim to hire people who are hungry to learn and have enough fundamental knowledge of networks, operating systems, and hardware to be able to ramp up on public cloud technologies. If you learned it one way, you can learn it this way and there's a massive amount of people here at AWS at least to help you along the way (training folks, support folks, solutions architects) as well as 3rd parties like ACloudGuru, Linux Academy, and others with training/consulting capabilities. The future of technology is moving towards public cloud and I'd argue serverless application development models whether you want to get on board or not. No one cried for the last blacksmith in the town when cars replaced horses.
Best practices are hard for people to follow no matter the technology, and the easiest solution is rarely the "best" (see people leave SG's wide open, open S3 perms, etc). We at AWS know its our job to make security best practices easier to grok and implement and there are tools that help with that today (IAM testing/reviewing tools, trusted advisor as examples). There's more we can do here.
Thanks for your reply!
- Chris
No argument here, good talent is hard to find. I guess what I meant is, mediocre talent doesn't hurt you as badly with traditional on prem because there are more guides someone can follow, or there is a lot of institutional knowledge around to help someone in areas they're weaker.
As you correctly pointed out, the cloud is going to be part of our lives going forward. We've already crossed the Rubicon when it comes to the cloud.
Any steps Amazon takes to make AWS configuration easier and more secure out of the box are always welcome.
> open S3 perms
Man it's like every other week we read about some company exposing sensitive data via S3 misconfiguration (NSA contractor data, GOP voter data...). Have you considered selling liability insurance along side it? ;-)
Sounds like an idea for the next YC class!!
-munns
We do a mix of cloud and physical servers, and I'm pushing our company to go more serverless (and cloud), but that could easily change based upon which projects gain traction. Some are high revenue like ecommerce, but others are more ad based and the cloud would probably dig into margins too much. We also aren't a VC backed startup nor are we a huge business so the needs are different. But at our scale, colocating, much less virtual servers, is not even a full time job.
And when I say requirements, I mean actual requirements. Many like to talk about 100% uptime. Then most seem to build their project to only work on a single region of AWS. Single region SLA of AWS is 99.95%. Guess they didn't need that 100% uptime. Of course, multi region is much harder, which is why they do it. But still, I wish people would stop talking about benefits they aren't actually taking advantage of. It makes it difficult to separate the signal from the bullshit.
Yes. Where I'm at, I like to believe we solve this reasonably well, and once you factor in all the redundancy for HA, DR & all the engineering around proper monitoring, security, log management... we find Lambda very attractive. Though in our case we tie Lambda back into our infrastructure, orgs that are expediently trying to solve a problem (greenfield projects) and don't have the staff to set all this stuff up, take a deep look at Lambda. At worse, you can always move your lambda functions into your own servers once you have the resources to devote to setting up all the stuff that goes for highly available setups.
API Gateway doesn't even come close to EC2. I could see maybe 200%, but not like 5000% increase. 39M requests per month is basically nothing.
I'm just guessing, but maybe it's related to the inherent flaw that Lambda only processes with a concurrency of 1 (per func container), instead of utilizing Node / Golang concurrency capabilities.
well it's good that more and more vm providers like DO allow you to create your own network. But many still do, so you need some kind of vpn to securly create a network on top.
> A lot of the comments here are comparing bare metal/virtual servers to a combination of two+ products with literally dozens of built in capabilities, security controls, built in scaling, monitoring, logging, deployment mechanisms, and built in high availability. These are not apples to apples comparisons in the least.
Yes, I acknowledge that buying a pair of dedicated servers will not give me the things you mention however these things are coming at a cost and while they're nice to have, when you're running at a scale where a mere 2 baremetal servers can handle peak load, I'm dubious that you really need these things enough to pay ~2x to have them.
> What now do you do for: high availability, nearly instant scalability, monitoring, logging, alarming, deployment and security controls.
HA: OVH has load balancers too. Logging: Log to disk, rotate logs, compress them and send them to somewhere like S3 for further analysis with a cron job if desired. Monitoring + Alerting: You have competitors like NewRelic and DataDog, they're pretty trivial to set up. Alternatives can be self hosted when things get too expensive. Deployment: Depends on the app. Auto-updates on the servers and a private package repository could be enough. Security Controls: Depends entirely on the application.
Though not everyone needs all these things and AWS adds $200/month in additional expense relative to baremetal, which is a lot of savings to be had.
> Are you going to run all of your software on these 2-3 hosts? What happens when they go down? How quickly are you recovering?
Are you going to run all your software on AWS? What happens when AWS goes down? How quickly is AWS recovering?
You have to deal with these problems either way, AWS is no silver bullet and you're going to need to have disaster recovery plans either way. Most dedicated hosting providers I'm aware of will give you a basic SLA to start with and a better one can be purchased later. That covers hardware issues. Software issues are up to you but from the sound of things, the author's software is little more than a proxy.
> How about patching the OS, the software you run on it, your tools, your languages/packages?
Turn on unattended upgrades, pay a sysadmin a couple hundred bucks once every couple years to handle major upgrades if it's too scary to do yourself.
I'm under no illusions that paying AWS is always the wrong idea but the benefits of AWS come at high cost and not everyone is willing to pay it. I'm certainly not. I pay $60/mo for a colo (let's say $100/mo factoring in hardware) and to get the same capabilities on AWS, I'd be paying $23k for bandwidth, $1.4k for disk and $152 for CPU/RAM (an m3.2xlarge on a 3-year reserved plan), for a total of ~$24500/mo, or 245x what I'm paying now and more than enough to pay 2 full time engineers if I so wished.
Does your business have an accountant? Why not just get a high-school grad - after all, totting up numbers is simple maths, and it's much cheaper.
> You have competitors like NewRelic and DataDog, they're pretty trivial to set up.
Ha! Complain about the price and then say "just install NewRelic". It's pretty clear from that comment alone that you've just got an axe to grind with AWS.
I'd liken AWS more to KPMG, Deloitte, E&Y or PwC than an accountant. You don't need your accountant to have a risk management consultancy any more than you need your server host to be able to provide instant and infinite scalability.
> Ha! Complain about the price and then say "just install NewRelic".
DD and NR are still cheaper than the 2x cost of using AWS and I also suggested self hosting. InfluxDB, Telegraf and Grafana are pretty trivial to set up.
> you've just got an axe to grind with AWS.
I only object to using AWS for everything or blindly. People don't seem to see the costs of using AWS or any of the other big cloud providers.
No, they're actually not. They're trivial to get to a state of "the process is running on my machine", but to actually get a set of suitable metrics presented the way you want them, that takes a while, especially if you're not already a skilled sysadmin. It's certainly more than "a couple of hours of sysadmin time every couple of years" that others in this thread suggest.
Time and time again I've seen applications painted as "trivial to set up", then when you go to actually try it, the devil is in the details. Getting it running is only the start of it. Unless you have the simplest of use-cases, the stuff that comes out of the box is rarely enough.
Ultimately, this is all penny-pinching nonsense at these numbers anyway. If your shop is bigger than a handful of people, $400/mo is chickenfeed compared to other business expenses.
Your own use-case may indeed not be suitable for AWS - they do gouge you on traffic, and it looks like you're pushing a quarter-petabyte over the wire monthly, so that's certainly not a happy marriage.
I said what I said not as speculation but out of experience. It's really not hard. Telegraf includes many of the metrics you'd want built in and others can be sent with minimal effort from most applications with StatsD.
I suspect you could fit this into a pair of dedicated server s from somewhere like OVH or Hetzner for <$150/month (two servers to provision for peak load, which I'm assuming is ~3x as high as the troughs).
Is there any reason why this wouldn't work?
WordPress is notorious for non-cached requests being very heavy as well. Hundreds of PHP files and database queries for a single page isn't uncommon. And the PHP opcode caches suffer from an avalanche problem at cache expiration.
I see only negatives, don't think about doing it that way.
While there are benefits, the biggest one the OP pointed out was reduced costs. The GP post was pointing out you could still likely achieve the scale the OP posted about for half of what they're paying on traditional hardware.
Anycast is nice, but niche.
Would it be worth introducing that support dependency for ~$200 a month? Maybe, I guess it depends on what you're making from the service.
Not sure about the price, to me $75 for a decently beefy server seems low? Compared to the cheap stuff that most people use because most people are only at like 1 or 2 r/s.
Also serverless might sound silly but isn't it nice to not have to do ops works? Surely worth $200/month.
You can argue whether Hetzner is reputable hosting or not, but $75/month will get you a v3 Xeon with 32GB of ECC RAM.
> Also serverless might sound silly but isn't it nice to not have to do ops works? Surely worth $200/month.
Hosting something on Amazon doesn't absolve you of operations work, it's just a different kind of operations. Instead of managing the OS/software updates, you need to manage S3, IAM, CloudWatch, optimise time/memory consumption to reduce cost, etc.
Maybe we just have different definitions of "ops works" but it bothers me when people claim that you don't need to have anyone minding AWS because it's serverless. You still need to have people monitoring, managing permissions, optimising to reduce cost, pushing updates, etc.
Thank you. Instead of being able to find a server ops guy from a large pool of experienced folks, you now have to find someone specifically with AWS-ops experience - a growing but much smaller pool of people, ime. Getting IAM permissions correct for complex stuff is just not simple, for example, and there's some use cases IAM stuff doesn't (or didn't) cover.
Had a client who couldn't give me EC2 access because - at least at the time - they'd have to give me access to all their EC2 instances, not just the ones I needed. Now... this may have been IAM-inexperience, although, at the time, my digging seemed to indicate it was the correct conclusion.
(I have clients I recommend AWS to as well, but reducing ops time and/or reduced cost are rarely if ever part of those discussions; AWS is the luxury option)
Sometimes it's a matter of politics and trst - AWS is trusted, and if their hosting cost doesn't add up to much of their total costs, then that trust is sometimes more worthwhile than squeezing the cost down. E.g. I'm working on a project for a VC now, and their yearly hosting costs are likely to be below what I'm charging them for the initial development of their system, so spending time selling them on a cheaper provider is just not worth it for either them or me.
Yes, and too many companies don't realize that by moving to AWS, they are not only narrowing the pool of developers/operations people they can hire from, but the salary of these people is higher.
I'm not against AWS, there are absolutely businesses where AWS makes sense (as the article describes). However it's more than just a question of dollars and cents to move to AWS, you also have to factor in the HR consequences of losing access to dime a dozen developers/sysadmins.
For most businesses I have worked with, I would argue losing access to developer/sysadmin talent outweighs their perceived benefits of moving to serverless.
- making sure you don't run out of inodes
- having to manage DB backups
- making sure your cache didn't stop working
- updating your box for security updates (beyond just your application requirements)
It's a tradeoff (architecturally as well), and for a lot of stuff you'll end up wanting full-ish control of your box. But for getting the ball rolling it's nice to not have to first spend a bunch of time getting postgres to start consistently on boot.
You still have to actually manage deploys themselves, of course. And keep an eye on costs, too. But a lot of these services are getting close to "the code that you are running locally, but in the cloud". A far cry from "OK, first lets get apache up, then set up some proxy servers, then set up supervisor" etc etc.
I wasn't trying to list every bit of ops work for server/serverless. My point is that as a business you cannot expect that once you move to serverless you don't need anyone to manage it anymore, or that the time to manage it drops to zero.
Yes, serverless does have its distinct advantages over rolling your own, there is no question about that. But serverless still requires experienced people to configure/manage/monitor it. I've seen way too many managers get giddy at the thought of having their developer/ops time completely free to perform other tasks, and that's not the case. You still have to budget for people and time to manage serverless architecture.
That doesn't happen when I deploy on dedicated servers.
It really does depend on your setup and cloud guarantees nothing.
Also, 2.37s for parsing a HTML doc and then making it more readable is awfully slow, assuming it gets sent the HTML doc.
http://reviewsignal.com/blog/2014/06/25/40-million-hits-a-da...
39M Requests for $370/Month is an order of magnitude more expensive then the average web application.
I read through the article and it did not give any explanation except that it hints at a big database: "database had grown to store a massive slice of the internet".
Without actual numbers and a description of the workload, it's hard to say if the described solution is good or bad.
For reference I've had times where I was getting ~1000 requests/day/site on a $17/year VM. This was done with Cloudflare caching and caching PHP output.
This comes out to...
* `ls /var/www/ -1 | wc -l`: 17
* (1000 * 30 days) = 30000req/month/site = 30k * 17/month
* 17 / 12 = $1.5/month
That means my request, on a crappy VPS serving Wordpress, some custom static sites, and a few other services (PayPal API handling, emailing, uptime monitoring, etc), cost only 30% as much as the requests described in this post!
PHP-FPM, Lighttpd, and MySQL's small config are magical.
>> How We Reduced Our Hosting Costs by Two Orders of Magnitude
So it was costing them that much more, before. Wow.
That's not to say Lambda was the wrong choice here, the overall cost is pretty low and it's hard to argue that maintaining their own servers would end up being less than that.
That's at least the third rewrite of the parser. The initial version was a js bookmarklet, I did the rewrite into Python in 2008 which kind of sat around and then got turned into the app by a different group about a year later. Fun to see it go back to JS.
I don't want to be the guy saying that 39 million requests isn't a lot, but especially if your API serves trivial responses to a lot more requests, API Gateway can end up being the opposite of cheap.
But my point was more about simply being cognizant of the API Gateway cost. Too many Serverless tutorials/testimonials assume no significant request volume and end up ignoring this aspect of AWS billing.
I see so many cases where people are bewildered at how they've come to spend as much as they do because nobody understands the cost model.
Obviously YMMV. Does anybody else have anecdotes about Lambda's pricing helping or hurting their application?
Maybe I'm just spoiled because I can put crawl/parse tasks in a work queue for each url without much concern for latency but I've gotten 1000 r/s (on async calls) out of $100/€100 dedicated servers.
Their setup really only makes sense for WAN facing public architecture that can spike because you have no control over the load. Otherwise, I'd suggest you just get some leased dedicated server(s) to meet your workload.
How big is each request? How peaky is it? Etc.
I had to read a big chunk of the article before I realized what units were and did the math to get the 15 rps number.
That brings me to my next observation: I don't want to demean the author, but 15 rps is really not extraordinary, so I'm wondering why these requests are so expensive. I assume there is a good reason they need to focus on "scaling" that many, I just can't figure out exactly what it is. They might be doing a lot of very inconsistent throughput with hard to predict peaks, I guess...
Head over to http://serverlesscalc.com/ and enable the "HTTP Requests" radio.
You find some website that's utterly polluted with content, click the Mercury Reader and you've got a very clean readable article (similar to Firefox's Reading mode).
I keep thinking, it'd be nice to have a Lynx Browser-like proxy that reformats existing webpages into readable, but minimum markup. Sort of like AMP-everywhere perhaps. The purist in me only wants to download the 2kB of actual information from most sites, and not the extra few Meg for tracking, ads, pointless styles, frameworks etc...
For example: We have a fairly consistent load of very small, short api requests (Billions a month). These requests can be easily served by a couple of small servers. We calculated API Gateway + Lambda to cost 30 times more than the solution we rolled on Elastic Beanstalk.
We have other services which run for nothing on API Gateway + Lambda. Most of them aren't doing very much though (< 1M requests a month).
Most web applications require a higher degree of persistence.
It is good but some more traditional way would also be fine as well. I kind of expected to see Elixir or Go in the backend handling the load more efficiently.
DPDK + DMA aware programming in C++
39 days/Month
80K sec/day
1M req/day ==> 12 req/s
They mentioned orders (plural) of magnitude difference in cost. Which means it was closer to $37,000/mo previously.
39 million requests X 2.39 seconds per request = 93.21 million seconds of compute time performed each month. That works out at 2.95 years of computation for $370.
Although he mentioned that about half the traffic comes from cached results. So maybe it is more like 1.5 years of actual new computing each month. Compare the $370 against the AWS cost of running a VM for 1.5 years to do the same workload and I think your saving a lot of money.
It is 2.95 years of computation on less than 1 CPU core.
http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduct...
> AWS Lambda allocates CPU power proportional to the memory by using the same ratio as a general purpose Amazon EC2 instance type, such as an M3 type. For example, if you allocate 256 MB memory, your Lambda function will receive twice the CPU share than if you allocated only 128 MB.
https://aws.amazon.com/ec2/instance-types/
> m3.large 2 7.5 1 x 32
> Each vCPU is a hyperthread of an Intel Xeon core except for T2 and m3.medium.
1 Core per 7.5GB RAM. Or ~1/28th of 1 core.
(2.95*365.25)/28 = ~39 Core Days.
That is really only impressive when you are comparing AWS prices to AWS prices which is a trap people seem to frequently fall into imo.
$150 (at most) dedicated server for instance.
No. I've pushed 1000 r/s (approx) on a price tracking tool at work for keeping track of competitors on one of those boxes. Their largest spike (even if you assume its 10x their average) would be at most 15% of that.
A) I'm not sure "one location" is a city, country, or province.
B) Yes, it was global as we had offices from China to the EU.
C) Given the OP is handling 15 r/s, 150 r/s is not going to be pushing 500 mbps unless something is very poorly designed.
D) Even at spikes of 40x, it would still only be 600 r/s which is still quite doable with a commodity server.
The only serious difference is it was very large batches of requests (20 * SKUs) so people were fine with waiting a few minutes.
As a concrete example, say you have an ecommerce app. Read requests take 80ms while write takes 200ms. Any good article/book I can read to get more insight regarding this?
With even moderate caching (say, 1m), you can saturate your VM's bandwidth before running out of CPU or memory.
Ultimately, you will always have a bottleneck. All you can do is identify the bottleneck (via load testing) and address it. If it's network (and you're in Amazon), you have to scale vertically until they remove that limit, or horizontally if you're already on unrestricted instances. If it's the CPU, you'll need to be more aggressive at caching completed computations, or work in a language with less overhead. If it's memory, start evaluating the tradeoffs between CPU and memory. If it's your DB, start investigating why it's slow (it's probably a poor index or configuration; DBs on decent hardware can handle millions of requests per second before degrading).
I don't have any book recommendations; I just recommend creating something and find its limits with a simple load test tool. Then figure out your bottleneck and fix that. Rinse and repeat, and learn.
> As a concrete example, say you have an ecommerce app. Read requests take 80ms while write takes 200ms.
An ecommerce site can be nearly 100% cached except for cart/session/search data. This gives it a massive advantage over everything else (since you can load the non-cached data via js and its relatively tiny).
I think the last time I benchmarked one of our front end servers it was ~500 r/s on 2 cores for our search page.
You really aren't going to know until you try it and benchmark it yourself tho.
It's unlikely that you'd get cheaper raw compute from any cloud vendor when using Lambda-like products, compared to their IaaS offerings. The savings are in lower ops costs, easier auto-scaling, integration with tons of other AWS products, no minimum cost for services with a small amount of requests, etc.
> In the last month, 52% (20.3MM out of 39MM) of our API requests were served from the cache, meaning less than half (18.7MM requests) required invoking our Lambda function. That $14 saves us around $240 a month in Lambda costs.