Handling 1M Requests per Minute with Go (2015)
marcio.io
marcio.io
I'm ... not very impressed.
If you read to the end:
"After we had properly configured our cluster and the auto-scaling settings, we were able to lower it even more to only 4x EC2 c4.Large instances and the Elastic Auto-Scaling set to spawn a new instance if CPU goes above 90% for 5 minutes straight."
From last time's discussion on this (https://news.ycombinator.com/item?id=9845820):
This comment is kind of insightful:
https://news.ycombinator.com/item?id=9846606
"I was confused by the numbers at first, so: that's 17k requests per second, spread out over 4 dual core Xeon (Haswell) machines, which works out to just over 4000 requests/s per machine. It's still a respectable number, but it's much closer to what one would expect given the task."
"Don't get me wrong, the most interesting part is definitely the implementation and as a Go noob I found it very useful - it's just a bit misleading for the headline to sum your request rate across all parallelized machines."
Edit: I read the comments at the end of the article and my question was answered.
S3 is an absolutely terrible financial choice for systems that need to store a vast number of tiny files.
A key-value store would probably work well, depending on how well its storage layer is architected.
There's a big caveat to any NoSQL database and that's how you handle aggregates/roll-ups. With a standard database it's easy to write these queries. If you do it without thinking on a NoSQL system it'll cost you in performance and where billed per access, money. There's a few ways to address this;
- batch ala map-reduce.
- streaming ala Apache Beam, Spark, etc.
- in query counting (aka sharded counters).
- Up to 1M writes per minute, or ~16k/second. Obviously they don't operate at full capacity 24x7 but let's assume they do for worst-case planning. That means they need 16k WCU, or $7500/mo.
- Their RCU would obviously be significantly lower, let's say just $500/mo for analysis.
- Storage is going to cost $0.25 per GB-month. Let's say a very small payload of 100 bytes per object + 100 bytes for Indexed overhead. 200 bytes * 1M/minute * 1440 minutes/day = +288 GB/day. By the end of month #1, you're looking at $2,160/mo. By year #1 (105TB) you're spending $26k/mo.
At the end of the day, they're going to need to massively scale down their write throughput and constantly perform rollups/aggregations or they'll go broke if they ever want to simply store the raw/original data using S3 or DynamoDB.
Cost analyses like these make me very hesitant to use S3 or DynamoDB for an operation like in this article. Sure, individual operations are a fraction of a penny, but it grows very quickly.
It seems like it'd be far more sensible to just get a few EC2 instances and run something (cassanda, sqlite, spark, hadoop) locally.
1. desired time to market.
2. operating cost.
3. people cost.
If you're dealing at that level of traffic you're not going to go broke unless you lack a decent business model. One of the companies I worked at was spending $100,000+ per month on AWS (more than they should've for the incoming traffic level but they survived for a number of years). A friend was working at another company that was spending $3,000,000+/month. The motivation for using DynamoDB/Datastore over say S3 is that you can do range queries, field sub-selection, offloading of systems administration, some elements of security, etc.
Most traffic patterns have windows and tend to be diurnal in nature. This depends on the number of timezones your service covers. Obvious exception being IoT type time-series. So you might experience peaks of 16k/s but that will be for perhaps a few windows in the day. I also wouldn't be treating DynamoDB as permanent "long-term" storage for granular data. I would be using a sliding window of whatever I consider to be a valuable retention period (e.g. sliding window of 30 days). If you don't you'd need to scale C* as well.
In order to run Cassandra and get streaming aggregation via Spark you need to run "2 regions". One as the primary online region for serving requests and a second region for stream processing using Spark. Minimum recommended memory requirement is 16GB for a C* node. For a C* + Spark node minimum is 32GB. The more the better for both. If you've selected your keys well C* will scale almost linearly with the addition of nodes. Using Netflix 1,000,000 writes/second as a model you could roughly state that dividing nodes (285) by 60 should yield an approximately similar throughput requirement to the articles. So about 5-6 t2.xlarge nodes could potentially work for the FE cluster, and whatever gives you adequate streaming performance for the streaming cluster, as a minimum 3-5 x t2.2xlarge. Let's say ~$2,000/month.
Another thing to consider is that the management and maintenance of a C* cluster doesn't come for free either. So figure you'll need a minimum of 2 sys-admins to administer the cluster. Depending on where you are that'll likely cost the company upwards of $400,000 per year or $33,000k/month once you combine base salary, pension contributions, taxes, etc. You could say 1 sysadmin but then you're going to burn out your sysadmins if 1 guy is always on-call.
So assuming a sliding window of 30 days that's $2,160/day for DynamoDB. With a C* cluster you're looking at a minimum of $1,100/day. More depending on how well it performs for what your system is doing. If you're not using a sliding window you'd also need to scale you're C* cluster as well. In order to configure the C* cluster in a way that provides back-ups, auto-recovery, etc you're probably looking at about 2w-2m depending on experience. Whereas DynamoDB time to market will be probably a 1-2w.
I was so happy at Go performance and since that day I realize just how slow other language is. I found Java is quite comparable to Go as well. I was coming from Ruby/PHP and I didn't realize how fast other languages are until I tried it.
1: There's basically one way to do anything, even if you pull in a library that collects those one ways to do another thing.
2: Those one ways are each developed by some of the most skilled engineers in the world and documented by someone with amazing clarity. (https://golang.org/pkg/)
I have always had a lot of trouble trying to learn to program (used to struggle more with focus and memory), but Golang is just easy. I can look at an API I wrote as a learning project after 6 months of not writing any code and understand it perfectly.
Node was not so bad when things worked, but I hit the same brick wall I did with [insert language] when something didn't go like the guide said it should.
CPU bound? For sure.
Fake CPU bound (as this one is, spending tons of time in go's concurrency machinery), clearly not.
I/O bound? I would have said no before Spectre and Meltdown ruined all our intuitions about performance. It's just a mess right now.
... Yeah basically it is an article about using concurrency.
This certainly doesn't read like "I'm talking about the title" to me.
Also, in other parts of the thread you take performance questions seriously, further diluting the impression your post was about the title or phrasing.