Hacker News Reader

Comment by cperciva | Hacker News Reader

At present, we put all incoming page views into SQS

Hmm, not the most economical of choices, but...

We had decided that we were going to simply increase our lambda concurrency limit to 8,000,000 requests per second (800,000 concurrents) and handle the spam attacks

... wait, what? If you're using Lambda functions for something as trivial as logging page views, you evidently have more money than sense.

The author is concerned about this story reading like an advert for AWS Shield, but to me it reads like a case study in when "Serverless" is a bad idea.

View on HN

My main thought here is how important backpressure is for managing queues. We all hate to admit defeat, but sometimes shit is broken and you just have to say "sorry, service not available" and drop the requests on the floor. No buffer is infinitely long.

And to add on this, the plan moving forward is to default to writing data to the appropriate places (aim is < 100-200ms) but fall back to SQS when services are unavailable.

Lambda concurrency wasn't actually increased to 800,000 concurrents. It's a stupid idea that someone would think up after fighting attacks all day.

Regardless of what you set Lambda concurrency to: Are you seriously invoking Lambdas for every page view?

We are indeed. Neither of us enjoy DevOps so we pay a premium to not have to manage servers. It brings a huge mental health benefit (we are a two person company), we're profitable and our monthly Lambda cost isn't that significant. Honestly, the biggest inefficiency (in terms of cost) is our use of SQS/RDS, which we are ditching soon.

Well, it's your money...

I'd be happy to help make this more efficient though (at no charge of course). Offhand I'd say "web server which accumulates data and uploads it to S3 every N requests or M seconds" would probably get you what you need at a tiny fraction of the cost of "lambda which posts to SQS". Create an AMI and toss it at an autoscaling group and you really won't need to worry about scaling issues either.

I bet you could set up something way cheaper. And I'm certain you know 10x more than me regarding servers, hardening, configuration, etc. And I'm certain you enjoy servers!

For us, the cost works and we have appropriate margin for it. The cost savings aren't worth the extra "we have to monitor these servers" thoughts. Our approach is 100% emotional.

Whatever works for you. :-)

The offer is open if you ever change your mind (e.g. if the cost becomes annoyingly large when you scale up further).

ApiGateway -> kinesis stream -> lambda consumes batch of N page views. You keep the serverless approach you like, and it will cost you a fraction of what you currently spend.

I saw Kinesis stream today with DynamoDB and I will have to review it :)

I would say this goes beyond emotional. You only have finite resources. You have chosen to spend some money to avoid spending time and effort. That's a very reasonable call. Businesses do this all the time when they hire more staff.

Hey, in the most non-antagonistic way possible -- do you know who "cperciva" (Colin "Did you win a Putnam? Yes, I did." Percival) is?

You might want to take them up on the offer, you don't get an opportunity like that everyday.

Absolute legend.

"AWS Hero" is probably more relevant than "Putnam Fellow" here. ;-)

But seriously, it's clear making this more efficient isn't a priority for them, and I completely understand and respect that. I ignore plenty of good advice for the same reason: "Working on more important stuff in Tarsnap right now".

Haha I didn't know who he was. But the fact Alex Debrie follows him is certification enough for me. And I was wrong. Looking at his Twitter, it looks like he knows 2000x more than me about servers.

If you want more blog posts for Hacker News, take him up on the offer and then blog about it: "Cperciva replaced our millions of Lambda invocations with one m6g.medium and now I'm stuck as a FreeBSD admin!"

If you were charging, how much would you charge to set up a server that you literally never have to bounce to get it back up?

That's the wrong approach -- you're going to want to be able to reboot (or replace) servers for security patches if nothing else.

But if you're regularly rebooting servers because they stop responding, something has gone very wrong.

Respect your work and the kind offer you made on this thread, and I broadly agree with your suggestion, but I was intrigued as to what the Lambda costs would be in relation to their account charges and if under normal circumstances using Lambda would be a bad fit here.

Fathom has a $34/month plan which handles up to 400k pageviews per month. 400k Lambda invocations is 16 cents, if they're doing it efficiently. Transfer pricing seems negligible also at only ~2KB a request.

I'm pleasantly surprised, actually, but of course when someone throws millions of improper requests at you every second.. I guess that's when the game changes ;-)

Reply on news.ycombinator.com