Hacker News Reader

Your setup, although appealing from an AWS ecosystem perspective, sounds like a bit expensive to me on a first reading. Of course everything depends on the specifics but Lambdas and DynamoDB are expensive at scale. I wonder how it compares cost-wise to a more traditional solution.

Yes, it's definitely expensive. But, as I stated, cost isn't much of a concern in our particular situation, at least not now. I'll also point out that the entire team is about 1/3 the size of other teams running comparable non-serverless services in production so there is a massive cost savings with respect to developer salaries. There are also multiple viable ways to incrementally migrate to more cost-effective implementations that I've detailed in other comments.

Indeed, I always get the feeling you need some sort of exit strategy to a traditional model for when your service starts lifting of. Never did the cost calculations though.

I've felt the same way. Serverless is most appealing when you're starting out and have low traffic. It enabled us at ipdata.co to have the most global infrastructure possible with the lowest latencies at an insignificant cost.

At some point I believe when we're big enough we might switch to using servers in all the regions where we currently run APIG+Lambda.

The advantage of Serverless to me seems that it already forces you in a somewhat sane design and separation of concerns so all work out into the lambda functions should be easily translatable into a different architecture.

Yes, very much this. A lot of the effort was extremely in-depth planning for scalability with respect to both unbounded traffic growth/adoption and expanding the team. In fact I coded an initial prototype in about a month that could have run quite happily on a single instance and subsequently broken out into a typical LB/Autoscaling group/DB architecture without much trouble. Now we have about 7 core microservices which are independenty scalable and deployable, many of which we anticipate handing off to dedicated teams as we expand and hire.

Very smart. Good point.

That gets to lock-in.

The stacks are super proprietary. Porting away from AWS to another serverless cloud vendor (e.g. Azure) would be a major project.

Porting to a server-ful architecture would be a full rewrite.

It really wouldn't be that bad - at least not any worse than any other migration of a massively complex project from one platform to another. We deliberately kept our implementation flexible enough to be able to move off Lambda if necessary. Our entire stack can be containerized using Docker. All database interactions are behind interfaces that allow us to swap DB implementations if needed (even to relational ones). All custom CW metric publishing is centralized in a single object that can just as easily publish somewhere else. All our APIs are defined using Swagger which is portable to lots of tooling. The worst part would be replacing IAM with whatever networking/permission model the other platform had, but even that could be approached programmatically to reduce the difficulty.

Edit: We also broke the stack into a number of independent microservices, each with their own API, DB, and dedicated CI pipeline. This would allow us to incrementally migrate chunks in parallel without disrupting the entire service.

At one point I was highly concerned with lock-in, but I'm becoming less so. in the last 7 or 8 years the companies I've worked for have exclusively used AWS, and prices have come down and stayed competitive. Giving up the platform-independence also allows you to take advantage of platform specific features, which can make development a lot faster.

I still think platform neutrality is a good goal, but I'm starting to view it like I do database neutrality. It's great in theory to be able to swap out postgres with mysql and vice versa, but you miss out on a lot of features of postgres that aren't portable. And in practice, I've never swapped out postgres for something else. Just some thoughts.

I wouldn't see it that black and white. If you do it wrong of course you will have a hard time moving away to a different vendor. But if you approach it right and make sure you have the proper abstractions in place and can switch to any other cloud provider or in-house solution without to much hassle. It's all about what cost you want to pay when. Do you want to invest upfront in all development time of frameworks and infrastructure to support your core business API or do you just want to get the MVP out and invest a little more once you established a solid user base?

If necessary, our most likely cost control measure would probably be moving from Lambda to ECS Fargate to get more fine-grained control on concurrency and warm pool size.

This is one reason we (FaunaDB) offer on-premise as well as managed cloud options. So you can run the database on any machines you want. When ease-of-use matters most, small scale apps are cheaper on cloud. With high transaction volumes, you can pre-purchase cloud capacity or run on your own iron.

>We're starting to ramp up traffic now by orders of magnitude (with many more to come) and it's soooooo awesome knowing the stack is pretty much bombproof.

This does come with additional cost. Serverless pricing doesn't scale, your costs increase linearly with your usage, and there're no discounts for bulk usage or reserved pricing.

We were recently on the receiving end of a massive HTTP GET Flood DDoS and although we did not experience any downtime as a result of it, I ended up finding out about it a few days later when billing alarms started going off.

I read in many places that you should limit your max parallel executions.

We were wary of limiting paid users. Even with lambda's max concurrent function executions limit, when the function completes in a few milliseconds, the number of invocations per second can still be high.

And Shield wasn't an option?

Waf would've required setting up cloudfront in front of our endpoints which would've increased latency.

Basic Shield is supposed to be on for all users but I don't think APIG is covered in the range of services covered by their Advanced Protection plan.

Curious - are you in an industry where you expected bad actors?

I'm not blaming you -- far from it! Just wondering how applicable your experience is to the world I normally work in.

Not particularly no.

Excellent writeup. Are you serving client web traffic (if applicable) via Lambdas? Or deploying your web infrastructure to traditional instances managed through CF?

Yes, we're serving client web traffic directly with Lambdas via API Gateway. We have no traditional instances or non-serverless components of any kind in the core stack.

So how do you store your state?

I assume DynamoDB part isn't serverless.

DynamoDB is very much serverless :) I'd highly recommend to anyone remotely interested in serverless and/or databases to read the Dynamo whitepaper to get an idea of how it works: https://www.allthingsdistributed.com/2007/10/amazons_dynamo....

I'd like to point out that the DynamoDB implementation is quite different than the Dynamo paper. They are totally different things, almost like the relationship between Java and Javascript.

DynamoDB is great, especially when it's used wisely.

DynamoDB is as serverless as Lambda. (There are servers somewhere for both, but in neither case do you operate them.)

I'll give you there are no hosts to manage, but Lambda is serverless in the sense of "stateless" and "ephemeral".

DynamoDB is still persistent storage.

It seems that every every "hosted" solution is now being dubbed "serverless". :\

Oh and of course 5+ if-statements is now "AI".

Lamdas aren't stateless or ephemeral either. Anything that occurs on code load will persist on that container (i.e., if you initialize something at the module level in Node, it will persist between calls to the same container; this can cause all kinds of weirdness if you aren't aware of it. For instance, I have seen where a dev read some data at load time, and then performing destructive operations on it as part of data transformations in the code, and then wondered why he was getting non-deterministic results back). And there's a half gig of temp space on each container you can write to as well.

While the definition of what constitutes 'serverless' is pretty ambiguous, no one includes ephemeral state as a systems requirement, else you have something useless.

DynamoDB is generally viewed as serverless because there's no management of an underlying VM, and for some definitions because it can scale out horizontally automatically, without downtime, to meet demand (as compared with RDS, or another managed database solution that can only scale vertically).

We actually rely heavily on Lambda statefulness to reduce latency - many objects/data are constructed/fetched on cold start and cached for subsequent invocations.

Having no hosts to manage is the defining feature of serverless.

When you create a DB with DynamoDB, you're just telling AWS "I need a database" and it gives you one. No need to worry about deciding how much CPU power, RAM, or storage you'll need for it.

> It seems that every every "hosted" solution is now being dubbed "serverless". :\

Got a good example of something being called "serverless" that you don't think should be? I mean, yes, DynamoDB, Lambda, etc. are all running on servers. But the idea is that you don't manage them. No packages to keep up to date. No worrying about whether or not the instance size you chose is big enough. No dealing with autoscaling to meet demand when a million reddit users hit your app.

DynamoDB, Aurora, Kinesis, etc all existed BEFORE Lambda, and noone called them "serverless" until Lambda and now everything is called that.

Meanwhile Kinesis requires you to specify the number of shards you have to use, so there is management even if they're not called "servers".

> No need to worry about deciding how much CPU power, RAM, or storage you'll need for it.

You realize you specify the RAM for lambda functions, which correlates to CPU.

And with Dynamo you specify RCUs and WCUs and you enable autoscaling which adds more...

I'm not trying to be pedantic about "The cloud being just someone else's servers". I mean that "serverless" to mean is a very explicit thing about Lambda and writing stateless code. And every existing hosted multi-tenant service shouldn't just be dubbed that.

DDB is serverless in the sense that you don't have to worry about scalability issues, but you have to worry about design issues[1] if you want to take advantage of everything DynamoDB has to offer. I find dynamo's autoscaling very problematic: It's both slow to kick-in and you can scale down 4 times in 24h, not cost-effective if you have spikes.

[1]: https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

now aws allow to decrease throughput 27 times day. https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

We got DoS'd with a few hundred million calls over a few days and the dynamodb cost was about $4.

To me, DynamoDB is not quite serverless, because users still pay for the reserved capacity units. In contrast, Lambda functions scale automatically.

Does Lambda have a good version control or CI workflow? My biggest question is how to develop severless functions with a team of developers.

You can utilize lambda traffic shifting in conjunction with CodeDeploy and Code Pipelines to do safe A/B (aka Blue/Green) deployments. It has some rough edges but overall works well. Check out these docs to get started: https://github.com/awslabs/serverless-application-model/blob...

It's also possible to use API Gateway canary deployments and dedicated "preprod" stages as well - the benefit being you can, in theory, use traffic shifting lifecycle hooks to deploy a new version completely isolated from customer traffic and test it before incrementally rolling out your main A/B deployment. I created an experimental proof-of-concept for this but haven't had the time to flesh it out for production usage, but would very much like to at some point.

tl;dr The answer to your question is yes, there is a good story around CI with Lambda using the AWS ecosystem. However Lambda alone is not something that should be relied on for long-term or definitive versioning on its own.

Edit: thinking about your question more, it seems you are assuming that individual developers will directly edit Lambda function code in the console and you want to track versions or trigger deployments based on that activity. You absolutely should not ever be editing Lambda code manually directly in the console outside of one-off experiments/prototypes that are completely unrelated to dev/test/production. Always keep your Lambda function code under source control and deploy using zips uploaded to S3 and CloudFormation (the serverless framework[1] provides good tooling for this, though we don't use it - instead we use SAM and our own internal tools).

[1] https://serverless.com/

This looks useful: https://github.com/awslabs/serverless-application-model/blob...

what is the business model around this - I mean 18 months to get "set up nicely" strikes me as hard to justify to upper management that is not invested already

Broadly speaking, it started out as a greenfield/experimental project with buy-in from senior management that is now going through the initial phase of productization and productionization. Although the AWS bill sounds expensive the whole project was effectively done by 3 developers including myself (2 backend/full stack and 1 frontend). There were also some deliberate management decisions that greatly prolonged implementation time - we could have easily shaved 8 months off that figure taking a more straightforward path.

That is still a year long, 1/4M or more speculative investment. I would not moan too loudly about mgmt interference prolonging the project - that was a rare piece of long term willingness to invest for future technical gains - rare in my experience. (But imo the only way)

well a moan bit loudly...

Yes, it was/is a rare opportunity and I'm very grateful I had/have it. (That being said, the potential upsides are in the range of tens to hundreds of millions of dollars, even with only minimal success - so there is definitely a very real business incentive to invest in the project.) The prolongment I'm referring to isn't to do with the project being put on hold or anything like that. Essentially what happened was it was decided we'd build an "alpha" version of the stack to 'validate' the value of the project even though it was plainly clear what we had to do. The alpha stack nominally was supposed to be a cheap, quick version of the real architecture (which we'd already designed) whose supposed savings were gained by substituting manual processes for some of the APIs rather than actually building them. The end result was a. confirming what we already knew in that yes, the proposed functionality is fundamentally useful (very obvious from the outset) and b. a huge diversion of time and effort doing throwaway work that was about 65% as much work as just doing it the right way would have been, with the additional burden of having to perform the manual "API" functions, operational overhead of maintaining that stack while implementing the real one, burden of having to migrate data from the alpha stack to new stack once it was ready, and work to deprecate/tear it down once it was completely out-of-use. The overall wasted time was easily 8 months.

Woah - potential upsides in 8-9 figures ?

So without digging in to who the client is etc, are you saying that serverless architecture is today, validated enough that the AWS bill can be cut by ... what 25%? more?

That sounds like great savings (plus nice fat contractor bills for rewriting as lambda) - but is it a hit for amazon

(I suspect amazon views it as "kill your own babies" survival but am interested in the margin effect on the data centre business - it's twenty years since i touched a business model of a DC)

PS I get the prolongment issue - it sounds sensible in the 20 seconds it was covered in the approval meeting, but not when the details are looked at.

This is a new project, not migration of anything existing. "Upsides" was referring to business value of the project itself, not cost savings of migrating to serverless.

Thanks