Should you use a Lambda Monolith, a.k.a. Lambdalith, for your API?
rehanvdm.com
rehanvdm.com
We figured out how to get Azure Functions to serve a complete webapp without any extra crap wrapped around it. No gateways/proxies/etc. One function that serves the root URL[0] and has conditionals for GET/POST methods. Beyond this point, anyone with old-school mastery of HTML/CSS/JS can get the rest of the job done with basic-ass SSR and multipart form posts.
For data, we just use Azure SQL Server. Authentication is like sticking legos together now. 100% managed identities, no magic strings, etc. Our functions get the user's claims principal right in the arg list. We have zero lines of authentication code in our latest products.
This stuff is absolutely the future for a vast majority of boring old business.
One thing to think about: If your FaaS offering is capable of serving content-type of application/json, is it not also capable of serving content-type of text/html? Do your providers go out of their way to prevent you from making a simple solve on 99% of B2B bullshit?
[0]: Note that serving the root URL on an Azure Function is an undocumented hack-around item. Worst case, we would serve the /app route and our users (B2B) wouldn't really care. My suspicion is that Microsoft detected our use case and saw that it stole a LOT of thunder from a wide array of other Azure offerings (API Gateway, Front Door, et. al.), so making it slightly more complicated (at first glance) is likely intentional.
Our largest assets are SVG logos that our clients bring to the party. We do work with dynamic PDF documents, but these are uncachable for a number of reasons.
Hypothetically, if we had millions of users and static asset IO was actually causing trouble, we'd consider using a CDN or other relevant technology.
We've got some experience with on-prem IT shops that have taken way more out of us than we probably should have allowed. Every one of their issues with getting our software to run in prod was unhappy in its own special way.
Who monitors it 24/7 and checks the alerts? Who drives to the data center to fix broken hardware on Christmas Day? It has to be multiple "whos" or one sick leave will leave you completely vulnerable.
For small-medium companies whose main business isn't hosting software this makes it untenable from a financial point of view. For the monthly salary of a single SRE/devops/admin whatever you can run AWS services for a year at that scale.
They have been remote-first (remote-only?) since forever, they still use Ruby in production, have never taken VC money and they've never even tried to do a hockey stick growth unicorn hack. They're still under 40 people.
The devops time you need to provide to manage most cloud setups is generally far higher than what it'd cost you to get someone to manage your managed servers both in time and retainers to have someone to escalate to. When I was consulting, cloud focused customers were gold dust because they needed so much more help and had no notion of cost control.
The idea that cloud is cheap is pure fiction from people who haven't run and costed both.
Put another way: The notion that anyone needs to drive to the data centre reflects a very outdated idea of both.the alternatives and the support options available if you actually choose to own the servers rather than rent managed servers.
I used to manage a setup spread over multiple data centers, and the only visits to the data centre were scheduled trips 3-4 times a year. We regularly priced out cloud alternatives, and every time it'd have bankrupted the company if we'd switched.
I love using cloud when working places with margins enough to afford it, but it's an expensive luxury and the idea it's a cost saving really needs to die, because it takes a really large setup where you have power to demand crazy discounts, and a lot of engineering, to even get close to break even against a moderately competently set up managed setup.
So, I've got 6 different backend services running as monolithic Lambda functions. They all must be up for a number of nines per year, each going down will cost our company actual money for every second of downtime. They've never been down except when AWS broke something - and then the customers couldn't have accessed them anyway.
Can you tell me how I can host 6 services that can't be on the same machine in a managed system cheaper than my current setup? 6 servers, preferably each duplicated.
We'll assume that all the other AWS services we use can "just" be replicated in managed hosting too. (DynamoDB, S3, Aurora and a bunch of others).
I've yet to come across a setup where we couldn't trivially halve costs. Often you can halve costs over a well optimized AWS setup (most AWS setups are nowhere near well optimized, and most AWS customers haven't even caught on to the ability to start pushing for discounts once their bills are high enough; you can still beat that with more traditional setups) even while just migrating to cheaper cloud services elsewhere. Often the cheapest is a hybrid, because it allows you to benefit from the resiliency of cloud providers while only rarely having to put any load that costs on them.
I get that you've left out a lot of details, but what you describe is not a complex setup. Try thousands of VMs over multiple countries with dozens of database and everything replicated at minimum in three locations, and with zero downtime migration of loads between countries on a hybrid setup across multiple providers, several colo facilities and managed hosting providers.
Even with the fully loaded devops cost a setup like that came out at less than just the egress bandwidth would've cost us at AWS.
There are plenty of loads where cloud providers are the right choices, but they are almost never a cheap choice, and when they are, AWS and tge other big ones are rarely even breaking the top 5 cheapest cloud options.
Security, correctness, compliance, performance and availability are gonna be far more important to the business. As long as the cost isn't absolutely insane no one is going to blink. Zero stakeholders care that we can do it marginally cheaper with a nerd sniper rifle.
At the same time, many other places I've worked or done consulting that is not a choice their businesses can survive. One place the cost difference between cloud and managed servers was large enough it'd have been several times their profit.
We're not talking "marginally cheaper". It's rare to be talking about cost savings lover than 50%. That still doesn't mean it's always worth switching away from cloud environments - if that 50% saving is small enough to you to not matter, that's totally valid.
But you ought to understand that you're making that choice because the costs are currently too insignificant to be worth it, not because it's the cheapest option.
The argument is not that you should never pick cloud, but that claiming to choose cloud because the alternatives are "untenable from a financial point of view" to quote part of the comment that I first replied to is totally backwards and suggests to me nobody has actually done a cost analysis.
Services like Vercel just wrap the Lambda and indeed deploy an entire server to the Lambda and gained a lot of users this way. Lambda Monolith are good replacement to the old Heroky dynos. Lambda monolith are super cheap for B2B SaaS startups with little traffic
I'll extend this to all three cloud services, and dispute the assertion. Developers are smart and figured out that these serverless platforms are great for APIs even if the initial tutorials didn't present that scenario. Also, the platforms just got better at this over time while those early tutorials were somewhat cast in stone. I've not looked lately, but I would hope that they now have tutorials that show monolith scenarios.
>Lambda monolith are super cheap for B2B SaaS startups
Again, I'll extend this to Azure Functions and Cloud Run. Truly an amazing place we now sit. My first public facing API required me to build a physical machine and place it in a colo facility in town. Now I just run a command-line or check in my code and my API is live.
A big reason is that lambdas were half-baked and gradually became better. They are still limited.
Sending a binary blob? You have to convert it to base64.
Streaming a response? You can only do that on the nodejs runtime.
Establishing a short-term ws connection? Hard/impossible.
The model of "every function should be a separate Lambda" is just moronic - I've seen people run into hard limits on AWS going all-in on this model.
Instead Build a single executable/library and deploy it as a Lambda function.
Now you have automatic versioning, just push and it's a new version. You can label versions and call them based on labels.
Deploying to prod is just swapping the prod label to a new function. Want to roll back? Swap it back.
Credentials: ran a top10 mobile game backend on AWS Lambda with zero issues using a C# monolith.
Oh, you say you have more than 200 endpoints in your REST API gateway, and suddenly your CloudFormation stop working? Yes, unfortunately CF only load 8 pages of resources in alphabetical order when starting, and your root resource starts with a ‘Y’.
No problem, implement a new custom resource that just creates and discards API gateways until you have one with your desired root resource starting with ‘A-F’. Did I mention the ‘delete a rest api’ API is limited to a single call per minute? The fun doesn’t end. Now you may be able to grow until 600 resources before you’ll have to tighten the limits once again.
You say you think this is bollocks? Of course, just switch to an Application Load Balancer. Just 100 rules max means you’ll only need to nest about 4 of them to fit all your API endpoints.
And people wonder why some give up on the cloud. Any $5 VPS has higher limits than any of these AWS services…
That must've been a proper monolith then? =)
If you start your url like /api/v1/service/endpoint, you immediately have 4 resources.
But yeah, it’s a proper monolith.
Because it doesn't have to be this way, there's php-fpm and such that could eliminate the http->http step
Unless you're referring to the 'load balancer' in the comments where someone builds their own multiprocessing by launching multiple servers and putting nginx in front of it. I mean do what you need to do but at least be aware you're hacking together something that should be a basic feature of the http server.
If you wish to push me I'd be forced to admit that I don't like load balancers either, but that's because they're doing routing on the application layer which is not ideal and creates a single point of failure. That ship has sailed though.
But we also used it internally for batch processes.
In one case latency was important so we hosted the app in an auto scaling ECS Fargate cluster behind Nginx and in the other case throughput was important so we could just deploy the same API to Lambda.
Yes I know about reserved concurrency.
The point of doing something like this "Lambda Monolith" is that transitioning to an always-on server is trivial. I've been doing this exact setup for a while and our static costs basically evaporated. Seeing your costs drop below a dollar/month on a project you are building is amazing. As soon as one of our projects is large enough, we move it to a dedicated instance or cluster. This allows us to delay that point until we have enough revenue to justify the costs.
If you have one or more professional developers working on a project how can you be bothered about an extra $20-$200/month when you know you'll also be spending a couple of engineering hours (worth more than $200) on migrating later?
If you’re in territory where running locally isn’t feasible you’ve either done something horribly wrong or have scaled way past the point where either a single lambda or single ec2 instance is sufficient.
> Seeing your costs drop below a dollar/month on a project you are building is amazing.
Honestly, this just means you don't have any users
Which is where many small ideas start.
My buddy had an idea for a few SaaS products and I've helped him develop them. The initial costs were small, but give it was an expense with no income in the near future, finding a way to scale to zero was the only way this became something he'd consider. It's got _very_ sporadic use currently and his bill for each service is literally around $1/month right now. Even the cheapest instance would be more than he's paying.
All system architecture is about designing a system to meet the defined needs. This system is designed to hit the starting needs and has an already designed migration to the medium-term needs. That idea of "YMMV" is very much relevant here as every system has different needs. Some of those needs are minorly different, others are wildly different.
Anyone on this thread that's discounting FaaS usage out of hand with statements like "just run a $20/month server" are demonstrating a lack of imagination. Would I use it as the primary server for a medium+ traffic site, almost certainly not. But there is a whole world of low-traffic services that are well suited to using a FaaS platform over a dedicated server.
With a Lambda function you can have a dozen different versions of the same code running simultaneously with zero extra cost and none of them will affect each other's performance. Every one of them will be versioned and you can instantly roll back to any version or pick any version to be "production".
And it also depends whether the database is MSSQL ($2k/month on AWS) or something like Aurora.
In my experience, running a single monolith server will be much simpler than 20+ lambda "monoliths" that call each other. I think the simplicity of lambdas vs a persistent server looks good on paper but falls apart when you have multiple times more deployments to manage.
Source: I've rand both a Lambda Monolith and ECS on production and would pick Lambda any day
My bills are around $0.02 a month from API Gateway =)
AWS Lambda NEVER goes down. And if it does, it's most likely someone messing with DNS or backbone routing again and half the world is broken anyway.
It does need skill though, the billing model isn't as clear as "you pay me X euros a month for a server, I don't care what you do with it", but it's not dark magic either.
yes we have some very spiky workloads and we got bitten by that, subsequently we moved away from ECS for this exact reason
Another couple of years and they will reinvent microservices on top of that monolith thingy.
Small functions that can be reused in multiple APIs, tiny deployments, small tests incrementally building a system.
Lambdas primarily shine in an event based system. I'd rather use the serverless tooling as they're intended i.e cognito, gateway, event bridge, dynamodb.
Each are designed to solve a particular problem in what I'd say as a different paradigm.
If you want a traditional monolith I'd look to fargate instead since you'll be rolling a permission system/auth system and probably a framework of sorts.
In my past using only a part of lambda eventually lead me to having to build an already solved problem with a serverless service and make it fit in. Lessons learned I suppose.
This worked for us because it mitigated the shortcomings of the other two approaches for example:
- lambdalith: very long build times and long deploy times (due to the high number of lambdas)
- one lambda per route: lack of permission granularity (as opposed to creating different IAM roles for different lambdas); lack of granularity for memory/cores configuration means you are missing out on cost optimisation; also, as the API grows, it soon becomes necessary to adopt some framework like Express or Fastify to keep things tidy.
EDIT: reworded last paragraph for clarity.
Microservices in various forms are unnecessarily distributed and that's a fundamental problem with many implications. When you pack your components into a monolith which can be easily reconfigured into microservices by replacing transport implementations, you would gain a lot.
Just run nodejs.
The cloud has made otherwise smart people into unthinking drones.
You don’t need any of the cloud stuff, except a Linux virtual machine. Stop drinking the cloud kool ade, it’s making things much more complex and expensive for questionable gain.
Learn how to, you know, run software on a Linux computer.
By flat out refusing to consider it, you become as much of a "drone" as the people you call out.
Such as?
If it’s regional latency then run a machine in the region.
That's all. The rest goes in VMs.
When people go home, the dev Lambda cost instantly becomes zero.
You can't do that with (virtual) servers, the cost is exactly the same 24/7. In some cases it's a good thing, some not.
You can't also trivially handle Hug Of Death level load spikes.
Not to mention Lambda uptime vs everything else.
I use Cloudflare Workers on hobby projects to provide server side work for things that have low volume, because the price is so low per user.
It's true that Lambdas will become pricier with consistent high volume usage, but for extremely variable usage or low usage, lambdas are a great option.
I am pretty sure this is way more expansive to spend engineering time making it possible for server(s) to scale using kubernetes and stuff, than just using serverless.
And later with many users, we estimate what better be implemented manually to save money, because at this point it would actually save us money to employ a new engineer than continue paying serverless due to number of users.
Don't you think serverless makes sense?
Do you really feel that serverless is somehow faster than a reasonably modern computer?
Nodejs on a single machine can manage tens of thousands of connections if not hundreds of thousands.
The vast, vast majority of sites would never be able to consume 50% of the computing power in a modern gaming OC.
I would still do it in AWS, as it is like golden garden, don't need to deep dive into security measures, to manage authorizations manually, to handle database sharding and optimizations, you get notifications systems, ML, data streams and everything just connects together.
From there using lambda is way faster than to develop your own api with caching and making sure it is 100% optimized so that 10 thousands ppl can connect, while in fact you might change it tmrw.
For example you can write your portable app in Node.js (with or without containerization) and push it to a "serverless" PaaS, without running your own linux machine or worrying about k8s.
e.g Fly.io, Render, Railway...
If each of these services was 1 server, and each server cost around $20, we'd have to spend $600/month. I know for a fact that there are times where each service request can consume 80% of the memory. For safety we'd want at least twice the memory or number of units, which are functionally doubling the cost. That $600 can easily balloon to 12 or 2400 per month if I were to host the services. Now add another $600 per month for the test environment. So we'd be looking at $1200 - $3000 a month if we kept things really lean.
Now here's the lambda side. Most are about $3 per month (KMS is the lion's share of the cost) for the test & prod. So $90 in service hosting. Yearly lambda is $1080. Yearly self hosting/EC2/droplets is $14,000-36,000. Now repeat this pattern across 5 teams with similar approaches.
The savings add up for us. Now is this a universally true statement? Nope. But there are valid reasons.
Fun fact, if we ever needed to move off of lambdas, we can because NestJS sits on Express. As long as we can access the datastores there is minimal work to move.
> The cloud has made otherwise smart people into unthinking drones.
The cloud has allowed smart people to focus their time on unsolved problems, not waste hours/days setting up linux for the upteenth time.
I’ve worked at big companies with smart people who burn days and weeks trying to get IAM, gateways, vpcs, firewalls and lambda to play together. Let alone the ongoing nightmare of ops/dev interaction.
Complete cloud fiction.
The worst problem is the giant pile of cloud spaghetti you end up with and no one has any idea what connects to what and what depends on what. Easier to just accumulate more and more resources which cloud companies love.
Just run a computer, it’s easier.
> Just run a computer, it’s easier.
Statements made confidently while also being totally untrue.
> I’ve worked at big companies with smart people who burn days and weeks trying to get IAM, gateways, vpcs, firewalls and lambda to play together
Working with incompetent people is not an excuse.
At the end of the day you still need to configure the instances for things like auto scaling, security patches, logging and so on. IAM & VPC still come into the mix when running on EC2, so you've avoided nothing.
You haven't avoided much of you replace lambdas with ec2. You do if you replace it with a rented managed server or even a colo.
We're arguing opinions and trying to apply logic.
Some people find lambda easier and it must be true that lambda fits certain workloads better. Some people prefer VMs or on-prem or other long-running services. I prefer both in different cases.
> The worst problem is the giant pile of cloud spaghetti you end up with and no one has any idea what connects to what and what depends on what.
Yes, it takes discipline to use the best tool for the job. "You should do X for everyting" is not the right approach, however. This argument is moot.
Right now I support:
* Lambdas for some very expensive infrequent number crunching
* Lambda-like on edge for fast response services that require low latency
* VMs for always-on services
* Computer in a closet for backups, logging, metrics, etc.
Did they use the CDK or even SAM?
Both suck real bad.
Infrastrucure is hard, thankless work. Complexity blows up whatever you do.
Edit: I just did it with ChatGPT 4 expecting it to just create the CDK app. It actually created inline Node sample code as part of the construct for the actual lambda to read from the database.
The last time I did that as a sample to show other developers I still had a little additional prompting to do
table.grantReadWrite(lambda function)
ChatGPT is well trained on everything AWS related. It can transform CloudFormation to idiomatically correct CDK or Terraform.
I hate to say this because it sounds like an appeal to authority. But I really want to set context. I used ChatGPT for projects while I was working at AWS ProServe and since I left. They were generic utility scripts with no proprietary business related code.
Cost in time to setup a bare metal, production quality k8s cluster with all the bells and whistles from the cloud: 2 weeks and a skill I will have forever. In fact, the second cluster took 4 hours.
Monthly cost: $240 per month.
Time to spin up a new worker, using cloud compute for elastic load beyond base load: 5 minutes.
Base load capabilities: 120ish cores, 1tb ram, ridiculous terabytes of storage.
Cost to run on managed k8s in the cloud: $5-20k per month.
It's my company so I think I'd know :)
Also what exactly are your estimates here? Where are you getting that kind of capacity for $240 a month?
I had to run a authenticated webhook forwarder outside of the out firewall. Yes, I could have made a real machine, but I've made a lambda instead. It's costing us less than $2/month and I spend zero energy on maintenance, security checks, etc. And all config is in a single git repo that anyone can read and understand - there is zero chance of someone ssh'ing into server for a quick fix, then forgetting to record what they did.
Real machines are great for heavy loads, but you just cannot beat lambdas for the lightweight stuff.
I hate managing a server, unless it's running a DB, relay, or other long-running service.
For most my backend work, it's "Please calculate this", and that fits so well into a lambda I don't understand why people hate it so much.
Zero maintenance, zero costs until used, much less security to manage, etc. It all works very well for me.
I still run all kinds of stuff on a linux server, but the comment seems a little gatekeepy. There's nothing strictly better (or even simpler) about running on a server.
> Learn how to, you know, run software on a Linux computer.
I know how to, I run NixOS at home. If you own the VM, you own the maintenance on the VM. Optimizing for the cost of setup is usually wrong, most people are better off optimizing for running costs and maintenance.
Main benefit I see to Linux machine is not having to deal with deprecated run times. With lambda every few years, I have to redeploy my Lambdas when their runtime gets deprecated.
> Now the big one, reusing code is by far easier in a monolith.
> Consider the scenario for patching an error in a library that is used in all your single-purpose functions:
> You have to publish shared code into packages that need to be used by all of your routes > You would have to context switch, make the change in the library, merge, publish the package > Then context switch again and update the version in each of your 100’s of routes to use the new version > Then you need to test and deploy all the 100’s of routes
I don't use a monolith mainly due to size. I had enough problems packaging all my functions in 1 archive and hitting size limits that I can't imagine a monolith would do anything but lock me into that issue with no solution other than splitting it up into smaller services. Currently I package all my functions individually so they only contain the code/deps that that specific function needs. This works very well and keeps my function sizes small (except for the Prisma engine, stay away from this if you can). Also I have shared code that multiple function use (wrapping code for catching errors, DB access, various other reusable services), it all works just fine and I don't package/publish my shared code, my packaging grabs what each function needs and includes it.
Everything else he says, especially about getter /more/ warm starts and being able to really take advantage of PC is true, but the size issue steered me away from a monolith. If size wasn't a problem I'd probably be running express in my lambda but instead I have a 1 function per endpoint and that seems to work pretty well for my company.
I've had one tell me that don't try to use this service, it won't work on your use case and if you try to force it, it'll cost you too much to be viable.
Another example could be permissions granularity (each lambda can have its own IAM role). For example one of your lambdas could have access to something in Secrets Manager.
There are also disadvantages to this approach, for example build times and deployment times.
There's nothing stopping someone from taking what we have and porting to an EC2, but for most use cases, this pattern seems to work best.
We break our Lambdas up by how they are used, to keep sizes down, with the option to break up further if we need to.
By the way, routes with ids are handled no problem by API Gateway, you can even set more complex ones like {proxy+} etc.
> Also with lambdalith you probably end up needing an application framework as opposed to just using mostly vanilla features of your language of choice.
The point I make is even with lambda per endpoint you need many framework features.
I create separate lambdas only when I have some large dependency that can be easily be separated from the rest of the application.
A few that come to mind:
1) AWS services are not easy to develop with locally. This incentivizes a lot of testing in production, which in turn can slow down development speed and increase defects. Yes there are some wrapper libraries and helpers, but it's not 1:1. (This same thing applies to Cloudflare workers as well).
2) Lambdas have constraints such as execution time limits, and a lack of fine grained control over resource usage. This makes it hard to use things like long lived requests and web sockets. But it can also mean you could end up paying a lot more compared to a fixed price setup, such as an instance in EC2. The flip side is also true: it can be an incredibly good deal as well.
3) There's a lot of tooling and knowledge around Lambda which takes some time to learn. You have to be quite careful around things like DB connections, logging, concurrency, and so on. This overhead might ultimately make things more complicated than a more traditional setup.
4) There is a maximum package size for a lambda. When you include all your dependencies in NodeJS and so on, it can become very tricky to keep things under this size in even a medium sized code base. A larger size also effects your cold start time. There isn't a lot you can do to change this.
5) Lambda's easy integration with other AWS services is both a blessing and a curse. AWS is not cheap, and given how trivial it can be to add on services, it can require a considerable amount of time to reduce spend. Lambda's integration with non-AWS services is very hit or miss (e.g. Kafka). You only find this out by getting very burned.
6) If you require anything like C libraries, be prepared to spend a lot of time working on builds and deployment - or sometimes it just magically works. You are operating inside a somewhat opaque environment, and it can require a lot of fiddling. Additionally, using tools like CloudFormation do not scale very far, and can sometimes result in quite bad outcomes. This stuff can steal a lot of time.
7) There are hidden costs everywhere. For example: Are you being a good engineer and adding a lot of visibility by logging and using tools such as CloudWatch? Be prepared to pay an extraordinary amount relative to the value you get.
8) Using lambda as a processing pipeline? Easy to get started, hard to get reliable. AWS does not have good inexpensive tools to manage these well, and you can end up with a very leaky and expensive system.
9) You would be surprised at how many bugs and weird behaviors exist when these AWS systems play together. In some cases, the only recourse was to delete the entire stack and redeploy because something got stuck. That could be quite bad depending on the use case.
10) Using lambdas over data engineering tools (e.g. Kinesis Data Streams vs Spark) might actually be more complicated. You should hopefully be familiar with and have some experience in this these tools before making a decision when its time to build.