Serverless to monolith – Should serverless lovers be worried?
beabetterdev.com
beabetterdev.com
Developers are losing touch on what process boundaries mean, and the result is brittle and poor performing systems.
If every time you add a new feature to your application you also have to add new infrastructure, I feel very, very strongly that you're doing it wrong. Sometimes? Sure. Every time? Embarrassing.
Cloud providers are surely pleased as punch that there is now an entire generation of developers that will argue vehemently that infrastructure that only grows over time is not only healthy, but the best way to build scalable software.
I have no idea what to call it other than "Marketing Material Driven Development".
And on the flipside - One of the funniest justifications I have heard for companies using new fangled untried tech is to attract talent - no one wants to work on boring tech apparently for the fear of being outdated.
I sometimes worry for the future of our industry because of exactly this. The culture of buzzword driven development has become enshrined in the recruitment process - often from both sides - because of this kind of vicious circle effect. Now anyone - again from either side - who tries to escape it and instead advocates using sensible, robust tech to build useful, good quality products risks becoming a pariah and seriously damaging their career as a candidate or their ability to recruit strong candidates as an employer. Of course this assumes we are judging career progress by level of seniority reached or compensation paid instead of more indirect achievements like producing something that provided real value to those who used it. But if we're being honest most people obviously will.
In one interview the engineer interviewing me explained how they didn’t have any paying customers, and then went on to describe an elaborate over-engineered micro-service based architecture he was working on.
The real kicker was that instead of asking me about my 20 years building products, or our infrastructure at my last gig that handled millions of customers across the globe, he instead wanted to watch me code some goofy coding quiz he had prepared.
It's horseshit but at least half of the people in charge of hiring think exactly that way and so I end up leaving the work I'm the most proud of off my resume a lot of the time.
I could see that. Lambda has a 15 minute timeout, and if you’re processing something large you want an orchestrator that can manage retries. I don’t like the SFN developer experience, but it’s only like 15% worse than boto3.
Cloud providers don’t care about your serverless function utilization for 95% of people. A huge chunk of people will never pay more than a single digit serverless bill, regardless of if they have 450 lambdas or 40. If creating a new function matches your mental model of how things works, simplifies ops and log reading, or gives you access to a library in a different language, I’d say you made a great call.
Like many of the problems (security patching, monitoring, logging) are part of the serverless pitch.
I have to call out your take because it's either outright wrong or is overgeneralizing, to put it politely.
A lambda is just someone else's computer running your function, which you call through a fancy rpc interface. There is no infrastructure.
Just because you can and should deploy a lambda with infrastructure-as-code tools such as CloudFormation or CDK, it still does not make a lambda "infrastructure". It's just your function running on space CPU cycles of someone else's computer, which you call through RPC.
Running your lambda as a background task in someone else's computer does not make it infrastructure, just like running code in your own server is not adding infrastructure.
Also, it seems you're missing the whole point of offloading background processes out of your servers. If they are computationally demanding and your servers are well provisioned them a background task can lead your system to scale up. You don't change the way your system is deployed or is operating if you run your background task as a lambda. You make a request and things run just the same. Zero potential for alarms and being paged in the middle of the night.
And more importantly, for some usecases lambdas are entirely free. Does this have no value to you?
Some people feel compelled to criticize everyone around them just to feel self important. A little humility would serve them well. Just because you fail to understand something's traits and properties that does not mean everyone else around you is wrong and you are the only enlightened soul who is right.
My personal take here is just that serverless functions have a time and a place in one's infrastructure, and it requires good engineering to determine where that time or place is or isn't. We use Cloudflare Workers fairly heavily, for example, to perform web redirects for some statically generated sites. It's a fairly sensical use-case for "serverless" in my opinion.
Implementation details don't really matter. The CloudFormation/CDK deployment of a lambda can include only the source code of the task you want to run. We don't know if AWS uses containers, an intern pressing enter, or purple unicorns.
> Therefore a Lambda is some kind of ephemeral invocation-based infrastructure.
Not really. A lambda is your function running somewhere. Unless you explicitly create your lambdas as runnable containers instead of simply passing your handler's code, you don't know or care how are things under the hood.
> My personal take here is just that serverless functions have a time and a place in one's infrastructure, and it requires good engineering to determine where that time or place is or isn't.
There is no magic or arcane wizardry involved. If you need to peel a background task out of a server that doesn't run that often and requires no internal data other than input parameters and takes far less than 15min to run, and you're already on AWS, then lambdas are far easier to implement and to run and to operate than modifying some server deployment to include support for a background task. In some cases, they can also be free to run.
There's also the usecase for plugging together AWS events, such as processing a file that's uploaded to a S3 bucket.
If you are already neck deep in AWS, lambdas are not an arcane trick that only fit weird corner cases. You need to know what's the free tier, lambda pricing scheme, and do back of the napkin math to figure out where it makes sense to you to switch from lambdas to a dedicated service.
I'd argue the intern pressing enter at AWS is my infrastructure.
> Not really. A lambda is your function running somewhere. Unless you explicitly create your lambdas as runnable containers instead of simply passing your handler's code, you don't know or care how are things under the hood.
I feel like you took my pragmatic explanation and obfuscated it.
> There is no magic or arcane wizardry involved.
Never said there was, I think you think that I'm more confused than I am. I run my own serverless platform, and make use of serverless platforms from other companies. I know how they work.
> then lambdas are far easier to implement and to run and to operate than modifying some server deployment to include support for a background task. In some cases, they can also be free to run.
You're trying to sell me on lambdas again. I said I already make use of serverless. I just am saying there is a time and place for them, and a time where they don't make any sense. You seem to be agreeing with me, where you placed a few qualifiers on when you should use a Lambda, so I don't know why you're framing this like we're in disagreement. Maybe you're confused as to what I actually said?
we used to call those "cgi scripts"
That’s the part that’s being critiqued.
Sometimes it makes sense. I see people criticizing stuff they know nothing about.
I worked on a project where I had to put together a REST API for an internal project. After considering the traffic projections, we noticed a API Gateway + lambdas fitted quite neatly in the AWS free tier, and we did everything in lambdas. We also did a proof of concept in ECS and everything was far more complex and convoluted. With lambdas, the whole project consisted of a single CDK package which included also the nodejs code for the handlers.
In another project, we even skipped the API gateway and instead had a NLB invoke a couple of lambdas. It worked flawlessly, and in the free tier.
I'm seeing comments about the need to do engineering, but these same comments are outright clueless and fail to do the faintest cursory research. It's just people commenting on stuff they know nothing about with a demeaning tone. Not cool, not smart.
IMO the less you need to code on top of your actual business logic functions - the better off you are for small projects.
Docker is still one of the best options, as if your tiny app is somehow business critical you can easily migrate it to other infra. With lambdas you’re stuck on AWS.
You're not getting stuck on AWS because you added a lambda to offload a background task, or because you implemented a low-traffic REST API with API Gateway. If you're rethinking where you deploy services, your decision won't factor in these irrelevant details. Odds are you are more concerned with S3 and CloudWatch than REST APIs.
“someone else’s computer” _is_ infrastructure. _everything_ you run in cloud runs on someone else’s computer. Besides, you need to specify the amount of cpu and ram you need for your lambda, the kind of scaling you want, whether you can live with a cold start or not, optimise that cold start, potentially configure networking and so on. You can claim all you want, in the end it is an extra piece of infrastructure that you have to be aware of and configure.
> You don't change the way your system is deployed or is operating if you run your background task as a lambda.
Seriously? You do not need some kind of authentication to call your lambda? The lambda does not need access to some kind of private data to store its results? You won’t introduce a queue or something to offload background tasks?
I think it is naive to state that introducing lambdas into the architecture of an application does not have an impact on the infrastructure you need for it. And it makes me honestly wonder how much experience you have with lambdas or functions on production systems.
That's meaningless pedantry. You don't talk nor care about infrastructure when you're adding a controller to one of your services, or when a container scales up.
The only thing that matters are the relevant attributes like performance and operational costs. You don't care if running a task in the background ends up scaling up your service, just like you don't care about that when running the same task in a function-as-a-service service like AWS Lambda. You only care if your system is stable and the invoice you get at the end of the month.
If your mental model cares about meaningless low-level details that do not factor in your operations, your mental model is brokenand you have no business lecturing others on how they are doing things wrong.
> Seriously? You do not need some kind of authentication to call your lambda?
You don't, if you configured them right.
> The lambda does not need access to some kind of private data to store its results?
What? It does not have any extra requirements than calling them from a service running on EC2, ECS, Fargate, EKS, whatever. It's an internal AWS service that, just like any AWS service, is covered by AWS' access management. In fact, arguably it has less requirements, specially considering least-privilege policies.
> I think it is naive to state that introducing lambdas into the architecture of an application does not have an impact on the infrastructure you need for it.
I argue that talking about infrastructure when talking about function-as-a-service is ignorant and naive, and reflects lack of insight and first-hand experience which materializes in a mental model that's fundamentally broken.
And people suffering from this problem should not be making bold statements about something they know nothing about.
If it’s a horrific monstrosity it’s either because you don’t understand it or it was poorly done.
Seen several .NET step function implementations which were clean and easy to understand. And we’re cheaper than what they replaced. And new devs could easily pick it up and understand it.
This is typically true with a lot of ClickOps teams.
Externally mediated taste is how a lot of people operate. And they may have a vague idea that they should not defer as much to it, but they don’t have the experience to take the risk. So they double down on consensus.
I feel like a lot of the problems are from misused Agile. Devs are building systems that haven't fully been thought out. I don't know when I've seen a fully built out process map in the past 5 years.
MVP1 is easily built as severless, but then what happens when the full implementation gets more complicated? Do we rewrite all the server less architecture as a monolith? Nah, just tack on some more Lambdas. No way can we slow down for a refactor, and all the original devs that wrote it have job hopped.
The most likely way to escape the current madness might be for a few companies with more enlightened management to actively look for people who have stayed in past roles for long enough to have real experience not just building but also maintaining software and pay them more. Same with loyalty incentives to retain existing staff. And then make sure those more experienced developers are choosing tools built to do a good job and not just tools built to make naive resume scanners happy.
If frequent job-hopping and leaving any junk you've just built to become someone else's problem stops being the fastest way to boost your career and compensation then good people and good employers will start to prioritise building stuff properly and looking after it for the long term and almost everyone will be better off.
If you’ve never worked anywhere long enough to have to live with the fallout and consequences of your decisions, you’re not senior in my mind. You’ve never received useful real world feedback to be given a chance to refine your approach.
While I do think lambda is sometimes overused, but no good engineer says "cuz serverless" for using lambda function as is implied in all these conversation. And there is no best choice in the absolute sense, and I wouldn't want to hire someone who claims that monolith is the better architecture for everything or everything should be serverless.
The issue is using them poorly, and in particular stringing them together into a sort of pipeline.
On the other hand, I've decreased costs significantly in previous positions by moving the cart functions to a lambda because it made it highly responsive, scaled up to thousands of transactions a second without concern during initial sales of certain fruit based devices which generate a rush, but basically sit idle 90% of the rest of the time.
I am not concerned with "the death of serverless", there is a time for lambda, and a time for long running processes. You can stick multiple rest endpoints in a single lambda, and that's the recommended thing to do. Moving from micro to monolith isn't the death of serverless, people not knowing when the appropriate time to use different techniques will be.
Exactly, people should take the time to study different techniques, learn about their advantages and disadvantages, and then use some basic common sense when choosing a specific technique in a specific context.
Thanks for re-sharing. I was quite disappointed to see the negative community reaction to the original Amazon article. I felt that those knocking it didn't analyze the circumstances that lead to the Serverless Architecture's gaps and just saw some problems with it and assumed "serverless bad".
In reality, the series of decisions made a lot of sense. Take something that exists, try using it, learn from it, and make better decisions as a result of it. It just so happened that in this case, serverless architecture was the wrong tool for the job and dedicated machines were a better fit.
Anyways, happy to see my take being shared. If you enjoy this kind of content, I have a newsletter and Youtube channel where I discuss / make videos about more AWS & cloud concepts:
Newsletter: https://mailinglist.beabetterdev.com Youtube: https://www.youtube.com/c/BeABetterDev
Cheers,
Daniel
I kind of want to take a Crack at what Prime Video did except serverless from a green field perspective, and I'm willing to bet I wind up provisioning a RedPanda cluster or similar and having them feed detector lambdas. The bottleneck and cost here appeared to be storing intermediate results and the orchestration. Orchestration is solved by using something like SQS or what have you. Intermediate results are somewhat harder depending on size - but I think you could get there and it'd be an interesting exercise.
About just how much money their ideas have thrown at $CLOUD that could've been used for things other than helping build Bezos' Freudian Me-Too Ego Trip What Go Zoom.
Or how much fragility and painfully slow development cycles have been inflicted on people in their org by the inevitable tight coupling to $SPECIFIC_PRODUCT_IN_SPECIFIC_REGION_OF_SPECIFIC_PROVIDER.
And, they should be worried that they've drunk so deeply of the Kool Aid provided at $VENDOR_CONFERENCE_PROBABLY_IN_VEGAS that they considered the cost and the friction a worthwhile trade-off.
>the original architecture was based on a tool that was not designed for scale. Based on the article, it read as if this tool was used for diagnostics to perform ad-hoc assessments of stream quality. This means it likely wasn’t designed for scale or put through the pressure tests of a formal design document / review. If that were to happen, any run of the mill Software Engineer would have been able to recognize the obvious scaling / cost bottlenecks that would be encountered.
Lesson learned: ensure your design is sane before turning your PoC into a full blown production system.
I was always vaguely irritated by the term and never understood what it meant bc I was like „there must be some server??“
(not working in web dev)
I think there are probably reasonable use cases for this, like automation that's going to run infrequently, but it seems like a terrible way to implement a typical application that's going to receive frequent requests.
Or no check at all since it scales to zero and the free tier is ridiculously high for infrequent / low usage.
PoC works, costs nothing, code delivered much faster. CTO impressed.
PoC tidied up and rolled out to production. CTO very not impressed when quarterly expenses are reviewed, but customers are relying on this now.
Exactly our usecase. Then the network admins wanted to click button and select their template from lists rather than having to write json, so website it was, and now we run a MVC on lambda.
It isn't used that much (at most, you'll get 20 simulaneous users, and its always during a short time period, most of the time nobody use it). Don't know if it's the best choice. Honestly it seems fine.
But we obviously haven’t.
Which, despite not sharing your opinion, probably indicates the name was poorly chosen. Since it still has be defended.
Or perhaps it has just turned into one of those gnome v kde / vim v emacs things?
A fast modern computer will give a whole lotta vertical scaling.
Even more so if you use a compiled language.
I wrote a web application with rust using actix that did about 7 million requests per second.
Personally I would never use something like Lambda for compute, unless it's something that needs to integrate between another AWS service and something else (likely also a service.)
Also it seems like the "serverless" hype started (atleast at my employer), because some higher ups didn't trust the developers in my department, when they first had an AWS account provisioned.
My general feeling about picking the right tool to solve a problem is to, firstly, pick the problem apart. Then you can evaluate technologies that can solve your problem. This includes code that you already own and it should most likely be this one that you pick. After that you can start to look at other technologies to bring in.
Arguably it's nice that with AWS you can spin something up relatively quickly, however sometimes I'm also stumped by their limitations and what workarounds that seems to pop up as a result of that.
As long as you can select a p99 for timeouts that's sane, lambda does a really good job handling highly heterogeneous workflows in general.
So if a request could take 10ms or multiple minutes depending on Reasons(TM) - lambda makes a lot of sense because you don't have to overprovision for theoretical peak demand, and a swamp of expensive requests is isolated from your cheap requests.
Fortunately, I see a trend in more "advanced" companies to stay away from microservices/lambdas in the first iterations of their products and embrace good old modular monolith with a simple architecture.
In short, there is a place for both.
Verifying that the call patterns between the functions are correct is challenging sometimes - but it helps there to have the architecture mapped out visually.
I cant figure out what that refers to. From a user perspective there is only 1 channel (maybe one per region, but that still doesn’t hit 100s)
And for streams I can’t think of any kind of definition of stream that wouldn’t have orders of magnitude larger number associated with it on prime.
Serverless is great for event handling and bursty workloads. It is not efficient for a long running or constantly running processes. That was never what it was designed for.
There is a time and a place for serverless.