Hacker News Reader

Comment by nailer | Hacker News Reader

> Testability?

Serverless is specifically a stateless paradigm, making testing easier than persistent paradigms.

> Framework adoption?

Generally we use our own frameworks - I do wish people knew there was more than serverless.com. AWS throw up https://arc.codes at re:Invent, which is what I'm using and I generally like it.

> Stability? Industry Skills? Proven Architectures...?

These are all excellent questions. GAE, the original serverless platform, was around 2010 (edit: looks like 2008 https://en.wikipedia.org/wiki/Serverless_computing). Serverless isn't much younger than say, node.js and Rust are. There are patterns (like sharding longer jobs, backgrounding and assuming async operations, keeping lambdas warm without being charged etc) that need more attention. Come ask me to speak at your conference!

View on HN

> Serverless is specifically a stateless paradigm, making testing easier than persistent paradigms.

No because Lambdas are proprietary which means you can't run it in a CI or locally. Also, it becomes stateful if it pulls data from a database, S3 or anywhere else on AWS which it almost always does.

> Serverless isn't much younger than say, node.js and Rust are.

AWS Lambdas which I consider to be the first widely used Lambda service was was released in April 2015 which is 6 years after the release of Node.js. Also, Node.js is way more popular and mature than Lambda solutions.

Overall Lambdas are only useful for small, infrequent tasks like calling a remote procedure every day.

Otherwise, things like scheduling, logging, resource usage, volume and cost make Lambdas a bad choice compared to traditional VPSs / EC2.

> No because Lambdas are proprietary which means you can't run it in a CI or locally.

Arc has a full sandbox, so yes you can run a lambda in a CI or locally. You should know this - if you're developing your apps by deploying every lambda as you edit it you're doing it wrong. Most lambdas don't really need you to simulate AWS.

> Also, it becomes stateful if it pulls data from a database, S3 or anywhere else on AWS which it almost always does.

Sure, persistence exists, but when people say 'stateless' they mean 'has no transient state'. They don't mean 'has no data'.

> AWS Lambdas which I consider to be the first widely used Lambda service

OK. You don't consider GAE (2008) widely used. I disagree - mainly because I was using GAE back then and thinking about this new world of no-memory, timeouts, etc - but lambda is definitely more popular.

> No because Lambdas are proprietary which means you can't run it in a CI or locally. Also, it becomes stateful if it pulls data from a database, S3 or anywhere else on AWS which it almost always does.

Lambda is a function call. So it makes no difference if it’s proprietary or not.

Are you saying that it’s difficult to test passing an object to a function and asserting that it’s functioning as intended?

Lambas are not simple functions because your environment is different in local compared to production.

If I run a Node.js function in AWS Lambda, my Node.js version might be different, my dependencies might be different, the OS is different, the filesystem is different, so I or one of my node_modules might be able to write to /tmp but not elsewhere, etc.

It's the reason people started using Docker really. If you don't have the same environment, you can't call it reproducible or testable for that matter.

It is far too easy to have all the tests pass locally and be completely broken in production for this reason.

As with any compute environment...

See https://github.com/Nordstrom/serverless-artillery

Note performance, acceptance, and monitoring modes

Nothing you mentioned has anything to do with the ability to test a Lambda. You’re trying to use limitations and restrictions as friction to backup your inability to test.

There’s a lot of annoying things about lambda. And a lot of stuff I wish was easier to find in documentation. But that doesn’t change the fact that Lambda is more or less passing an event object to your function and executing it.

Writing a function in node 12 and then running it on node 4 and throwing your hands in the air cos it didn’t work isn’t the fault of Lambda.

It's great to see that factual evidence is answered with ad-hominem by the Lambda hype crowd.

In any case, if you have a Node.js module or code with a native C/C++ build, that runs shell commands, that writes to disk (not allowed besides /tmp in Lambda) or makes assumptions about the OS, your "simple" function will absolutely return different results.

e.g: My lambda is called when somebody uploads an image and returns a resized and compressed version of it. This is done using Node.js and the mozjpeg module which is dependent on cjpeg which is built natively on install.

If I test my function on my machine and in Lambda it's very possible that I get different results.

Also, certain OSs like Alpine which are heavily used for Docker don't event use glibc as compiler, so again, another difference.

I think you are right with your assumption that Docker images that don't resemble the production environment aren't sufficient to test.

But isn't the idea of Docker that you can recreate the production environment? If you can't why use Docker in the first place?

You are absolutely right that you could recreate a similar environment to Lambda in Docker. But you would first need to reverse engineer Lambda's environment to discover how it is actually configured and the limits that are set.

Even if you did find a way, you would still need to keep it up to date in case AWS decides to update that environment.

At least the Docker image used by AWS SAM CLI is created by AWS.

Also, you compile before packaging, so you dev/CI system already has to be able to compile for Lambda, independenly from testing/debugging with Docker.

Logged in to say that this has actually been done (not by me) and my team has been finding it very helpful for local “serverless” SDLC: https://github.com/lambci/docker-lambda . It‘s billed as “ A sandboxed local environment that replicates the live AWS Lambda environment almost identically – including installed software and libraries, file structure and permissions, environment variables, context objects and behaviors – even the user and running process are the same.” We test our functions mocked and against local deployments of that lambci container . There also lambda “layers” (container images for building custom runtimes for AWS Lambda) but we have not used that feature at this point. Interesting space with lots of room for improvement in this tool chain though for sure

Nice!

I saw that the SAM CLI uses an Alpine based image, does yours use Amazon Linux 2?

I'm jusz asking, because I compiled some libs on Cloud9 (which uses AL) and they worked on Lambda, so I assumed it's the same dist.

I’m not 100% sure as I didn’t create the image (though I’m evangelizing as someone who has found it truly helpful for daily dev.) . I believe the creators tarball’d the entire distro/execution environment from a running lambda so the file system layout and libs likely match Amazon Linux if that’s the default lambda execution distro image. If not I assume it matches the default

Why not then have lambda run the same container you can run and test locally?

I don't use lambda but we have our jenkins spin up the same ec2 to run tests that we would spin up to do development so that we never run into this problem.

I'm not sure I understood your question correctly.

If you mean running a Docker container in Lambda, that is to may knowledge to possible. You could schedule Docker tasks in AWS ECS (their managed container service) but it's not meant for anything realtime and more for cron job type tasks.

If you mean emulating the Lambda environment in Docker, then I wrote an answer with the difficulties of doing that below to another user.

"In any case, if you have a Node.js module or code with a native C/C++ build, that runs shell commands, that writes to disk (not allowed besides /tmp in Lambda) or makes assumptions about the OS, your "simple" function will absolutely return different results."

This is true, but it's not Lambda qua Lambda. That's just normal production vs. testing environment issues, with the same basic solutions.

Lambda may offer some minor additional hindrances vs. something like Docker, but I wouldn't consider that catastrophic.

Saying that local dev and Lambda are different is a strawman. How is that harder than developing on a Mac or Windows (or even Linux) and then testing on a different OS and config via CI/CD?

You shouldn't be testing "on your machine" - that's the oldest excuse in the book!

You should build your function in a container based on AWS Linux, just the same as you should for a Lambda deploy. That guarantees you the same versions of software, packages, libraries, etc. It makes it possible for me to develop Lambda functions on a Mac and test binary-for-binary to the deployed version.

"Nothing you mentioned has anything to do with the ability to test a Lambda" is not ad-hominem, it's a statement of fact.

> > Writing a function in node 12 and then running it on node 4 and throwing your hands in the air cos it didn’t work isn’t the fault of Lambda.

> It's great to see that factual evidence is answered with ad-hominem by the Lambda hype crowd.

I don't think that was a personal attack.

We've answered technical questions with technical answers.

- You have a definition of stateless which includes having no persistence layer, which is at best at odds with the industry.

- You think serverless was created with AWS Lambda which we've been kind about, but most people would say you're simply wrong.

- You're advocating for containers, which are well known for having their own hype as people write their own cloud providers on top of the cloud provider their employer pays for with dubious benefit.

The place where I work, we have "cloud in a cloud" initiative, total waste of time But you can't blame containers for it

> Node.js version might be different

Yes, in major only. Your lambda has node 10, you might be running 10.x or 10.y

> my dependencies might be different

No, the lockfile does that.

> the OS is different, the filesystem is different

Agree, but how much does one Linux+systemd different from other Linux+systemd? How much does the FS?

> It's the reason people started using Docker really.

VMs, docker and having to care about and manage isolation platforms is the reason people started using serverless.

> No, the lockfile does that.

No, your lockfile doesn't care about build steps so any post-install script might run differently for the many other reasons listed.

> Agree, but how much does one Linux+systemd different from other Linux+systemd? How much does the FS?

Plenty. For example filesystem change events are known to have filesystem and OS dependent behaviours and quirks / bugs.

When a Node module runs a shell command, it's possible that you have a BSD vs a GNU flavour of a tool, or maybe a different version altogether.

The Linux user with which you are running the function might also have different rights which could become an issue when accessing the filesystem in any way.

> VMs, docker and having to care about and manage isolation platforms is the reason people started using serverless.

Maybe, but serverless doesn't answer those questions at all. It just hand waves testing and vendor independent infrastructure.

> No, your lockfile doesn't care about build steps so any post-install script might run differently for the many other reasons listed.

You shouldn’t be uploading node_modules folder to your deployed lambda so this is an issue of your development environment, not lambda.

> Maybe, but serverless doesn't answer those questions at all.

“Serverless” or lambda/Azure functions, etc, are not a silver bullet that solve every single scenario. Just like docker doesn’t solve every single scenario, not does cloud servers or bare metal. It’s just another tool for us to do our job.

> > > my dependencies might be different

> > No, the lockfile does that.

> your lockfile doesn't care about build steps

Then you're not talking about dependency versioning are you? you're talking about install order. In practice it hasn't been an issue, I should find out how deterministic install order is but I'd only be doing this to win a silly argument rather than anything that has come up in nearly a decade of making serverless apps.

> For example filesystem change events are known to have filesystem and OS dependent behaviours

> When a Node module runs a shell command, it's possible that you have a BSD vs a GNU flavour of a tool

Are you generally proposing it would be common to use an entirely different OS? Or a non-boring extX filesystem?

All your issues seem to come from edge cases. Like if you decide to run FreeB or ReiserFS locally and run a sandbox it it, fine, but know that's going to differ from a Linux / systemd / GNU / extX environment.

> > VMs, docker and having to care about and manage isolation platforms is the reason people started using serverless.

> Maybe, but serverless doesn't answer those questions at all.

Serverless exists precisely to answer the question. I can throw all my MicroVMs in the ocean with no knowledge of dockerfiles, no VM snapshots, no knowledge of cloudinit, no environment knowledge other than 'node 10 on Linux' and get my entire environment back immediately.

> Then you're not talking about versioning are you? you're talking about install order.

I didn't mean build order but install scripts and native module builds.

The first type can create issues when external resources are downloaded (Puppeteer, Ngrok, etc.), which themselves have different versions or that fail to download and where the Node.js module falls back to another solution that behaves slightly differently.

The second type can occur when you have say Alpine Linux that uses MuslC and Amazon Linux uses GCC or when the native module tries to link with a shared library that is supposed to exists but doesn't.

> Are you generally proposing it would be common to use an entirely different OS? Or a non-boring extX filesystem?

I haven't checked but Amazon Linux by default uses XFS on EBS disks so I wouldn't be surprised if Lambda's used the same. So not a boring extX filesystem. ZFS is also relatively common.

> Serverless exists precisely to answer the question.

No, it clearly doesn't because your function will fail in local and succeed in Lambda, or the reverse, exactly due to the issues I mentioned in my various comments here and you will be left debugging.

Debugging which starts by finding exactly the differences between the two environments which would have been solved by a VM or Docker.

> > > my dependencies might be different

...

> I didn't mean build order but install scripts and native module builds.

OK. Then you're still not talking about your dependencies being different. The dependencies are the same, they're just particular modules with very specific behaviour...

> external resources are downloaded (Puppeteer, Ngrok, etc.), which themselves have different versions or that fail to download

That's more a 'heads up when using puppeteer' than a indictment of serverless and a call to add an environment management layer like we did in 2005-2015.

> Linux by default uses XFS on EBS disks so I wouldn't be surprised if Lambda's used the same.

That's worth checking out.

> Debugging which starts by finding exactly the differences between the two environments which would have been solved by a VM or Docker.

I see what you're saying, but planning your whole env around something like a given puppeteer module dynamically downloading Chrome (which is very uncommon behaviour) isn't worth the added complexity.

"No because Lambdas are proprietary which means you can't run it in a CI or locally."

It kinda is, but not really. You get an event object as param and often only need a few fields from it.

Also, you can run Lambda locally, AWS SAM CLI lets you run them in Docker for debugging purposes.

I found SAM to be extremely difficult to use, especially on windows where docker is just plain old terrible.

True.

I stopped using windows for dev machines long time ago, but I'd guess with the Linux subsystem stuff it will catch-up in the next years.

I honestly have no problems developing on windows (except when my macos coworkers dont consider windows). Docker is the only real issue.

Yes, it's unusable on macos and windows.

Hopefully WSL2 will solve this.

Docker largely works fine on macOS? At least for testing, haven't run into any issues other than bad filesystem performance if you're writing a lot of data to a mounted FS. Our team uses it daily - a far cry from "unusable".

Maybe you just got a higher pain tolerance or better machines?

I have the latest Windows insiders build on a spare machine. I've found all the aws tooling including SAM works pretty well under WSL 2 now that WSL uses a true Linux kernel.

Tried getting SAM to work on OS X, no luck, lots of pain.

Just to be clear: lambdas are not stateless if you, for example, connect to a DB or use any other external service.

State could be somewhere else, but if you are not also "pure", you don't have any improvement over a normal service.

State is saved somewhere, sure. How many useful applications are truly stateless when considered in their entirety?

This isn't the sense in which servers are usually said to be stateless, however. A Lambda that reads and writes state from a database when invoked is not maintaining state outside of the request-response cycle itself, so it can be said to be stateless.

Yes, but the parent said "statelessness that makes it easier to TEST". It is not stateless in that sense: purity makes it easier to test. Here you need to mock all interactions with external services, just like you'd do with non-serverless applications.

It is stateless in that sense. To run your lambda, there needs to be zero things in memory, unlike a traditional express/django/rails or other non-serverless apps.

If your lambda involves persistent storage, your testing might want to wipe or prepopulate the DB with some objects first, but that's not hard and doesn't add complexity, and as mentioned you don't need anything in memory.

It's funny because the only use case I was considering lambda for was a pure / stateless function. As soon as you add state / side-effects I assume you've greatly increased your complexity and testing (you'd want to have separate tests for all of the inputs / outputs that resemble production... which is when things can get complicated).

I'm probably looking at it wrong I guess. I considered using lambda to unload some CPU intensive calculations / logic and that's it. I figured it would be good for that as long as the latency didn't outweigh any benefits.

On-disk state isn't what people mean when they say stateless. * They mean there is no memory state, which is true and which absolutely means serverless functions are easy to test.

* you could argue that 'stateless' should include no long term persistence, but you'd be fighting an uphill battle. Like saying 'serverless' isn't correct because there are servers.

That's not what I've understood "stateless" to mean. Sure, anything more complicated than a calculator app is going to rely on data stored elsewhere, and in Lambda world that means you're reading from DynamoDB, S3, RDS or whatever. Those are definitely dependencies and that's where the state lies. But the pattern encouraged by Lambda is that your instance contains no state of its own. The disk is used as scratch for each invocation, objects are created for each request and not shared between them, etc. That's what people mean by stateless.

I see your point, but this definition is so lax that it applies almost perfectly to any application I've ever deployed to the cloud. Using external services to save state isn't unique to lambda, any app hosted on a normal ec2 instance needs to follow the same pattern because the instance and its filesystem can go away at any time (unless you attach a volume, but I've always considered that to be bad practice).

No it's not. There's no in memory state. Your Django rails phoenix app has in memory state.

What in memory state?

Sure, it's a common pattern and the right one in many scenarios. But especially for non-cloud deployments it's very common to store session data in process memory or temp data to local disk, etc.

Lazy initialization and caching of a resource literally every instance of a Lambda uses doesn't fit the definition of "state" in my opinion.

Reply on news.ycombinator.com