Comparing AWS Lambda performance of Node.js, Python, Java, C# and Go | Hacker News Reader

Comparing AWS Lambda performance of Node.js, Python, Java, C# and Go

read.acloud.guru

113 pointsdfirment25 comments

Open articleView on HN

This is a good example of an absolutely pointless comparison. The author is making some bold statements:

> Go performance is comparable to Java

> .Net Core 2.0 can execute up to 3x faster than Go.

> Consistent performance of compiled vs dynamic

But what are they measuring? How fast can a lambda function return "200 OK hello"? This is just sad.

They are effectively measuring how fast AWS infrastructure can reach whatever it needs (VM?) for each language.

But these are not even cold start times. I'm assuming after the first request this is using the same VM (/backend job, whatever) to reply, so now it just depends on latency to get to that VM. (I'm assuming the runtimes' ability to return "200 OK" is similar and the latency of that is negligible compared to AWS infra that is involved.) Not completely meaningless, but not super useful either, esp without any details while making those bold statements. This is not "Go is equal to Java". This is "AWS Lambda infrastructure for Go is comparable to AWS infra for Java". Article is unclear on that.

Maybe this tells us AWS lambda backend infra is way different for .NET Core than Go and Java? ¯\_(ツ)_/¯

> This is "AWS Lambda infrastructure for Go is comparable to AWS infra for Java"

I wouldn't even go that far. They are not measuring anything that any real-world application would do. Heck, it's not even a todo app, it's just a "200 OK hello".

It's more like "AWS lambda function invocation overhead for Go is comparable to AWS lambda function invocation overhead for Java".

Sure but if you start adding in language specific Web frameworks / json handling / db work you're no longer testing AWS.

I agree there's questionable value here but at least it's bring constrained to as few variables as possible.

Benchmarking the lambda-add latency is something that is useful as long as it's called out as such a benchmark.

It's a pity there's no C/C++ example as well (maybe with openssl?) For low level comparison

Why the hate? The methodology is sound, the results are interesting and they show that there is some room for improvement in language support. Also the source code has been provided in case anyone wants to alter the benchmark to be more comparable to their workloads.

Hopefully this sparks a race between the different teams within Amazon. Great job!

Oh! These test scenarios seem to have a pretty good acceptance in case you decide to do another benchmark in the future: https://www.techempower.com/benchmarks/#section=code&hw=ph&t....

To the extent that this benchmark is valid, this is interesting:

"In the original 2017 benchmark results, the compiled languages of Java and .Net Core 1.0 were clearly more consistent compared to the dynamic languages — Python and Node.js.

Based on the latest 2018 performance testing results — we are no longer observing any significant variances between Java and the newer compiled languages (.Net Core 2.0 and Go), and the original dynamic languages."

TLDR: You may be able to use a "slower" language for Lambda without necessarily losing much.

>> You may be able to use a "slower" language for Lambda

at least for small projects

Ok, now being serious, they need to include a real example here. Whenever I run a Lambda function is to process something and make a call to at least one external service. I was a .NET developer for a long time and I seriously doubt .NET Core has been improved to the point where blocking http calls are handled better than in Go runtime. I may believe .NET Core may have some performance advantages in some CPU bound scenarios, but nothing more. Until then, this benchmark proves nothing.

In which scenario would you use a blocking http call?

Not necessarily HTTP, but I imagine hitting some sort of database from a lambda is not uncommon.

Isn't the intent with a system like Lambda that you'd not do any kind of blocking action. Instead, you'd trigger some event that the AWS database solutions would receive. Once the DB has finished the query it sends the data back and triggers another lambda function.

EDIT: I write this as someone who does no web programming and has only toyed with AWS. This was my understanding from what I read, but not something I actually made.

The relevant methods in the AWS .Net SDK are all awaitable, i.e. not blocking. Even if you did block on waiting for it, an operation on DynamoDB such as writing a record should typically take under 10ms, which would be ok.

So there is no reason at all not to do it in a lambda.

No. "Once the DB has finished the query it sends the data back and triggers another lambda function"

What mechanism is the DB 'sending' data via? Is it just the response part of the initial query? Then it's executing in the context of the same lambda function. You -could- then call another lambda function from the running lambda function (but why?), but more, what was the function that executed the query doing while it waited?

If it had something it -could- be doing, it should be doing it, so non-blocking is important. If it had nothing it could be doing, it literally does not matter if it was a blocking call or not; it doesn't effect anything.

Now, if you meant you have a trigger in the DB that calls another lambda function, such that the original lambda could complete, yes, of course you could do that. However, if this was a GET style call, not a PUT, your options to get the data back to the user are limited if you don't just keep the REST request open and respond that way. Certainly, any other solution is more complex, probably more expensive, and may not be possible.

From what I've seen in organisations, no. In fact you start 10 threads all of whom start blocking calls, several to DBs hosted outside of amazon.

Inefficient ? Yes. The quest for the $15 page refresh continues.

> hitting some sort of database from a lambda is not uncommon.

of course. Like e.g. dynamoDB?

Here are some examples of non-blocking calls from C# to dynamoDB that you could make from a lambda

https://matthiasshapiro.com/2017/03/21/tutorial-dynamodb-in-...

The rest of the AWS SDK for .Net core is like that too. All those Async methods.

Http should also be done in a non-blocking, awaitable way too.

The deleted comment is exactly right - Async/awaits operators are the .Net mechanism to avoid blocking calls, and they are available here.

Lambdas bill at a 100ms minimum. What's the point of a few ms of difference here, unless you're running some computationally-intensive code? If that's what you're doing, the efficiency of the language itself is going to be what matters, not Lambda's speed of executing the runtime.

> we are no longer observing any significant variances between Java and the newer compiled languages

Go on par with Java? .NET 2.0 beating everything else out?!

I feel like my world has just flipped upside down.

I have been doing a lot of experiments with Go and Lambda and have an open source boilerplate app here: https://github.com/nzoschke/gofaas

This benchmark leaves some room for improvement. With Lambda and X-ray you could really deep dive into where time is spent in a request. Here’s an example trace for a Golang function that does some actual work against KMS and DynamoDB:

https://twitter.com/nzoschke/status/969598630033162240?s=20

One thing I found surprising about the Go support is that the ‘lambda.Start’ helper uses reflection to invoke your go func. This could contribute to some of the cold start time.

Yeah that certainly seems...non ideal.

I am glad that the person who created this intended for the results to be reproducible. I will try my hand at what's in the git repo, and perhaps compare with some other languages too.

Comparing their open-source ecosystem, I still prefer to build my lambdas with NodeJS (or even Python) over C#. (if my only concern is performance)

What would be the right way to test this?

Reply on news.ycombinator.com