Hacker News Reader

Comment by brunoluiz | Hacker News Reader

I am curious to see if this will mean a shift to more efficient languages (Go or Rust) for Lambda services, as usually people default to JS

View on HN

From the research I did, here's how languages stack up in Lambda runtime (lowest first):

1. Python & JS 2. Go 3. C# & Java

I couldn't find any data on Rust.

The understanding at the time was that Python & JS runtimes are built-in, so the interpreter is "already running" Go is the fastest of compiled languages, but just can't beat the built-in runtimes. C# and Java were poorest as they're spinning up a larger runtime that's more optimized for long-running throughput.

https://docs.aws.amazon.com/lambda/latest/dg/best-practices....

https://medium.com/the-theam-journey/benchmarking-aws-lambda...

https://epsagon.com/development/aws-lambda-programming-langu...

https://read.acloud.guru/comparing-aws-lambda-performance-of...

Of course, benchmarks like this only go so far. Use as a starting point for your own evaluation; not as an end-all-be-all.

I’m not sure I’m interested in a hello world benchmark if it takes Python 5 seconds to import its dependencies in the real world.

Dependencies for the dynamic languages matter A LOT! Take a look at what it'll cost you for requiring the AWS SDK in Node.js, for your cold starts https://theburningmonk.com/2019/03/just-how-expensive-is-the....

This is very true. Just importing Django + Django Rest Framework + some other minor libraries in Google App Engine (standard) leads to painfully slow response times when a new instance spins up. Like, more than 10s to spin up an instance. Although App Engine seems to be 3-4 times slower than my desktop computer from 2014 on this particular task. I wonder if AWS lambda is better.

In the real world you don't import 5 seconds worth of dependencies into a lamdba, and a 5 second boot time for a longer-lived service is acceptable.

> In the real world you don't import 5 seconds worth of dependencies into a lamdba

Laughs in data science.

> a 5 second boot time for a longer-lived service is acceptable.

Not every application can tolerate the occasional 5-second-long request. Just because Python can cold boot "hello world" 3 seconds faster than Go doesn't mean that's going to hold in the real world.

You're mixing arguments here. It's not the occasional 5-second long request, it's "the app doesn't start serving requests for 5 seconds".

Using data science tooling in a lambda seems iffy, especially ones that are not production ready. And good luck getting such libraries in go.

Python cold booting an interpreter 3 seconds faster than Go is a big deal, especially if your target execution time is <50ms and you've got a large volume of invocations, and are not being silly and importing ridiculously heavy dependencies into a lambda for no reason other than to make a strange point about Python being unsuitable for something nobody should be doing.

> You're mixing arguments here. It's not the occasional 5-second long request, it's "the app doesn't start serving requests for 5 seconds".

Lambdas cold-start during requests. So the unlucky request that triggers a cold start eats that cold start.

> Using data science tooling in a lambda seems iffy, especially ones that are not production ready.

Nonsense, there are a lot of lambdas that just load, transform, and shovel data between services using pandas or whathaveyou. Anyway, don't get hung up on data science; it was just an example, but there are packages across the ecosystem that behave poorly at startup (usually it's not any individual package taking 1-2s but rather a whole bunch of them scattered across your dependency tree that take 100+ms).

> And good luck getting such libraries in go.

Go doesn't have all of the specialty libraries that Python has, but it has enough for the use case I described above.

> Python cold booting an interpreter 3 seconds faster than Go is a big deal, especially if your target execution time is <50ms and you've got a large volume of invocations

According to https://mikhail.io/serverless/coldstarts/aws/languages/, Go takes ~450ms on average to cold start which is still up a bit from Python's ~250ms. To your point, if you're just slinging boto calls (and a lot of lambdas do just this!) and you care a lot about latency, then Python is the right tool for the job.

> not being silly and importing ridiculously heavy dependencies into a lambda for no reason other than to make a strange point about Python being unsuitable for something nobody should be doing.

Not every lambda is just slinging API requests--some of them actually have to do things with data. Maybe someone is transforming a bit of audio as part of a pipeline or doing some analysis on a CSV or something else. Latency probably matters to them, but they still have to import things to get their work done. And according to https://mikhail.io/serverless/coldstarts/aws/#does-package-s... (at least for JavaScript) just 35mb of dependencies (which will buy you half of a numpy iirc) causes cold start performance to go from ~250ms to 4000ms.

My rule of thumb (based on some profiling) is that for every 30mb of Python dependency, the equivalent Go binary grows by 1mb, moreover, it all gets loaded at once (as opposed to resolving each unique import to a location on disk, then parsing, compiling, and finally loading it). Lastly, Go programs are more likely to be "lazy"--that is, they only run the things they need in the main() part of the program whereas Python packages are much more likely to do file or network I/O to initialize clients that may or may not be used by the program.

Curious why it would take 5 seconds?

The way I'm using lambda, I compile the lambda build image beforehand which contains the python packages already installed, and the only "time" restraint is that of the lambda spinning up itself.

If you ran e.g. "pip install -r requirements.txt" inside the lambda, then yes it would take time to install the packages.

Installing packages onto the system (“pip install”) is different than the interpreter importing them (loading them when the interpreter hits an “import” statement). Not only is it resolving imports into file paths and loading them into memory, but it’s also executing module-level code which tends to be quite common in Python, so it’s not at all uncommon for imports to take 5s or more.

Meanwhile in Go, dependencies are baked into the executable so there is no resolving of dependencies, and the analog to “module level code” (i.e., package init() functions) are discouraged and thus much less common and where they occur they don’t do as much work compared to the average Python package.

Interesting, I see what you mean, but in my time working with python I've never seen that as an issue. Perhaps in different domains such as big data it might be a problem.

The numbers you link don’t support your ranking, unless you’re specifically ranking by cold start alone. Even then it doesn’t make sense to group Python and Node but not Go, as Node and Go are significantly closer than Node to Python.

Interesting, thanks for sharing! This is the opposite of what most people would expect.

A lot of this was based around the fact that we've seen languages become just so much more performant. This includes Go/Rust/etc, but a lot of Node.js workloads are also sub 100ms, or fast enough that they'd benefit from this pretty well.

- Chris, Serverless@AWS

I've got bad experiences with go startup (i.e., cold runs). They're much more expensive than I would have expected. If node can indeed run in 40ms (as https://news.ycombinator.com/item?id=25267211 says), then I'm surely going back to JS.

What is your experience? Go is an AOT compiled language so the only thing I could imagine you running into on startup is loading the binary into memory? Theres not a cold-start issue with Go, as its not an optimizing JIT.

Edit: Bizarre. Seems like Go on lambda does actually have slower cold start than JS or Python. I wonder if its just that the binary is likely larger than the equivalent JS source code? https://levelup.gitconnected.com/aws-lambda-cold-start-langu...

My experience is that Go cold start takes around 900ms. Processing (parsing a JSON message, verifying it with one DynamoDB look-up, and storing it in DynamoDB) then takes between 11ms and 105ms. Go does use less memory than node, though, and that also counts in Lambda.

I hadn't expected it either, but it loads node faster. Perhaps via some VM trick?

I'm not sure about lambda, but cloudflare workers use v8 isolates to avoid starting a new process at all.

> The function did nothing except emit ‘hello world’.

A more realistic benchmark would be parsing a 1kb protobuf blob and printing some random key from it.

(this would require importing a non-stdlib parser)

Without knowing how it's implemented, my guess is that they're conserving python/v8 processes, so that they're not cold-starting the interpreter on each lambda execution.

You can't [1] do the same thing for a Go binary, so they have to invoke a binary, which might involve running some scans against it first.

This leads to some pretty counterintuitive conclusions! If you want minimal latency (in Lambda!!), you really should be using JS/Python, I guess.

[1]: OK. Maybe you could. Go has a runtime after all, although it's compiled into the binary! I have never heard of anybody doing this, but I'd love to read something about it. :)

Go binaries are pretty large compared to say, C.

Go is not AOT compiled.

... yes it is?

I wrote this in another comment here, but just so you don't miss it: Don't.

Personal benchmarks puts Rust as the most optimal language that I've tried to run on AWS Lambda so far.

same. I went through the trouble of implementing my function in Rust (Rocket), and it's actually quite slow because (a) startup is slow and (b) async/await is still pretty painful to use so I'm blocking on IO

JS is a great choice for Lambda thanks to great cold performance. I’m seeing runtimes in the 40ms to 100ms range.

Most of the time in Lambda is usually spent waiting for IO, which is slow in any language. If you’re using Lambda for heavy computation, that’s not a great choice.

Yep, we ran ~50M Node invocations last month on a small function. AVG was around 100ms but lots of sub 50ms invocations too.

On the one hand I read that JS Lambdas were often already under 100ms (30-50ms)

On the other hand I heard legends about under 10ms Rust Lambdas.

That's the point- billed per ms, a Lambda that executes in 5ms is 10x cheaper than one that takes 50ms. Billed per 100ms interval, the total cost of the two is the same.

Yes,

I'm not questioning that 5 is a tenth of 50. I'm questioning the Rust speed :D

I just checked cloudwatch for my rust-based function, I'm now being billed 1ms :D

IME Lambda functions are mostly sitting around waiting on I/O, so I don't think it would make much of a difference for those workloads. The important technical factors for those workloads are startup time and I/O capabilities...JS is strong in both of those areas. For simple Lambda functions JS still seems like a great choice, along with Go. Rust would be overkill IMO unless you need to share a codebase or aren't I/O bound or have some other unique requirements.

Reply on news.ycombinator.com