LLRT: A low-latency JavaScript runtime from AWS | Hacker News Reader

LLRT: A low-latency JavaScript runtime from AWS

github.com

217 pointsascorbic82 comments

This is awesome, I'm incredibly happy to see more alternatives on the Javascript Runtime Space!

In a nutshell: LLRT aims to have incredibly fast startup times compared to Node, Bun or Deno (which is critical in Lambda environments).

LLRT is built using Rust, Tokio and QuickJS under the hood. Here's their compatibility with Node.js APIs [1]

We (at Wasmer), have been working on a very similar approach: WinterJS, which is using SpiderMonkey instead (but also built on top of Rust and Tokio) [2]. HN announcement was just three months ago [3]

Really excited to see how the ecosystem is evolving!

[1] https://github.com/awslabs/llrt?tab=readme-ov-file#compatibi...

[2] https://github.com/wasmerio/winterjs

[3] https://news.ycombinator.com/item?id=38047872

  curl https://github.com/wasmerio/winterjs | grep -i license # :-(

Oops, will add the license right away!

Edit: MIT license added! We had it on the Cargo.toml and wasmer.toml, but forgot to add the LICENSE file to the repo https://github.com/wasmerio/winterjs/blob/main/LICENSE

Faster than Bun?

Not bad!

Easier than you'd think!

I guess!

But I was already blown away by Bun's improvements over Node.

Are there any benchmarks comparing startup time of LLRT, Deno, and Bun?

Edit: particularly curious about comparison with Deno deploy.

Deno’s and Bun’s Isolates should be spawning faster than a QuickJS process, they also utilize snapshots for more optimization. QuickJS is going to be faster at startup from 0 though

This seems ideal for places you use Lambda as glue, like for routing requests or authorization policy decisions. But for application that doing a lot of thinking, keep in mind v8 jitless is about 3x faster than QuickJS, and with JIT it’s about 30x faster than quickjs. I’m curious to see where the break-even point is for various workloads comparing the two, and also numbers versus Bun, which starts quicker than Node but has comparable top-end JIT performance.

One good thing Lambda provides is contractual guarantee, if you want 100000 code spun up you can put it in writing and hold AWS against it

but then you realize you can achieve the same thing at 98% discount with traditional "monolith" setups with a load balancer soaking up all the requests

Serverless is a major failure

The Cloud(tm) experience

Serverless has two things that I really like about it:

1. Scale to zero, which allows me to do things like automatically deploy a full environment for each branch in a project, with a full infra, frontend, backend, database, etc.

2. Good integration with IaC tools, so that I can define my infra and my runners in a single language/tool, be it terraform/cdk or something else. Most "monolithic" setups split configuring the infra and what runs in it into two tools/steps (please let me know of ones that don't!).

But if I actually run a application for a long time with somewhat consistent load there are always cheaper, more performant and flexible solutions than a serverless setup.

We've found it really nice for worker queues for specific workloads that fire infrequently'ish.

That’s a depends situation. If you’re using direct service integration instead of Lambda, it can be cheaper. And, if you use VMs only as the front-end to push batches of requests to serverless compute you get the best of both worlds and the cheapest by a long shot. Not everything can fit in that model, but the things that do are magically inexpensive.

I’m actually working on a product to make that architecture more approachable (and cheaper yet). I’d be happy to hear from folks running network services on VMs and wishing there was a better way.

Do you mean that you can get actual scale-to-zero using that approach? I think the cost of the VM would not scale down like that, right?

That’s true, the front-facing VMs always need to be there (hence the product I’m working on — so at least you don’t need to run them). The real work is done in serverless, in batches. Basically I think the world needs something like API Gateway for UDP and custom TCP, but that targets serverless backends rather than VMs.

This is only true for certain workloads. Specifically, workloads that you can put behind a load balancer.

One huge advantage of FaaS infrastructure like Lambda is when your workload doesn't need to process >1 rps. Lambda, for example, has tight and unique integration with foundational AWS services like S3, SNS, SQS, and EventBridge.

Lambda is awesome when you need small scale reliability. I shouldn't have to run a VM 24x7 if I don't need to!

Running 1000's of EC2's at scale? Want to handle instance health notifications?

My favorite use of Lambda is S3 object event notifications, it makes it very easy to handle a variety of odd jobs by decoupling storage and compute and managing each independently. S3 Events are exactly the right use case for LLRT.

Yeah, that was what I was alluding to as "aws glue"; for something like S3 object event notifications or IAM policy checks this kind of light weight runtime seems ideal.

> Serverless is a major failure

Sure, if you try to shove everything into that mold it is, but there are absolutely times where it's the best tool for the job. I run my entire, personal, company on top of it and it saves time, money, and energy.

How does it save money, can you give us an example? Thanks.

Sure, I pay less than a dollar most months then in my busy months I’ll pay a max of ~$20 for my backend.

It’s event software, like for a food festival. Most of the months there is almost 0 traffic, for 1-3 months there is lower traffic as pre-sales start with random spikes when the event organizers advertise, then for 1-5 days there is high traffic peaking on a single day or two during the event.

Because the load is incredibly spiky it’s nice to let Lambda scale automatically for me. Trying to scale a server, even bare metal, manually would be a pain in the ass. Even as I run this software for multiple events it’s cheaper than $5/mo and a $5 instance wouldn’t be able to handle the load at peak.

If my lambda bill ever got higher than an average of $20/mo I might consider a bare metal or VM server but honestly I prefer not thinking about it. And the profit dwarfs the server costs, like it’s not even close. The lambda server costs, even if they were 50x, would still be minor.

Faster what? Runtime performance or startup performance? I bet QuickJS would be the fastest to startup from 0 (no JIT = no warmup)

I'm guessing for HTTP the difference in performance is negligible. QuickJS is like 50x smaller than V8 so the gains in startup time and memory usage will have a deeper impact for this use case.

https://bellard.org/quickjs/bench.html

Interesting project: they are trading lower raw throughput in compute-heavy applications for lower latency at startup and (likely) better behaviour w.r.t memory and "random GC jank".

If you are writing an I/O bound JS service that doesn't do a lot of thinking, but does a lot of "moving stuff around", this will probably be a win.

Yes, though don't think of "traditional" JS engines as being all the way on the throughput side. One of the main distinguishing characteristics of JS engines, at least those that are used in web browsers, is that they have to care a lot about startup time and have to juggle a lot of tradeoffs that other managed runtimes typically don't need to bother with.

A surprisingly high percentage of downloaded JS code is never run at all. (I can speculate: event handlers, unshaken trees, shaken trees with paths that happen to not be taken, ...) Of the rest, the vast majority is run once or a handful of times. That means startup time is a high percentage of total time, and therefore engines will JIT very little of that code, and the little that is JITted will use a baseline JIT that pretty much just translates opcodes to unoptimized machine code. The fancy optimization engineering that requires much cleverness and effort is actually applied to a very small percentage of incoming code. It will need to run something like 1000x before it is seen worth bothering with. Latency is more important than throughput for almost all of the JS that the engine sees. (Note that throughput-sensitivity can easily dominate most of the runtime, but that varies widely depending on workload.)

So browser-embedded engines already work hard at optimizing latency. We're not talking Python or Ruby here.

That still leaves a lot of potential for improvement by stripping the engine down and prioritizing latency above all else, though. There's definitely a niche for things like LLRT. And WinterJS, too.

(Source: I work on SpiderMonkey)

Hermes is a big one as well: low startup latency, low memory usage, pre-compiled and the newest version comes with static-optimization

https://hermesengine.dev/

The benchmark shows Node.js taking 1.5s(!) to start vs. LLRT taking under 100ms.

But the benchmark appears to be a very small program that imports the AWS SDK.

Later on in the readme, it's mentioned that LLRT embeds the AWS SDK into the binary.

So... How much of what the benchmark is showing here is just the fact that the embedded SDK loads much faster than application code?

I mean, it's certainly a valid optimization for Lambdas that are primarily making AWS API calls. But it'd also be interesting to know how much faster LLRT is at loading arbitrary code. As well as interesting to know if the embedding mechanism could be made directly available to applications to make their own code load faster.

(Disclosure: I work on Cloudflare Workers, a competitor. But, honestly interested in this.)

Even faster would be just having FFI, so you could skip Rust <> C <> JS conversion every time you call a function.

This is what hermes is currently doing.

True but probably very tedious to achieve full compatibility with the existing JS API that way, which seems like it was a big goal here.

Not really. I have built a fetch-like API for hermes on top of libcurl using generated FFI-bindings (they have a bindgen tool in the repo). All I had to do is just write the JS-wrapper that would call the underlying C-lib and return a Promise.

It feels wild, people have been calling UI-libs from JS as well: https://twitter.com/tmikov/status/1720103356738474060

I can imagine someone making a libuv binding and then just recreating Node.JS with the FFI

I mean that the AWS SDK is a large API surface, they'd potentially have to re-implement all of it in Rust and then keep up with future changes, otherwise you have to tell people it's not 100% compatible.

I'm told this library is multiple megabytes of JavaScript. (That's why Node takes so long to load it...)

This benchmark is really an esoteric performance comparison, because the AWS SDK isn’t monolithic anymore and you can only include the packages you need (assuming you aren’t using a minifier and tree shaker to reduce the parsed code anyway). It was only a monolithic library back in v1 and they’re on v3 now. For most use cases, you only need to load a small subset of the AWS SDK. I have certainly written node lambdas with only a few hundred ms coldstarts using multiple aws SDK libraries bundled in (secrets manager, dynamo, s3, parameter store…)

AWS generates all their SDKs from something like Smithy. Not to say that that doesn’t have its own problems, but given that they have an existing Rust SDK, it’s not an insoluble problem.

I am also a bit surprised they chose a bechmark that includes network latency as well. Also the 1.5s NodeJS cold-start seems quite high and is not what I would expect at all. Especially when looking at https://maxday.github.io/lambda-perf/

The SDKs are "bundled" in all lambda runtimes but normally not into the binary, what additional performance would that bring?

> The SDKs are "bundled" in all lambda runtimes but normally not into the binary, what additional performance would that bring?

I haven't looked at LLRT's internals, but if I were them, and I were bundling some JavaScript code into a binary and seeking to really optimize startup time, I would probably pre-parse the JavaScript text to produce QuickJS bytecode (or whatever data structures QuickJS actually interprets at runtime; no modern interpreter is actually processing raw text as it goes). In the best case, embedding something like that into the binary could mean that startup processing of the embedded code is O(1) (just like how startup time for a native-code binary is independent of its size, as long as it doesn't have global constructors).

From all that I have read, file-size is a major part of cold-starts as your zip needs to be fetched internally first so there is some app size where increasing the files by pre-parsing might make it slower and only after a certain size would it make sense. Edit: Sorry, you were talking about the runtime, not the app bundle, just realized this now. If it‘s in the runtime it probably is free as that will probably be available in all lambda hosts I guess. Lambda already has an even more aggressive optimization than just pre-parsing, which they call SnapStart; where it takes a RAM snapshot after init and loads that afterwards but I think it‘s inly available in some regions and some runtimes like Java with very slow cold start times.

> offers up to over 10x faster startup and up to 2x overall lower cost

i've been burned by Amazon with these type of claims can we get confirmation this is backed up actual real world testing from a non-amazon affiliated source?

It’s because they never get sued for saying stuff like this

Well it does say "up to", that's pretty easy to prove with a niche contrived example

Up to 2x, or more.

The full quote is this:

> LLRT offers up to over 10x faster startup and up to 2x overall lower cost compared to other JavaScript runtimes running on AWS Lambda

I read this to mean that AWS only cares about performance on AWS Lambda, and it's probably not this much faster outside of the Lambda contxt

I would love to see similarly low startup times with a real JS-to-WASM compiler (rather than today's approach of compiling a JS runtime to WASM).

With WASM GC landed, and WASI preview 3 taking on async / event loops, it seems like this should be possible soon, except for the very large task of actually writing the compiler.

There's a VERY good reason why JS runtimes use JIT's, it's more or less impossible to properly fully type a JS program w/o leaving runtime holes.

If you restrict yourself to a subset or are willing to risk incompatiblities then it will work.

There's some TypeScript to Wasm compilers that should work reasonably well (If you're breaking TS rules you've got yourself to blame a bit more than with JS).

Ref: My thesis was on JS AOT and even if I had the idea of making it commercial I didn't pursue it in the end.

AssemblyScript works fine too! WASM and JS isn’t a good combination atm.

> My thesis was on JS AOT

Is it available online?

Yes, it was written for other students so it's not as clean as most of the referred research papers. Part of the work was implementing a compiler whilst part of was surveying how well real world JS developers would understand language subset limitations.

Most relevant for my above comment re AOT vs JIT is that JS semantics (I think I touched upon it in the thesis) will have the same issues that made JIT's win out with BigInts in the Agesen&Hölzle paper (whilst the paper is on AOT's my goal was to find a solution for game developers so numerical performance was a non-moving goal for me).

My thesis http://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A8232...

Agesen & Hölze on JIT vs AOT https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&d...

Thanks.

QuickJS and SpiderMonkey in WASM don't use JIT though - they're both just interpreters.

WASM itself is often JIT'ed so a JS-to-WASM compiler that functions like the common JS baseline compilers would still get some benefit of JIT'ing even though the types available to the ICs in WASM may not be as useful as in JS.

https://cfallin.org/blog/2023/10/11/spidermonkey-pbl/#future... has some interesting ideas on enabling the SpiderMonkey baseline compiler for WASM and possibly using partial evaluation to see better types.

If I understand it correctly, it exchanges execution speed (by not using JIT) for faster starts. Instead, they could pick up the old FB project https://prepack.io/ to optimize both start AND execution speed. It would be very interesting to see it together with node's snapshots: https://blog.logrocket.com/snapshot-flags-node-js-v18-8/, which reduces the start significantly.

I am working on similar runtime QuickJS/C/C++ using async stackful coroutines. Last year tests were about 14 times faster for networking related to node.js Here are tests results: https://voda.io/test.html

I plan open source it later this year - have improve build and remove some client's code.

Wish someone pay me to build stuff like this, such a cool project to work to

Fastly, Cloudflare, Vercel, Fermyon, Wasmer, Azion, Deno, Netlify, SecondState, Cosmonic are paying to build stuff like this, just to name a few

Forgot to mention Shopify as well

https://shopify.engineering/javascript-in-webassembly-for-sh...

And Bun of course :)

pljs (https://github.com/plv8/pljs) is a postgres language plugin that also uses quickjs. given how much faster the startup is compared to v8, and that most postgres functions are very small and do very little compared to a node.js program, it is quite good.

I can definitely see aws' lambda operations gaining quite a bit from this.

I think Jarred Sumner (creator of bun) summarised it well:

Only faster for code that runs under 5 ms. But a lot of code runs under 5 ms.

(I hope the quote is right, I'm not on twitter.)

I think it makes Lambda better, but not more attractable to use cases that are better served by scale-to-zero managed containers runtimes like Fargate.

Does anyone know if this is what is used for Cloudfront Functions ? (Not lambda@edge)

LLRT does not use a JIT. Does that mean it’s just old style slow code from an interpreter?

I can certainly see how that would substantially reduce cpu and memory cost but the code would execute 50-250x slower.

Here are some benchmarks:

https://bellard.org/quickjs/bench.html

If startup time is relevant to you then you are probably not doing much computation in the first place.

Makes sense for a project like that. Lambdas are mostly used to glue AWS APIs anyways

Wonder how it compares to https://aws-lite.org/

i wonder why no one is talking about workerd, cloudflares much more mature approach that is behind cloudflare workers. deno is great but has much broader scope than llrt, workerd or winterjs. these are really low latency edge runtimes not general purpose ones.

For ref:

- https://blog.cloudflare.com/workerd-open-source-workers-runt...

- https://github.com/cloudflare/workerd

I wonder how this compares to QuickJS.

I wonder how popular "sever less" would be if it were called "depending on the kindness of strangers"

Since we're paying for it I'd call it more of "depending on the competence of service providers." AWS Lambda experience has been great for us in that sense, so I'd be looking forward to more general availability of this for Lambdas.

I think this project runs on the QuickJS engine

It's built on QuickJS.

https://github.com/awslabs/llrt/blob/e2cc8237f4bd04e161d46b3...

This wraps QuickJS and there is not a single mention on their front page. At least a hello or shout-out to Fabrice Bellard would have been nice.

But I guess it's AWS and they do this with other projects, so not too surprising.

> At least a hello or shout-out to Fabrice Bellard would have been nice.

Mentioned in the third-party file: https://github.com/awslabs/llrt/blob/630ec79/THIRD_PARTY_LIC... / https://archive.is/0NUAa

> AWS and they do this with other projects

Remember similar concerns raised viz Firecracker (which is a hard fork of Google crosvm: https://news.ycombinator.com/item?id=22360232), but don't recall it being a pattern.

> AWS and they do this with other projects

Probably a nod to the ElasticSearch licensing debacle.

Gross.

Amazon's repo here is open source at least so perhaps this could be remedied with an edit to their README and a PR - go for it !

Agree with you there should be credit where credit is due - I have been using QuickJS for some time and its awesome. For the cost of about 1MB you can get entire modern JS in your C/C++ binary.

We use QuickJS in Minecraft for our scripting engine. It's nice that he has got back into it these days and been adding ES2023 support.

I also had to double check but we do indeed list it on our website, albeit in a sea of others.

https://www.minecraft.net/en-us/attribution

They just added the reference to QuickJS [0].

> LLRT is built in Rust, utilizing QuickJS as JavaScript engine, ensuring efficient memory usage and swift startup.

[0]: https://github.com/awslabs/llrt/commit/054aefc4d8486f738ed3a...

It seems they just added the mention to QuickJS, I assume, based on your feedback:

https://github.com/awslabs/llrt/commit/054aefc4d8486f738ed3a...

Props to them on the quick fix!

Public shaming: it's one of the few avenues that work against the likes of AWS.

Not a good look tbh, you could glue this together in an evening yet they're ...proud?

I'd definitely spend an evening watching a video of you gluing an equivalent solution together. And if you did finished in one evening, you'd be justified in being very proud.

Well, if people would watch it I kinda have to now...

My only wish for AWS team is that they measure success based on how complete a product is. There are still 4 year old outstanding critical github issues around Cognito. There are lot of things that are advertised that aren't delivered.

In fact our trust in AWS and the team has plummeted that we ultimately ditched serverless and went str8 to VPS

The days of zero interest rate series B buying up reserved EC2 instances in the millions are gone. As capital gets expensive we see more reluctance to moving to cloud.

AWS claims of "10x faster startup @ half the cost" is confirmation that the Serverless movement is DOA

Quick everyone submit an update README pr

And there's not a single mention of Charlie Gordon in your comment about QuickJS. Would have been nice.

Reply on news.ycombinator.com