Hacker News Reader

Comment by Tainnor | Hacker News Reader

But in this example, the entire code is still blocking because it has to wait for one call of "callRemote" to complete while waiting for the next one to be executed. So I don't get the point.

  for (int i = 0; i < count; i++) {
    resp.add(callRemote(client));
  }

View on HN

Before virtual threads, Java server would reserve a full OS thread while that code is being processed. OS threads use lots of memory and there might a limited number available. With a single incoming request or low traffic that is not a problem, but with many parallel requests the server would choke on using too much memory or OS threads.

Previously in Java if you wanted to avoid code that blocks the OS thread you would need to use callback based code or reactive programming: https://www.alibabacloud.com/blog/how-java-is-used-for-async...

Javascript moved on from callbacks to async/await, which colors the async functions and requires the programmer to be aware of the issues related to async/await.

For Java Project Loom takes a different approach. For example with server applications, the server would start running the code in a virtual thread and most of the code that looks like blocking will actually work like unblocking code. You get to use regular blocking code but you get async performance.

When the code needs to wait for a response, JVM can unmount the virtual thread from the OS thread and then the JVM can run another virtual thread in the OS thread. Once the response is returned and the OS thread is free, JVM can continue executing the original request.

There are some caveats to virtual threads. There is some overhead (but not a lot), some IO calls still block (but you don't get an indication about using blocking IO) and you might stumble on new kinds of bugs related to resource exhaustion (try reserving a million OS file handles).

But no more async/await, reactive programming, callbacks. Mostly just regular imperative blocking-looking code. I'm very excited about that.

what are the issues with async/await?

the mental gymnastics.

Having fun using async/await in loops? https://zellwk.com/blog/async-await-in-loops/

An actual blocking model is much easier to grok.

This could be solved by using for..of in an asnyc function though.

It might look like surprsising behavior maybe, but putting the async keyword outside and looping without non-awaited callbacks is the only possible way to keep the semantics sound, no?

In other words, forEach would have to return a Promise too. That's where the function coloring rightfully annoys programmers to catch their mistakes, there are if cause also lint rules for this.

AFAIK there are two active proposals that aim to provide this natively (and additonaly enable working with lazy and/or infinite iterators or generator functions):

AsyncIterator and tc39/proposal-async-iterator-helpers

Things don’t have to be that way. You can reject the use of async/await in contexts that aren’t aware of it (delegating it to special APIs that properly handle it).

Can you elaborate? Do you mean like the Promise API with the "spesialisti APIs"?

You could have a forEach that awaits the promise created each iteration, for example.

Because that's what "blocking" calls were invented to avoid.

A better example would be

    var remoteResultA = callRemoteA(client);
    var remoteResultB = callRemoteB(f(remoteResultA));

Where handling a request means you have to make two http calls and those two calls have some order dependence. I.E. the result of one is used to construct the call to get the other.

What virtual threads give is the ability to write the code like I did above and still get ideal performance, as opposed to.

    callRemoteA(client)
       .then(remoteResultA -> callRemoteB(f(remoteResultA)))
       .then(... your world lives here now ...);

Virtual threads are, from a language semantics point of view, the same as regular threads. Virtual threads map maybe 10000 "logical threads" to a handful (maybe 8 or so) actual operating system threads.

The big difference is that for each regular Thread you have exactly one backing Operating System thread.

I agree that this would be a better example that would make more sense (and I also think that the reasoning you provided should be added as context to that code snippet).

That entire section is running on a virtual thread, which is scheduled on a platform (OS) thread. While it blocks for each callRemote invocation the platform thread is free to process other virtual threads.

https://docs.oracle.com/en/java/javase/20/core/virtual-threa...

In addition, this implementation detail might help in gaining some insight: "The synchronous networking Java APIs, when run in a virtual thread, switch the underlying native socket into non-blocking mode. When an I/O operation invoked from Java code does not complete immediately (the native socket returns EAGAIN - “not ready” / “would block”), the underlying native socket is registered with a JVM-wide event notification mechanism (a Poller), and the virtual thread is parked. When the underlying I/O operation is ready (an event arrives at the Poller), the virtual thread is unparked and the underlying socket operation is retried."

https://inside.java/2021/05/10/networking-io-with-virtual-th...

Yeah but if for some reason, no other (or few other) request is coming in concurrently, then the CPU will just sit by idly. I don't understand why you wouldn't issue the "callRemote" calls concurrently, that seems like it's missing the point of writing non-blocking code.

Exactly. That's the point. It is beneficial only when you have a huge number of requests relative to the number of platform threads.

The idea is that you can map a million of these virtual threads on to a small number of platform threads, and the JVM will schedule the work for you, to achieve maximum throughput without needing to write any complicated code. Virtual threads are for high throughput on concurrent blocking requests.

EDIT: the sequential calls to "callRemote" still process in sequence, blocking on each call. Don't get confused there. But the overall HTTP request itself is running on a virtual thread and does not block other HTTP requests while waiting for the callRemote invocations to complete.

> EDIT: the sequential calls to "callRemote" still process in sequence, blocking on each call. Don't get confused there. But the overall HTTP request itself is running on a virtual thread and does not block other HTTP requests while waiting for the callRemote invocations to complete.

Yeah I got that. But then the use case seems to be much narrower, because not every scenario is one where many concurrent requests come in at the same time. If you write non-blocking code you'll get high throughput no matter the number of requests. Ok, maybe non-blocking code is harder to write (and I don't think it's actually that hard if you use, say, Kotlin coroutines), but honestly this seems to me like something that developers should learn eventually anyway.

Or maybe it's me being weird. But I remember 10 years ago when node.js was being hyped for being "fast" because it was "async" and now suddenly using threads and blocking operations is again all the rage now.

The difference now is that it's implemented in the JVM instead of a library / framework. It is easier, simpler, and probably more efficient. You can get higher throughput from existing code with minor refactoring.

  Thread thread = Thread.ofVirtual().start(() -> System.out.println("Hello"));
  thread.join();

Also there are some traceability issues involved in using asynchronous APIs: "In the asynchronous style, each stage of a request might execute on a different thread, and every thread runs stages belonging to different requests in an interleaved fashion. This has deep implications for understanding program behavior: Stack traces provide no usable context, debuggers cannot step through request-handling logic, and profilers cannot associate an operation's cost with its caller. "

https://openjdk.org/jeps/444

In theory an async-aware runtime can stitch together a coherent and useful backtrace, but in practice most legacy tooling won’t :(

Collecting those potentially very expensive stack traces, specially in those highly concurrent environments with high throughput requirements would be great for debugging but probably is gonna kill the performance of the system. But it would be nice (even if as a very heavy hammer) as an option.

I'm not saying that virtual threads aren't a good thing (other runtimes than the JVM have had green threads for a very long time now), but that it doesn't seem like such a paradigm shift. It's still threads (with all the benefits and drawbacks), it's just now that they have less overhead.

Presumably I could just keep an existing server written in Spring MVC or a similar technology and just wait for the underlying container to support virtual threads to get the same benefits. I believe Jetty already does support them. So why would I need a new framework?

It removes the need to write async code in many cases. It is a more straightforward and efficient way to accomplish what we have already been doing for years on the JVM.

You don't need to switch frameworks

https://spring.io/blog/2022/10/11/embracing-virtual-threads

You certainly could make the callRemote calls in parallel, and there are easy and safe ways to do that (good old CompletableFuture or the new structured concurrency stuff). But doing that or not is completely independent of using Nima, so I think here they're just showing some very simple code, because this is an example of using Nima, not an example of writing concurrent code in general.

Because concurrent calls are confusing and create bugs.

The point this train wreck of an example is trying to make (and failing) is that it's fine to write sequential blocking code in your request handlers. They can take as long as they want ... seconds, minutes, days .... because the handler threads are now an infinite resource as far as the JVM is concerned.

Reply on news.ycombinator.com