Hacker News Reader

Comment by hn_throwaway_99 | Hacker News Reader

Really excited to see this! Seeing some of the early comments here I think folks may not realize how awesome this would be in the server space.

After all, a big reason that NodeJS won a lot of popularity on the server is that, for many types of common webserver workloads (i.e. lots of IO, relatively minor CPU usage), NodeJS can actually scale much better than Java with its thread-per-request model.

With these virtual threads, though, you could get the best of all possible worlds - a webserver that scales like NodeJS, but without some of the "CPU starvation" issues you can hit in Node if one executing request doesn't yield, and also without having to worry about "function coloring" like you do in Node with async vs. non-async functions.

Really, really fantastic development, have been waiting to see when this would come out.

View on HN

> For many types of common webserver workloads (i.e. lots of IO, relatively minor CPU usage), NodeJS can actually scale much better than Java with its thread-per-request model

Linux can handle a ginormous amount of threads quite well, would be interesting to see a deeper investigation to this theory.

Java uses some memory for stack, about 1MB for each thread.

This is configurable, and 1MB is very generous. I think the JVM automatically grows the stack size as needed nowadays and starts low.

This doesn't in practice limit scaling though as it's linear and small in absolute terms vs what you can put in a server.

The problem with doing it all native is that stack sizes are quite variable, especially in managed languages where modularity and code reuse works better, so it's common to have tons of libraries in a single project. The kernel won't object to lots of threads, but once those threads have been running for a while a lot of stack space will be paged in and used.

Loom solves this by moving stacks to and from the heap, where there's a compacting concurrent GC to clean up the unused space.

Am I understanding correctly that this is basically goroutines for Java?

Yes. I am not as familiar with the underlying implementation of goroutines, but this description in the linked JEP sounds exactly how I understand goroutines to work:

> The JDK implements virtual threads by storing their state, including the stack, on the Java heap. Virtual threads are scheduled by a scheduler in the Java class libraries, whose worker threads mount virtual threads on their backs when the virtual threads are executing, thus becoming their carriers. When a virtual thread parks -- say, when it blocks on some I/O operation or a java.util.concurrent synchronization construct -- it suspends, and the virtual thread's carrier is free to run any other task. When a virtual thread is unparked -- say, by an I/O operation completing -- it is submitted to the scheduler, which, when available, will mount and resume the virtual thread on some carrier thread, not necessarily the same one it ran on previously. In this way, when a virtual thread performs a blocking operation, instead of parking an OS thread, it is suspended by the JVM and another one scheduled in its place, all without blocking any OS threads (see the Limitations section).

Yes. But, since there are _two_ kinds of threads in Java (os and virtual), you still have to be very careful never to block a virtual thread. In Go/JavaScript/Beam, it doesn't matter because you literally can't block a thread (while idle). This is the kind of thing that's not terribly useful until nearly every library you interact with is using it as well.

Also, there's no new syntax, so you're stuck with all the same thread pool concurrency we've been using for decades.

EDIT: It looks like I'm wrong about this:

> My understanding is that you won't have to worry about blocking a virtual thread, because all IO APIs are being modified to park when executed in the context of a virtual thread.

My understanding is that you won't have to worry about blocking a virtual thread, because all IO APIs are being modified to park when executed in the context of a virtual thread.

That said, you'd still need to worry about unsafe code, like JNA/JNI or other such thing that could still block. And I'm not sure there will be a way to prevent long running CPU task from clogging up the virtual thread executor threads.

> My understanding is that you won't have to worry about blocking a virtual thread, because all IO APIs are being modified to park when executed in the context of a virtual thread.

And, from what I read in the original JEP, the underlying system thread pool (which all virtual threads float between as needed) will be expanded when a virtual thread gets pinned, so you don't have to worry about exhausting your pool. (If you pin too many threads, obviously you'll be consuming more OS resources than you may have expected, but that's a different problem.)

What do you mean by pin here? Do you mean that a blocking IO will block the thread, but it will also add one more thread to the virtual thread executor pool? So blocking won't starve your virtual threads?

That's right. From the linked JEP, under "Scheduler":

> Some blocking APIs temporarily pin the carrier thread, e.g.most file I/O operations. The implementations of these APIs will compensate for the pinning by temporarily expanding parallelism by means of the ForkJoinPool "managed blocker" mechanism. Consequentially, the number of carrier threads may temporarily exceed the number of available processors.

Just to clarify, though, most currently blocking IO operations will not pin the carrier thread, because most IO operations you make from a webserver are network calls (e.g. to another API or the database), and those network APIs have been modified to not pin. From just a bit further up in the JEP:

> The implementation of the networking APIs defined in the java.net and java.nio.channels API packages have been updated to work with virtual threads. An operation that blocks, e.g. establishing a network connection or reading from a socket, will release the underlying carrier thread to do other work.

If it's added to the VM then won't that allow languages like Kotlin to add support?

Kotlin coroutines could take advantage of virtual threads but they still will have the syntatic problem of colored functions.

Kotlin can call regular Java APIs, though. Doesn't have to take the coroutine route.

Yes but then your code isn’t idiomatic/multi platform/whatever. It’s a trade off (and one where I would always chose Java).

I learned an hard lesson in the Borland ecosystem.

Always go with the platforms languages, and the IDEs from the platform owners, even if others are more shinny.

Long term it always pays off to be the turtle, as the platforms move into directions not forseen by the shinny objects, and 3rd party IDEs keep playing catching up with SDK features.

What if the company that makes Kotlin is the one that makes the Java IDE?

Eclipse and NetBeans do exist, and... ehhhhh. I used NetBeans for a long time; couldn't stand Eclipse; and these days I only use IntelliJ. But the others absolutely exist, and it'd be hard to say that Apache and the Eclipse Foundation aren't deeply embedded in the Java ecosystem.

Eclipse tried their own language: https://www.eclipse.org/xtend/

Eclipse is fine. Especially from VsCode where it uses the Eclipse language server. It boots fast, and when you run it with a modern JVM and GC the memory usage is leagues lower than IntelliJ.

> Especially from VsCode where it uses the Eclipse language server.

Sure, but my particular complaint isn't with the functionality; it's with the UI. Yes, VS Code absolutely improves the experience.

They make one Java IDE, zero contributions to the JVM, and are all cozy with "screw you Java devs" Google godfather.

IBM does Java and the IDE (Eclipse).

Red-Hat and Microsoft do Java and the IDE (VSCode).

They contribute to the JDK, mostly via the Swing project. For instance they're a major contributor to Project Lanai.

I missed that. Most likely because they are a long way to reboot InteliJ on Compose for Desktop.

If you need multi-platform then coroutines is still your best bet. But many people don't use Kotlin in a multi-platform way, and lightweight threads will be an easier migration path (and more compatible with Java libraries if you cant avoid one) compared to coroutines.

In abstract, yes.

In the real Kotlin world of taking a random Kotlin library and call it from Java, most likely "it depends".

I didn't mention calling Java from Kotlin.

Kotlin can call a Java API to spawn a lightweight thread. There's no reason to use coroutines when you can do that.

Only if the Kotlin code is to be tied to the JVM, if you want that Kotlin library to be usable on Android, that isn't an option.

Yes, which is perhaps the best part about virtual threads. Java, Kotlin, Scala, Clojure, Gradle... everyone benefits.

Guest languages always have to deal with taking decisions that don't go along with the platforms, regardless how they boost being "better".

In Go one can block the native thread via using API that use blocking OS calls, like Linux file IO. In this case Go runtime allocates more native threads to run other language threads.

That's case for virtual threads also. It uses ForkJoinPool.ManagedBlocker to add additional threads.

"File I/O is problematic. Internally, the JDK uses buffered I/O for files, which always reports available bytes even when a read will block. On Linux, we plan to use io_uring for asynchronous file I/O, and in the meantime we’re using the ForkJoinPool.ManagedBlocker mechanism to smooth over blocking file I/O operations by adding more OS threads to the worker pool when a worker is blocked."

And you are incorrect on the other point as well: https://openjdk.java.net/jeps/8277129

Yeah, they seem very similar on the surface level.

Though loom doesn't have support for preempting green threads that are blocking the scheduler like go does, I think.

> NodeJS [...] with its thread-per-request model

Node.js doesn't create a thread per request; it's single-threaded with evented I/O. You can use node-cluster to start more than a single thread to saturate multi-core CPUs and load-balance HTTP requests across these, but that doesn't make it thread-per-request.

Suspect they're talking about Java - a lot of frameworks do exactly create a thread per request.

I can't think of any framework that still does one thread per request. Normally there is a a queue of incoming requests and they then get dispatched on a thread pool as threads return to the pool.

The challenge is normally that if any of the threads in the pool, as part of processing a request, needs to itself make an IO call, it will block. Ideally you'd want to park the request processing, return the thread to the pool, pick up the next request, until the IO is done where then on the next thread available from the pool you'd resume that request instead of picking another one. This is what the virtual threads will make really easy I think.

I think my high school English teacher would agree with you that the sentence is written awkwardly (I can see the 'awk' note, in red, on my paper right now :) ). Here's how I parsed it:

> a big reason that NodeJS won a lot of popularity on the server is that, for many types of common webserver workloads [...], NodeJS can actually scale much better than Java with ~~it's~~ [Java's] thread-per-request model.

Ah that makes sense; didn't get this reading at all!

> ~~it's~~

Why are you calling that out? The original "its" was correct without the apostrophe.

Tildes are used for strikethrough in some markup dialects (including markdown), so I think they meant to depict replacing "its" with "Java's".

No clue on the apostrophe.

(Forgive my grammar nazism.) The possessive form of "it" is "its": "The dog wagged its tail". But for basically everything other than pronouns and plurals, the possessive form involves adding "apostrophe s". In recent years, many people have tried to apply this rule to "it". But the problem is that "it's" is understood to be a contraction of "it is" or "it has"; furthermore, "its" already exists as the standard possessive form.

One thing I say to people using "it's" is that by analogy, you also need to say: "He got he's skills. She missed she's ride. They have they's meeting."

Thank you a ton for posting this! I've been doing this for most/all of my life and it didn't really make sense till now. I've had people explain it before but it didn't really make sense. Here's what I got from what you wrote (please correct me if this is wrong / kinda off in some way)

For most words, the possessive form is "<word>'s"

For pronouns (including it) there are different rules. He becomes his, she goes to hers, it goes to its.

Also, words that already end in s don't get the " 's " treatment.

(Question - for words that end in "s", we put the apostrophe after the existing, ending 's', yes?)

Thanks again for posting this - viewing the possessive form of it as (yet another English language) exception to the normal rule of " 's " is really helpful.

> One thing I say to people using "it's" is that by analogy, you also need to say: "He got he's skills. She missed she's ride. They have they's meeting."

This is a great distillation of the intuition I've always had, but never quite verbalized.

Twisol was right - I was trying to imply strikethrough using the Markdown syntax in an attempt to depict the idea of replacing "its" with "Java's". It didn't work as well as I hoped. In my mind I can see more 'Awk' scribbles on my post, and looking at it I agree :)

Adding in the 's is 100% my mistake. I've been guilty of using "it's" as the possessive form for most of my life, but that changes today! :)

> but that changes today! :)

Exciting! :)

Sorry, yes, my sentence was poorly written with the ambiguous antecedent. Most Java webservers use a thread-per-request model, which is why Node can usually scale to more concurrent requests.

>you could get the best of all possible worlds

Maybe not _all_ possible worlds. You still have original Threads for things that need an actual OS thread. Its not a solution for UI threading.

There will be code that needs a native thread or non-preemptive threading and shouldn't be run on a virtual thread. In that sense there is method coloring but its yet to be seen how common a problem that will be.

Library writers and frameworks will need to sort out patterns for how to call Runnables in a safe way.

Still, its a nice tool to have.

That’s not function coloring, it is up to the caller whether to start it in a virt thread or a real one. Function coloring is having two methods do the same thing differing only in name and signature (eg. there is a blocking sleep and a non-blocking one).

>it is up to the caller whether to start it in a virt thread or a real one.

Sorta kinda but not when you're working in a framework that will call your code or working in some library where the abstracted code is non-obvious or uneasy to configure.

Maybe its not function coloring, although I wouldn't know what else to call it and I think its quite similar. What would you call the problem?

Yes but you always need original/kernel threads, regardless of what approach to async you need. The concept of a thread and a stack is hard-wired into the CPU.

W.R.T. code that needs a native thread: at the moment there's only two types of such code. One is code that uses Java's synchronized statement. That's supposedly just a, ehm, small matter of programming to fix. The other is calling into non-JVM controlled code. That's fundamental and no approach to scalable concurrency can fix it, not CPS/async/await or anything else because it's a foreign compiler.

But fortunately the JVM has some really interesting tricks up its sleeve there. For instance you can compile your native code using LLVM and then execute the bitcode on the JVM. Well, OK, currently GraalVM doesn't support Loom but hopefully Graal will be upgraded to do so as Loom gets integrated into HotSpot. And when it does, you will be able to call into code written in C/C++/Objective-C/Rust as long as that code can be recompiled with your own toolchain and as long as you can tolerate it being JITCd, also whilst benefiting from Loom's scalability.

Why do you say CPS can't fix it? C# works around this by having a synchronization context and ways to bounce around contexts. In this way C# async/await is able to ensure code is run on a specific native thread. Is that not a fix?

Java still eats like 5x ram compared to node, Java tools are slow, Java frameworks are gargantuan, even those claiming "lean".

Java is fine if you don't care about RAM and start time, though.

https://quarkus.io/guides/building-native-image

You should try Quarkus. It is a production framework built by Redhat. It uses Java-GraalVM under the cover to compile your entire webapp to an executable (like golang does).

It's just as fast.

Java is the highest performance and most tuned VM there is. I think you're really thinking of java from a long time ago, if ur thinking this

Ah that Supersonic-Subatomic-Java. Whatever Redhat lacks in quality side in Java frameworks, they more than compensate with corny marketing taglines.

> Java is the highest performance and most tuned VM there is.

Not defending the opposite argument, but V8 is also pretty impressive. It's rooted in work done for Smalltalk long before JavaScript was a thing.

> It's rooted in work done for Smalltalk long before JavaScript was a thing.

The same is true of HotSpot. https://en.wikipedia.org/wiki/HotSpot_(virtual_machine)#Hist...

If you use gargantuan Java frameworks, you'll use a lot of RAM. Just don't do that. With Spring Boot and similar frameworks, the RAM usage is really just very modest. I'll give you startup times, since I am not a believer in Quarkus and Graal. And I wouldn't use Java for a serverless function that needs to spin up and respond quickly. But for a typical (blue/green-deployed) application in my world, startup time is still only a few seconds, which is fine for many applications. And I am not settling for "fine", just saying that the startup time isn't a big consideration, against a lot of things the Java (or Spring, in my case) ecosystem offers me.

Startup time is fine.. just don't use spring where it has to read every class at runtime to determine what to inject.

Well that's the Spring's great feature: To convert as many compile time errors in to runtime errors as possible.

To be fair there is the new Kotlin based wiring API which avoids that with the caveat you need to instantiate everything manually etc. Which is probably a decent tradeoff for some folks.

There’s other alternatives like Avaje Inject and Quarkus which uses the same annotations but does the injection generation at compile time.

Care to share more thoughts on Quarkus? I'm evaluating it for an upcoming app. From my limited reading it can be used with and without Graal.

My favourite bit of Quarkus is the same as my favourite bit of Micronaut - DI is compile time.

See my sibling comment: my fears are irrational.

I had a Quarkus server app start up in 0.1s the other week. And BTW Spring Boot does native compilation now too [1]

Your "belief" is putting you at risk of ignoring a wide range of Java use cases unnecessarily.

[1] https://docs.spring.io/spring-native/docs/current/reference/...

You are probably right. It's one of those things where I've not seen a need to jump aboard. I'm still being fearful of reflection going to break on me. Probably irrational fear, but fed by me not understanding how it wouldn't break. Which I should study up on. Which I don't, since I don't have the need. And here I am ... vicious circle.

Javalin is _very_ lightweight, and starts up fast. Use the framework that best suits your requirements.

Also, Java is working on reducing ram usage: https://openjdk.java.net/projects/lilliput/

there's Helidon: https://helidon.io/ as well from Oracle. though at the moment, I'm using Javalin.

I thought so too. And wrote simple web service using Helidon SE. It eats 300+ MB of RAM. I spend some time trying to optimize GC and all that stuff. Similar node service would eat 30 MB of RAM.

May be Graal would save us all. Until then Java is beyond salvation.

Other than a very very niche usecase, I really don’t see how eating 300 MB of RAM is so problematic when we quite literally have servers with terabytes of RAM. Yeah java can be configured to run GC all the time and target <100M of ram, but it rather runs the GC only seldom (jvm is actually one of the most energy efficient runtimed languages out there!) and trades memory usage to throughput.

Because in the cloud you're paying hefty price for every MB of RAM. For example with Jelastic you have 128 MB per cloudlet. And it's 2x difference between 120 MB and 130 MB. And with dedicated servers I don't have terabytes of RAM, I have two server with 24 GB each.

And no, you can't configure Java to target <100 MB of RAM. I configured it with -Xmx64m and it still eats around 300 MB. Java just fat and you can't do nothing about it at this time.

Or you care about actual performance of the system?

Reply on news.ycombinator.com