Hacker News Reader

Comment by pron | Hacker News Reader

It is true that the experiment exercises the OS, but that's only part of the point. The other part is that it uses a simple, blocking, thread-per-request model with Java 1.0 networking APIs. So this is "achieving 5M persistent connections with (essentially) 26-year-old code that's fully debuggable and observable by the platform." This stresses both the OS and the Java runtime.

So while you could achieve 5M in other ways, those ways would not only be more complex, but also not really observable/debuggable by Java platform tools.

View on HN

This.

Writing the sort of applications that I get involved with, it's frequently the case whilst it's true that 1 OS thread/java thread was a theoretical scalability limitation - in practice we were never likely to hit it (and there was always the 'get a bigger computer').

But: the complexity mavens inside our company and projects we rely upon get bitten by an obsessive need to chase 'scalability' /at all costs/. Which is fine, but the downside to that is the negative consequences of coloured functions comes into play. We end up suffering having to deal with vert.x or kotlin or whatever flavour-of-the-month solution is that is /inherently/ harder to reason about than a linear piece of code. If you're in a c# project, the you get a library that's async, and boom, game over.

If loom gets even within performance shouting distance of those other models, it's ought to kill (for all but the edgiest of edge-cases) reactive programming in the java space dead. You might be able to make a case - obviously depending on your use cases which are not mine - that extracting, say, 50% more scalability is worth the downsides. If that number is, say, 5%, then for the vast majority of projects the answer is going to be 'no'.

I say 'ought to', as I fear the adage that "developers love complexity the way moths love flames - and often with the same results". I see both engineers and projects (Hibernate and keycloak, IIRC) have a great deal of themselves invested in their Rx position, and I already sense that they're not going to give it up without a fight.

So: the headline number is less important than "for virtually everyone you will no longer have to trade simplicity for scalability". I can't wait!

A couple of points to consider.

1. Demanding scalability for inappropriate projects and at any cost is something I've seen too, and on investigation it was usually related to former battle scars. A software system that stops scaling at the wrong time can be horrific for the business. Some of them never recover, the canonical example being MySpace, but I've heard of other examples that were less public. In finance entire multi-year IT projects by huge teams have failed and had to be scrapped because they didn't scale to even current business needs, let alone future needs. Emergency projects to make something "scale" because new customers have been on-boarded, or business requirements changed, are the sort of thing nobody wants to get caught up in. Over time these people graduate into senior management where they become architects who react to those bad experiences by insisting on making scalability a checkbox to tick.

Of course there's also trying to make easy projects more challenging, resume-driven development etc too. It's not just that. But that's one way it can happen.

2. Rx type models aren't just about the cost of threads. An abstraction over a stream of events is useful in many contexts, for example, single-threaded GUIs.

I think my point is more that you end up having to pay the costs (of Rx-style APIs) whether you need the scalability or not, because the libraries end up going down that route. This has sometimes felt that I'm being forced to do work in order to satisfy the fringe needs of some other project!

And sure, if you are living in a single-threaded environment, your choices are somewhat limited. I, personally, dislike front-end programming for exactly that reason - things like RxJS feel hideously overcomplicated to me. My guess is that most, though not all, will much prefer the loom-style threading over async/await given free choice.

One additional - as noted, it's been 26 years since Java's founding. Project Loom has been around since at least 2018 and still has no release date. It'll be cool for Java projects whenever it comes out, but I just...have a hard time caring right now. I can't use it for old codebases currently, and new codebases I'm not using one request per Java thread anyway (tbh - when it's my choice I'm not choosing the JVM at all). The space has moved, and continues to move. In no way to say the JVM shouldn't be adopting the good ideas that come along the way, that is one of the benefits of being as conservative and glacial in adoption as it is, but I just...don't get excited about them, or find myself in any position in relation to the JVM (Java specifically, but the fundamentals affect other languages) other than "ugh, this again".

> and still has no release date

JEP 425 has been proposed to target JDK 19, out September 20. It will first be a "Preview" feature, which means supported but subject to change, and if all goes well would normally be out of Preview two releases, i.e. one year, after that.

> I'm not using one request per Java thread anyway

You don't have to, but not that only the thread-per-request model offers you world-class observability/debuggability.

> other than "ugh, this again".

Ok, although in 2022, the Java platform is still among the most technologically advanced, state-of-the art, software plarform out there. It stands shoulder to shoulder with clang and V8 on compilation, and beats everything else on GC and low-overhead observability (yes, even eBPF).

> I'm not using one request per Java thread anyway

The point is with Loom you can, and you can stop putting everything into a continuation and go back to straight-line code.

>> The point is with Loom you can

The point I was making is that Loom isn't released, stable, production ready, supported, etc, and there's no still no date when it's supposed to be, so what you can do with Loom in no way affects what I can do with a production codebase, either new or legacy. I'm not sure how you missed that from my post.

I'm not defending reactive programming on the JVM. I'm also not defending threads as units of concurrency. I'm saying I can get the benefits of Project Loom -right now-, in production ready languages/libraries, outside of the JVM, and I can't reasonably pick Project Loom if I want something stable and supported by its creators.

> and there's no still no date when it's supposed to be

September 20 (in Preview)

> I'm saying I can get the benefits of Project Loom -right now-, in production ready languages/libraries, outside of the JVM

Only sort-of. The only languages offering something similar in terms of programming model are Erlang (/Elixir) and Go — both inspired virtual threads. But Erlang doesn't offer similar performance, and Go doesn't offer similar observbility. Neither offers the same popularity.

I'm not saying there aren't tradeoffs, just that if I need the benefits of virtual threads...I have other options. I'm all for this landing on the JVM, mainly so that non-Java languages there can take advantage of it rather than the hoops they currently have to jump through to offer a saner concurrency model, but that until it does...don't care. And last I saw this feature is proposed to land in preview in JDK19; not that it would, and...it's still preview. Meaning the soonest we can expect to see this safely available to production code is next year (preview in Java is a bit weird, admittedly. "This is not experimental but we can change any part of it or remove it for future versions depending how things go" was basically my take on it when I looked in the past).

Meanwhile, as you say, Erlang/Elixir gives me this model with 35+ years of history behind it (and no libraries/frameworks in use trying to provide me a leaky abstraction of something 'better'), better observability than the JVM, a safer memory model for concurrent code, a better model for reliability, with the main issue being the CPU hit (less of a concern for IO bound workloads, which is where this kind of concurrency is generally impactful anyway). Go has reduced observability than Java, sure, but a number of other tradeoffs I personally prefer (not least of all because in most of the Java shops I was in, I was the one most familiar with profiling and debugging Java. The tools are there, the experience amongst the average Java developer isn't), and will also be releasing twice between now and next year.

Again, I'm not saying virtual threads from Loom aren't cool (in fact, I said they were; the technical achievement of making it a drop in replacement is itself incredible), or that it wouldn't be useful when it releases for those choosing Java, stuck with Java due to legacy reasons, or using a JVM language that is now able to migrate to take advantage of this to remove some of the impedance mismatch between their concurrency model(s) and Java's threading and the resulting caveats. Just that I don't care until it does (because I've been hearing about it for the past 4 years), it still doesn't put it on par with the models other languages have adopted (memory model matters to me quite a bit since I tend to care about correct behavior under load more than raw performance numbers; that said, of course, nothing is preventing people from adopting safer practices there...just like nothing has been in years previous. They just...haven't), nor do I care about the claims people make about it displacing X, Y, or Z. It probably will for new code! Whenever it gets fully supported in production. But there's still all that legacy code written over the past two decades using libraries and frameworks built to work around Java's initial 1:1 threading model, and which simply due to calling conventions and architecture (i.e., reactive and etc) would have to be rewritten, which probably won't happen due to the reality of production projects, even if there were clear gains in doing so (which as the great-grandparent mentions, is not nearly so clearcut).

Erlang is very cool, and Go has certainly achieved a notable measure of popularity — both served as an inspiration here — but a super-popular language like Java plays in a different world and a completely different scale than languages with a smaller reach. Virtual threads will bring the benefits of lightweight user-mode threads to an audience that is many times that of Erlang and Go combined.

As to legacy code, Java programs have been using the thread-per-request model for over 25 years (there's been a lot of talk of reactive, but actual adoption is relatively low), and Java's threads were designed to be abstracted from day one (in fact, early versions of Java implemented them in user mode). So the right fit has been there all along. Migrating applications to use virtual threads requires relatively few changes because of those reasons, and because we designed them with easy adoption in mind. This particular experiment is about simple, "legacy" Java 1.0 code enjoying terrific scalability.

BTW, Java's observability has come a long way in recent years (largely thanks to JFR — Java Flight Recorder), and even Erlang's is no match for it, although Java still lags behind Erlang's hot-swapping capabilities.

[1]: BTW, I always find talk about the "average Java programmer" a bit out of touch. The top 1% of Java programmers, the experts, outnumber all Rust (or Haskell, or Erlang) programmers several times over, and there are many more reliable Java programs than reliable Erlang programs. The average Java (or Python, or JavaScript, the two other dominant languages these days) programmer, is just the average programmer, period.

And hopefully we can bury Reactor Core in the garden and never talk about it again

... at which point we can also "undeprecate" RestTemplate and pretend that never happened either :-)

What has the space move to?

Threads (whether lightweight or heavyweight) can’t fully replace reactive/proactive/async programming even ignoring performance and scalability. Sometimes network code simply needs to wait for more than one event as a matter of functionality. For example, a program might need to handle the availability of outgoing buffer space and also handle the availability of incoming data. And it might also need to handle completion of a database query or incoming data on a separate connection. Sure, using extra threads might do it, but it’s awkward.

> Sure, using extra threads might do it, but it’s awkward.

It's simpler and nicer, actually — and definitely offers better tooling and observability — especially with structured concurrency: https://download.java.net/java/early_access/loom/docs/api/jd...

Let me preface by saying I am a Johnny-come-lately loom fanboy. Amazing work and huge impact. Re structured concurrency: I wonder if there’s any way to combine with generic exceptions such that we can not force a wrapping exception class. So maybe have an executor class that’s generic on the thrown exception type, and then have the join or get apis explicitly throw that type? This thought process is inspired by the goto-considered-harmful trail of logic: I think it would get us even closer to concurrency encapsulated in function blocks.

Convenient polymorphism over exceptions is something I would very much like to see in Java, but it's a separate topic. Given that structured concurrency is normally used with things that can fail, and whose failures must be handled, I hope (and think) you'll find that the use of checked exceptions is not onerous at all. If we're mistaken, we can consider solutions during the incubation period.

Totally agree that we need explicit handling of the concurrency-specific exceptions like interruptedexception. It’s just that concurrent apis by their nature take callable/runnable apis which lose any formality over exceptions thrown by client code, and thus someone up the stack is always forced to write a catch( Throwable ) block. So the concurrency leaks up the stack, and forces unsafe default clauses. You’re clearly correct that the topic is separate, but it has great impact on the leakiness of these apis.

Thanks for the response and the amazing work!

I think we're in agreement. Ignoring under the hood - Loom's programming paradigm (from the viewpoint of control flow) is the Threading programming paradigm. (Virtual)Thread-per-connection programming is easier and far more intuitive than asynchronous (i.e. callback-esque) programming.

I still attest though - The 5M connections in this example is still a red herring.

Can we get to 6M? Can we get to 10M? Is that a question for Loom or Java's asynchronous IO system? No - it's a question for the operating system.

Loom and Java NIO can handle probably a billion connections as programmed. Java Threads cannot - although that too is a broken statement. "Linux Threads cannot" is the real statement. You can't have that many for resource reasons. Java Threads are just a thin abstraction on top of that.

Linux out of the box can't do 5M connections (last I checked). It takes Linux tuning artistry to get it there.

Don't get me wrong - I think Loom is cool. It's attempted to do the same thing as Async/Await tried - just better. But it is most definitely not the only way to achieve 5MM connections with Java or anything else. Possibly however, it's the most friendly and intuitive way to do it.

*We typically vilify Java Threads for the Ram they consume. Something like 1M per thread or something (tunable). Loom must still use "some" ram per connection although surely far far less (and of course Linux must use some amount of kernel ram per connection too).

I don't quite follow your argument.

Saying "Linux cannot handle 5M connections with one thread per connection" isn't a reasonable statement because no operating system can do that, they can't even get close. The resource usage of a kernel thread is defined by pretty fundamental limits in operating system architecture, namely, that the kernel doesn't know anything about the software using the thread. Any general purpose kernel will be unable to provision userspace with that many threads without consuming infeasible quantities of RAM.

The reason JVM virtual threads can do this is because the JVM has deep control and understanding of the stack and the heap (it compiled all the code). The reason Loom scalability gets worse if you call into native code is that then you're back to not controlling the stack.

Getting to 10M is therefore very much a question for the JVM as well as the operating system. It'll be heavily affected by GC performance with huge heaps, which luckily modern G1 excels at, it'll be affected by the performance of the JVM's userspace schedulers (ForkJoinPool etc), it'll be affected by the JVM's internal book-keeping logic and many other things. It stresses every level of the stack.

> But it is most definitely not the only way to achieve 5MM connections with Java or anything else. Possibly however, it's the most friendly and intuitive way to do it.

It is the only way to achieve that many connections with Java in a way that's debuggable and observable by the platform and its tools, regardless of its intuitiveness or friendliness to human programmers. It's important to understand that this is an objective technical difference, and one of the cornerstones of the project. Computations that are composed in the asynchronous style are invisible to the runtime. Your server could be overloaded with I/O, and yet your profile will show idle thread pools.

Virtual threads don't just allow you to write something you could do anyway in some other way. They actually do work that has simply been impossible so far at that scale: they allow the runtime and its tools to understand how your program is composed and observe it at runtime in a meaningful and helpful way.

One of the main reasons so many companies turn to Java for their most important server-side applications is that it offers unmatched observability into what the program is doing (at least among other languages/platforms with similar performance). But that ability was missing for high-scale concurrency. Virtual threads add it to the platform.

Reply on news.ycombinator.com