Loom comes from the same place of skepticism of async that you are expressing. Its goal is to make the imperative, blocking style as efficient as the async style while retaining familiar debugging tooling.
Personally, I'm more skeptical of imperative, blocking programming but Loom does seem like it can only be a good thing to have access to on the JVM platform.
If you're running on Linux, this is rather less the case. People see 1MB Xss values and assume that 1MB of 'real' memory is required to spawn a thread. In reality, on a standard linux system with overcommit enabled, stack pages are only consumed as they're written to - the rest is just virtual.
Possibly you're talking about something else, but I commonly see this raised as a concern about creating threads in Java.
1. Once stack memory is committed, it is never uncommitted, and creating a new thread is relatively expensive.
2. Committing virtual memory is done at a page granularity.
These make threads far too costly to serve as a construct for any single concurrent task.
Look, I'm looking forward to Loom as much as anyone. It is clearly going to be a big leap forward to the java ecosystem, and I'm looking forward to some threaded model being appropriate for any job rather than just most of them. It's just I see this terror of creating a thread or two out in the wider world and it seems a touch overwrought to me.
I don't think this is true if you create have approximately 1 thread for each CPU core (which is the ideal scenario in a non-async app - why would you create more threads than CPUs?).
A JVM thread is basically an OS thread. So on a 64-core server you'd have 64 threads, give or take. While a server can handle a few thousands threads quite seamlessly.
For workloads that involves a lot of IO. If you perform much more than 2-3 IO ops per request, a lot of time spent per request is simply waiting for IO response, meaning your CPUs are mostly idle.
One can allocate thread pools sensibly relative to CPU count and achieve ideal task<->CPU allocation.
So a modern choice would be either to have specialized background job processes, or microservices. Each such process would have manageable thread pools and a good CPU fit.
It also is a non-trivial fact that servers can perfectly have 128 cores or such, just like TB-sized memory isn't unheard of for JVMs. So that's another option for "less modern" apps: go the thread pool route, increase CPU count.
That makes the context switching problem worse. If you're running 100 services with 8 threads each on an 8 core CPU, that's got the same overhead problem as running 800 threads in a single service - potentially worse if the kernel has to do a more thorough context switch (e.g. TLB flush for spectre mitigation) when switching between separate processes. (And if you're going to solve that problem by running each service in its own container or VM then again that only makes the context switches heavier - fundamentally if you're running 800 threads on 8 physical cores then you've got to switch between them somehow).
> It also is a non-trivial fact that servers can perfectly have 128 cores or such, just like TB-sized memory isn't unheard of for JVMs. So that's another option for "less modern" apps: go the thread pool route, increase CPU count.
Well no shit if you just buy more CPU cores whenever you want to run more code then you don't need to worry about getting decent performance out of your hardware. But most of us are in businesses that can't afford one physical CPU core per IO-performing function in your codebase, which is what you'd need to sustain that approach (not to mention perfectly predicting how much load every single function needs). I guess when you add the 129th function to your codebase you then throw away all your super-expensive servers and buy even more expensive 256-core replacements? Or you forcibly split each service at the 128-function boundary so that you can deploy it as two microservices on two machines?
Quite obviously I meant running a microservice / job processor in a different "node" (aws instance, whatever).
It barely makes sense to let microservices compete with each other for CPU - one precisely seeks greater isolation.
> Well no shit if you just buy more CPU cores whenever you want to run more code then you don't need to worry about getting decent performance out of your hardware.
Large codebases relate strongly to large businesses that can afford hardware matching their scale.
> I guess when you add the 129th function to your codebase you then throw away all your super-expensive servers and buy even more expensive 256-core replacements?
Ok I regret spending time over your replies.
That's still just sweeping the problem under the carpet. Either you're getting a physical core for each function, in which case you're paying for lots of cores. Or you're multiplexing functions onto cores at some level (whether that's your own code, shared hosting, AWS lambda, or whatever) in which case someone (you or your service provider) is doing a bunch of context switches that have to be paid for somehow.
> It barely makes sense to let microservices compete with each other for CPU - one precisely seeks greater isolation.
For services big enough to need a full CPU (or a full core) all the time, you may as well give them a full CPU. But that implies a much coarser level of granularity than you've been suggesting. If your services really are "micro" then it makes a huge amount of sense to share a CPU between multiple services because most of them don't need a whole CPU the whole time.
> Large codebases relate strongly to large businesses that can afford hardware matching their scale.
The code grows faster than the hardware use. I've worked on codebases that were deployed on thousands of machines - but they had millions of lines of code and tens if not hundreds of thousands of functions. One of them even did take the approach of running multiple steps of the same logical computation on separate machines - but this was still done by having a yield point in the code and a scheduling framework that dispatched tasks onto grid compute machines, both because even though the code gets compiled into a state machine representation (i.e. one-way flows, de facto unstructured programming - the same as your pipeline approach) you don't want to have to maintain it in that form, and even if you did have code that looks like that, manually deciding how many cores to allocate to processing each state would not have been tractable.
> Ok I regret spending time over your replies.
Look, I honestly thought you were doing some kind of elaborate troll. What on earth kind of environment are you working in where you have more cores than code?
As far as using a fixed pool goes: others have replied similarly but by doing that you are completely letting your CPU cores go to waste whenever you are waiting for any underlying IO. For many classes of applications, they will end up topping out at only a few percent CPU usage.
Right, so then how do you structure your program - particularly an I/O-bound program - to use that small pool of threads effectively?
Imagine you're building a REST frontend that queries a couple of datastores and aggregates the results - you get a web request, fire off a request to some coordinator datastore, get a response back from that, fire off a bunch of requests to other datastores based on that, then form the results into some JSON and send them back to the client. If you do blocking requests then you waste most of your threads most of the time (they'll be idle waiting for responses from the datastores).
If you use NIO then that solves that problem, but you've still got to get each thread to actually do the right kind of processing - you want each thread to run an event loop where it checks for returned results from the datastores, matches those up with the requests that they belong to, and does the next step of processing for that request whether it's firing off more requests to other the datastores or composing together the results and sending them back to the original client. And you've got to also handle timing out stale requests etc.
Now you can write a server literally like that, with a global buffers for each possible request type that hold the state that you need to pick up handling that request again. But it's a nightmare to maintain, as you're effectively forced to write unstructured programs where you do a GOTO for each I/O operation - there's no easy way to trace through the handling of a single request or run part of your program for testing.
So people generally prefer to use an async framework that will handle keeping track of each suspended request and what to do next when the result comes back, and multiplex those onto the native threads for you. That could be something where you call an I/O API and pass a callback to run when the result comes back (and the framework takes care of running the callback on some thread when the result is ready, without blocking a thread in the meantime), or something where there's a special operator to say "suspend this function here, package up the rest of it as a continuation, and continue with that continuation when the I/O result comes back". Or, as Loom is doing, it could be something similar that works completely invisibly whenever the programmer calls particular functions.
There are tradeoffs to all these approaches to async, but any of them is going to perform a lot better than the naive approach of just using a native thread for each request, and they're all (hopefully) going to be more maintainable than manually writing an event loop that does the right thing each time.
I've hinted over the responses to my OP how I tend to structure the programs I design.
Other than that, arguing over specifics is somewhat besides the point.
It would be great to have a standardized benchmark/challenge so one could let the code speak for itself.
(I know C10k is a thing... not ideal, particularly because it makes concurrency a goal in itself, as opposed to a solution to a root problem)