Sub-10 ms Latency in Java: Concurrent GC with Green Threads
jet-start.sh
jet-start.sh
Who was it that turned GC off entirely, minimized allocation and just restarted their VMs when they ran out of RAM every couple of hours, was that Netflix?
Either way. It makes me excited for Rust and the languages it'll inspire, all this labor gone away.
OTOH, if you think other languages let you do away with a GC without pretty significant extra work, especially in concurrent systems, well, then you haven't had experience with those languages.
This is still a lot of work for a video game (because you never want any latency, the only way to achieve this is with an arena allocator or going full @nogc).
But for apps where the latency requirement is bounded, D doesn't make it hard and the language is nice and ergonomic.
Isn't that how most GCs work?
Why would you do anything else? To release memory to the OS? That's not really a priority in most runtime systems, and I think wanting to do that is a pretty niche requirement.
You can definitely write useful Java applications where nothing gets heap allocated, through using existing objects, and through using scalar-replacement-of-aggregates.
I guess it depends whether GCs are always scheduled in an allocation or can be triggered another way. Either way that should be easy to disable.
I read somewhere that D doesn't have write barriers, so I would assume they have a hard time implementing more advanced GC features like generational collection or concurrent marking. It's not suprising that the GCs in the JVM achieve much better pause time.
I'd say the opposite is true. Relying on GC requires significant extra work. Because you always need to think about memory (exception: small script-like applications). The only thing a GC does is that it enable you to not think about it, but the moment you don't you will write bad code and realize it was a disservice all along. And by then it is too late.
So in a GC language you need to constantly be aware of when you take something for granted. Which is more work than just doing it manually yourself.
generally you design your application the way you normally would, do some testing, and then go back through the hot path and make some adjustments.
Second, while there have been similar claims made in the past, they always apply for a certain target throughput and latency. GCs are constantly making great strides in that regard. Next year, ZGC will have a <1ms worst-case latency for pretty serious allocation rates, and with an acceptable hit to throughput. As you can see in this post and previous ones, G1 offers great throughput with acceptable latencies.
Either it moves it into PagerDuty, like G1, or it moves it into your GCP bill like Shenandoa and ZGC.
The trade-off has always been latency-throughput-footprint, nothing I've seen yet has changed that. The innovation in Rust is realizing you can do all the tracing work at compile time.
> The innovation in Rust is realizing you can do all the tracing work at compile time.
This is spectacularly false. For all non-trivial allocation/deallocation patterns, Rust also uses a runtime reference-counting GC, which is significantly slower than the tracing GCs you find in OpenJDK. The benefit comes from not relying on it too much, but this comes at a considerable cost of lost abstraction, which means more costly maintenance over the years. This is the same for all low-level languages; the difference Rust brings is that it (conservatively!) checks for memory access errors.
Another difference is that most people who talk about Rust haven't actually written a significant application in it and had to maintain it for years. I'm not saying it's impossible -- people do this for C and C++, which make a similar tradeoff in this regard -- but it does come at a substantial cost.
I used to be a hard-core zealot for the JVM as the performance platform of the future - and did talks arguing just as you are here why HotSpot outperforms $LANGUAGE in real applications. I feel like I'm hearing myself in your argument..
I wrote a signficant portion of the Neo4j storage engine, which is in Java. Now I'm writing another database engine in Rust (sidenote: not for replacing the Neo4j engine, just because it's interesting). Arguably database engines qualify as "significant applications".
I find - subjectively:
- Maintaining performant code in Rust is easier. I do the same patterns as I did in Java, except it doesn't rely on easy-to-break assumptions of how HotSpot happens to work (ex: stack allocations)
- Like you said elsewhere, the issue is often about fragmentation, ultimately stemming from object churn. I find that Rust makes it, culturally perhaps, easier to maintain low allocation code than Java. (sidenote: I think these two points is also why Go code generally has "better GC behavior"; Go doesn't have a better GC, it has a language that encourages less heap allocation)
- The engine I'm writing in Rust is faster than anything I've written in Java and - critically - runs for days and months without notable stalls.
Come to the dark side!
The JVM works exceptionally well when you are dealing with long lived apps which deal with a lot of allocations and the machine running it has a substantial amount of memory. Things like webservers, for example, are near perfect fits for the JVM.
Rust does really well when you need high performance, low memory, and ultra fast startup times. It won't necessarily outperform the JVM when you talk about doing a lot of heap allocations (due to heap fragmentation) and it unfortunately suffers from the same heap fragmentation if you are dealing with a long lived server that does a lot of allocations. But then, maybe that performance loss is acceptable for the ability to very quickly scale up and down servers.
Now, the JVM is making great strides towards getting faster startup times and even fast performance at startup (AppCDS). However, those strides often involve trade offs with either build complexity or performance losses (such as Graal's AOT). The benefit for rust is that it is as fast as it ever will be without any special build steps or tweaks.
Oh, and let's not forget diagnostics. Flight recorder for the JVM is simply AMAZING. The ability to hook up to a poorly behaving production server, start flight recording, and getting detailed information about things like "where are allocations happening" or "what are the hot methods" is simply amazing. No other platform that I know of has the level of detail you can get right out of the box with flight recorder. Certainly not without restarting the application with additional configuration. For example, you'd need a special build of rust with profiling turned on to even start to get the same level of info, doing such also significantly negatively impacts performance.
What is happening now is that the free beer Java users are also getting those features on the package.
Are you talking about `Rc` and similar smart pointer types? If so, the twist is that in Rust almost all allocations in Rust are trivial in this sense.
Lest anyone read this and think it's true, it's not. Using Rc is a design choice, and not one that is a given. I have written tens of thousands of lines of Rust code doing very heavy data processing and used Rc only a handful of times. In fact, I find using Rc without a very good reason to usually be a bad idea that enables lazy thinking.
This is not really a good comparison because you can't write the same code in either language. In Java practically everything gets allocated on the heap barring some optimizations. Meanwhile Rust programs can selectively allocate memory on the stack when it makes sense to do so. Reference counting is is just one of many different allocation strategies available to Rust. It is not the first tool you grab when you want to allocate memory in Rust, therefore absolute throughput of reference counting might not be as relevant in Rust as the absolute performance of the GC in Java.
In our benchmarks we never saw a GC pause of more than 2 ms on either ZGC or Shenandoah, but the end-to-end latency, the one the user cares about, is impacted by much more than a single GC pause. Sometimes there would be several pauses in a rapid sequence, or just the background GC thread would do too much work at once.
Even after dedicating a core or two to the GC, you still face the issues of cache pollution and RAM throughput stealing that heap walking incurs.
You say that, but there seem to be no end to the stories of people spending enormous amount of time fighting the gc. It is not difficult to avoid heap allocations in other languages as well as freeing them deterministically.
Second, as this blog post series shows, the "fight" is not what it used to be. You don't really need to control allocation any more until your rates are really high. Java's GCs have just gotten so much better in JDK 14 and beyond.
I'm extremely skeptical about that. My experience is that with modern C++ you lose very little elegance and gain a huge amount of control by giving up a garbage collector. Memory management becomes a very minor problem. The vast majority of memory allocations are avoided and those that need to be there can be done ahead of time.
I have never heard anyone writing a latency sensitive program in C++ (games, trading etc.) say that their life would be easier if they were using a gc or that they wished they could do it in java. I have however seen decades of people talking about all the extreme lengths and rabbit holes they go down to deal with the java gc.
From a broader perspective, pretty much any language with a gc ends up having a perpetual conversation around how the next gc will solve the problems with the current gc. You can see it in java, go, julia, and D. The only one I never hear about is LuaJIT, but maybe I just haven't seen it or maybe the expectations are lower.
Do you have any references handy on the julia bit there? I actually have seen very little conversation in the julia community about replacing or upgrading the GC. Mostly just the ocassional post from an inexperienced user who thinks that a borrow checker would be a good fit for julia.
Discussion seems to almost always revolve around showing users who need it, how to manually manage memory when necessary by pre-allocating arrays, using in-place operations or writing stack allocated code so that they avoid the GC in performance critical code.
I've never seriously used a language without a GC, but my feeling in Julia has always been that I never really had gripes about the GC because it's so easy to avoid the GC and take memory management into my own hands.
One complaint I will say I've heard though is that while these people find they can get the allocation behavior they want (i.e. none), some of them would like semantic guarentees that they will not hit allocations, rather than needing to test and make sure their code doesn't start hitting the GC if they switch Julia versions. Someday we might be able to provide such guarantees, but for now GC behavior is just an implementation detail that can change across minor versions.
I can sympathize with those who find that uncomfortable for sure, but in practice, Julia versions have been consistently better at getting more automatic stack allocations, not less, so it hasn't really been a problem.
The cost of maintaining a large C++ application (>1MLOC) with a large team over years is very significantly higher than a similar Java application. In some cases the footprint and/or performance benefits are worth that extra cost, but in the vast majority of cases they're not.
> I have however seen decades of people talking about all the extreme lengths and rabbit holes they go down to deal with the java gc.
Again, 1. Java's GCs changed dramatically in the last two years -- the GCs described in the post, are brand new/recently revised and 2. that's because that's Java's particular rabbit hole. C++'s rabbit holes, from undefined behaviour, through partial evaluation with templates and constexprs, to compilation times and sheer language complexity are far deeper.
> I have never heard anyone writing a latency sensitive program in C++ (games, trading etc.) say that their life would be easier if they were using a gc or that they wished they could do it in java.
Their lives would be easier if they could do it in Java, but sometimes they can't. I think that the changes in the last couple of years and the upcoming changes in the next few years will make Java more appropriate even in domains where it hasn't been used before, but it's fine if not. Its market reach is so huge as it is. But games are not often maintained for many years, and telemetry isn't that important, so Java's benefits are not as big as for servers.
I am very skeptical of this, I don't know why it would be the case. My experience is that with modern C++ and avoiding inheritance programs end up much more direct and clear since a type isn't fragmented into multiple classes and base objects don't need to be used for generic programming and data structures.
> C++'s rabbit holes, from undefined behaviour, through partial evaluation with templates and constexprs, to compilation times and sheer language complexity are far deeper.
This seems like what would be said by someone who has just read a few comments on C++ here and there but not actually used it for non-trivial projects. These are rarely issues. I don't know what 'partial evaluation with templates' means and constexprs didn't even exist until recently. Compilation times do seem to be a big problem, mostly because many projects don't do anything with their structure to mitigate them.
Rust or other upcoming or future languages might change that.
† The threshold varies between VM and GC, but usually <10ms is easily achievable
10µs ? You can't even afford that many function calls.
Seriously, 10µs is not a sensible target for a general purpose OS, maybe not even for a general purpose CPU. Achievable? Perhaps. But sensible? Not really.
I suppose that if you're working on small amounts of data every time your code executes, then this becomes vastly more reasonable.
But beyond that, the actual code you write is fairly natural for those languages. You can't allocate, and your code and data have to fit in the cache. But you can use normal language constructs, and most of the standard library - neither of which is true for Java.
The memory work remains the same, you can do it yourself or let the GC handle it. For 99% of applications, the GCs are good enough and getting better every year, but low-latency still needs predictability and would ideally choose manual memory management.
The realities of the job market and IT deployments are different though and that's why we still have JVMs involved with low-latency scenarios because of talent, tooling and productivity.
Steve lays it out far better than I could, coining the term "static garbage collection": https://steveklabnik.com/writing/borrow-checking-escape-anal....
And to your "without pretty significant extra work" qualifier: I really don't find that to be true with Rust. The initial learning curve was a bit rough, but certainly far less so than other languages/platforms I've picked up (hello, ML). In the end, I find that it's just a nice, helpful, productive, and ludicrously performant language.
Since the automatic deallocation in rust, per default, happens at the point where the value goes out of scope, if a resource destruction could block, or perform expensive operations on destruction, it can not be allowed to go out of scope in a latency sensitive thread.
Usually not that much of an issue unless you are chasing really low latencies.
But things can still happen in rust that catch you off guard. Say a value goes out of scope, and because of reasons, it does so with a destructor doing a logging call, which has accidentally become blocking on a socket send call, which it did because nobody realized that the AWS/GCP/whatever logger adaptor actually didn't perform all IO in a thread with which was only communicated with locklessly, which nobody noticed before because it was only if a buffer was full, which only happened today because ....
Not a big deal, it's almost all the same things which mess upp latencies in C++ code. And that's the thing. It's not necessarily easier to get low latency in rust than in C++, but the work required for hitting a quality/performance/latency target in rust is probably still lower than for C++. Unless you are lucky to have a very mature C++ low-latency stack, together with all the utility functionality you need, which seems to be exceedingly rare. Is the work required lower than Java on a custom/tuned JVM?
We'll have to wait and see. It probably is, but it's a complex balance between access to utilities, language complexity, and several more parameters which ultimately decide which platform provides the best environment for low latency code, especially if the complexity is non trivial.
The thing is that most of what you said about Rust is true of C++ as well. It really isn't hard to write a good fast program in C++ once you know it. The problem -- as those of us, like me, who have been writing in low-level languages for a couple of decades now know -- is that low-level programs in any low level language are necessarily rigid. Changing something in one place often has a much bigger impact on the codebase than in a high level language, making the overall cost much higher.
Low level languages have their place, but they won't replace high-level ones for "ordinary" application development. People will only pay the price when that extra 5% is important or when running in a constrained environment. This is the same equation that's been around for twenty years and there's no sign it is changing.
Otherwise, if you wrote your image processor in C# or Java, it becomes hard to call your library from Python or Node because you have to require the entire VM. Likewise, you can ship an application binary that has no requirements for a runtime. (Your application binary doesn't require a JVM, CLR, Mono, Python, Node, or some other runtime.)
I've been through the Rust book twice but I'm just getting to the point of trying to write something in it. The mental model is very different. Coming from C# / Java / Javascript / Objective C; I'm wondering how many hours I need before I can get my head into Rust?
And when their count reaches 0, you have your stop the world, unless you move the destruction into a background thread, thus manually emulating a tracing GC.
I explicitly mentioned how moving into a background thread is poor man's tracing GC.
Languages like D and C# offer mechanisms for deterministic deallocation, while keep the productivity of a tracing GC.
Then there native memory allocation and safe handles.
IDisposable and Finalizers aren't the same thing, actually, although they happen to be used together as means to combine deterministic destruction alongside GC based destruction.
You can also make use of lambdas or implicit IDisposable implementations via helper methods, that generate code similar to memory regions or arenas in C++, but in .NET.
Finally, many tend to forget that .NET was designed to support C++ as well, so it is also possible to generate MSIL code that ensures deterministic destruction RAII style, naturally this falls into a bit more advanced programming, but it can be hidden away in helper classes.
Well, not really. You cannot stack allocate anything but primitive buffers. So no objects or even strings. So it cannot replace heap allocation for anything but smallish "arrays" of simple types like char and int. This also means you can't use normal structs/value types, only primitives.
> You can also make use of lambdas or implicit IDisposable implementations via helper methods, that generate code similar to memory regions or arenas in C++, but in .NET.
> Finally, many tend to forget that .NET was designed to support C++ as well, so it is also possible to generate MSIL code that ensures deterministic destruction RAII style, naturally this falls into a bit more advanced programming, but it can be hidden away in helper classes.
I'm pretty sure there is no way in C# or in MSIL to explicitly free/deallocate a heap allocated object.
MSIL defines a Newobj opcode, but no Freeobj or anything like it that I have ever seen. You can use custom allocators or unmanaged memory to deterministically allocate and free buffers of structs/values types. But only those that do not include references to managed object references, otherwise you would need to pin and the references yourself and keep the GC aware that there were non-tracked references to those objects. It gets messy fast.
As for stack allocation, apparently you missed structs.
Here is your string allocated on the stack.
unsafe struct cppstring
{
const int BuffSize = 1024;
fixed char data[BuffSize + 1];
int current;
public cppstring(System.ReadOnlySpan<char> buffer)
{
for (int i = 0; i < System.Math.Min(BuffSize, buffer.Length); i++)
data[i] = buffer[i];
current = 0;
}
}
public class StackDemo {
public void Myfunc() {
var str = new cppstring ("Hello from stack");
}
}
Yes I realize they can be used to store the same data, but you've defined a new type that is incompatible with an actual string object.
The resources I've reviewed are:
- Rustlings: I personally don't like the project; it's exercises for the sake of exercising, which may be good or not, depending on the person/approach.
- Ferrous systems exercises (set of mini projects): very small and simple, but interesting to work on. I think they're a very good resource to try immediately after reading the book.
- Rust Programming By Example (book): if one finds one or more projects interesting to work on, this is a very fun book. it's "not-perfectly-produced" though, if you know what I mean.
- Hands-On Data Structures and Algorithms in Rust (udemy): even if one doesn't like algorithms, I think working with data structures implicity develops an understanding of "getting the head into Rust".
- Build your own Jira with Rust (https://github.com/LukeMathWalker/build-your-own-jira-with-r...): An exercise-based approach to practice; probably a good alternative to the previous resource, for those who really don't want to work with A./D.S..
- Talent Plan (https://github.com/pingcap/talent-plan): like the previous, but more technical.
Have fun :-)
And that's the problem that I have. In high school I was handheld into C / C++ with weekly lessons. By the time I started my career I abandoned C because the things I worked on professionally had no benefit from manual memory management.
Now the thing that I want to write, an embedded database, requires manual memory management and no runtime. I could, in theory, go back and do it in C. It'd be slow working in a language that I haven't done anything in since 2002, but at least I'm familiar with all the conventions.
Do I basically need to spend 40-80 hours doing silly exercises just to ease into the new conventions and mental model?
Even ignoring that, it's a matter of big picture.
If learning Rust is only for this project, or only to be "quickly proficient in a new language", then I don't think it fits the specific case. The may be alternatives; I don't have experience (somebody else can surely advice better) something like C++ with smart pointers or memory safe D, I guess, could fit.
In the big picture of a career, or even in the context of a single company, spending 40/80 hours to be proficient in a language is essentially an insignificant time.
At this point in my experience, if I want to learn a language, I write something "easy" in the language that I want to write for my enjoyment.
For example, when I was between jobs I wrote a personal blog engine in NodeJS so I could get up to speed in modern Javascript and the node ecosystem: https://github.com/GWBasic/z3
"Silly exercises" implies a programming exercise that has little point outside of instructing a basic concept: The kind of exercises I did in high school when I learned C are an example; there was no outside purpose to the code itself. IE, there's no tangible use to the code when it's complete.
What I did 3 years ago was write a small program in Rust that opens links listed in a text file. (I've written many versions of this program over the last 18 years, mostly for self-education.) When I first wrote the program, it was mostly copy & paste, but it compiled even though I didn't understand most of it.
Last night I decided that I was going to recompile it on Windows as my first exercise. I had to change the "open a link" library because it only compiled on Mac, which required changing some code: https://github.com/GWBasic/open_links
Now I'm going to try porting my in-browser Javascript in Z3 to Rust + WebAssembly. Let's see how far I can get!
How does root scanning work wrt to Loom? Are stacks of virtual threads treated as roots? I guess there is no other option?
I believe most GC implementations have non-concurrent "initial marking" phase, but that's typically fairly quick. It has to scan roots of your object graph, think stack, JNI, etc.
I found C++ to be treacherous around corners cases on this subject.
There are a few places in C++ where allocation can happen pretty much invisibly. A copy constructor is an example of that. You might see a new allocation simply by calling a method.
With rust, you usually won't see an allocation unless it is explicitly called for. You can follow the call tree and very easily pick out when those allocations are happening.
The C++ copy constructor comes up fairly frequently.
You would need to ignore the documentation and bend over backwards to use it for an actual type conversion, because you can only return references.
For example:
T t;
a = b;
T t2 = t;
let t: T; // won't allocate
let t = T::new(); // might allocate
a = b; // won't allocate
let t2 = t; // won't allocate
let t2 = t.clone(); // might allocate
And if you do `MyAwesomeStructure::new()` you might actually trigger a whole bunch of allocations which are invisible.
So a "yes" from my side on Rusts ability to remove allocations if you try hard enough. A "no" however on allocations being extremely explicit and easy to see for non experts.
Meanwhile C++ has a lot of implicit memory allocation and things that might or might not allocate.
This was common practice in trading firms that got on the Java hype train. Turn off GC and just restart the JVM outside of trading hours.
> Either way. It makes me excited for Rust and the languages it'll inspire, all this labor gone away.
The JVM gives GC a bad name. There are plenty of GC languages which don’t have the level of pain of hotspot except in extreme cases. For the vast majority of GC languages you never even think about it. Rust / C++ are great when you need full control but it’s not necessary for most things.
https://stackoverflow.com/questions/27813539/what-happens-in...
Also, if you don't mark your methods and classes as final, you're gonna be allocating functions everywhere as well.
You're going to allocate a single memory block that contains all the state of that object, which of course includes the superclass state. But that has nothing to do with the depth of the class hiererarchy.
> you're gonna be allocating functions everywhere as well.
If by "function" you mean a "Java method", they are code and not data, and they are not allocated dynamically at all.
In any case, this would have nothing to do with them being private or final. These options decide whether there will be an invokevirtual or invokespecial instruction to call them, where invokevirtual has more cost only before getting JIT-optimized.
Probably the best thing about the JVM that, AFAIK, no other platform does is it gives you an option for which GC you use. Not only is that an option, but you are spoiled with options. ZGC, G1GC, Shenandoah, The parallel collector, the serial collector. Pick the right one for the job!
But beyond that, they are infinitely tuneable (Not that you should usually). Again, not something that almost any other platform offers.
I know that lots of people have built careers on the JVM and defend it vigorously. Whenever I make a comment that is negative on the JVM, I always get downvoted without fail on HN. I used to be one of these people, after all, so I can relate.
If I truly cared about performance these days, I’d use C++ or Rust. Otherwise, I’d use a language like golang with reasonable default GC behavior and allocation patterns - where I know the daemon I write won’t use more than 100-200Mb, and probably even less, and will typically have sub-ms pauses. I think even most scripting languages, like Ruby/Python, have reasonable memory usage and GC patterns for web development, but I’m not as familiar with them.
I probably care more about energy consumption these days though. And again, the JVM is the worst offender. The average developer building a Java web app will create something which consumes gigabytes of memory usually. These apps tend to spend the majority of their CPU time allocating and GCing memory. I can only guess how much server time has been consumed to satiate the memory hungry needs of the JVM.
No. If you use the JVM you might well care about performance. If you care about performance, you should consider the JVM.
> I spent years as a performance engineer on HFT trading systems tuning JVMs.
The JVM is not designed to give you 100% performance (of the ideal offered by the hardware). It's designed to give you the easiest means of achieving ~95%, provided you're OK with some nondeterminism.
> I’d use a language like golang with reasonable default GC behavior and allocation patterns - where I know the daemon I write won’t use more than 100-200Mb, and probably even less, and will typically have sub-ms pauses.
You'd be wrong, because JDK 14 will most likely give you substantially better performance than Go out of the box; JDK 15 and 16 even more so.
BTW, the footprint and performance profile of JDK 14 is worlds apart from JDK 8. The JVM has been undergoing some very big changes in the last couple of years.
> And again, the JVM is the worst offender
Nope. https://greenlab.di.uminho.pt/wp-content/uploads/2017/09/pap...
> If a webapp performs similar to this experiment, I could rewrite it in almost any other language and I’d cut the required number of servers in half or more.
Nope. Because you gain in performance where you lose on memory. If you add more RAM, you'd pay for less hardware and consume less energy.
Anyway, if you last experienced Java in version 8, try 14. It's not what you might remember. You'd likely see a 10-40% reduction in cost-per-transaction, plus very low pauses with the new GCs. OpenJDK's VM in 14 is just significantly different from what it was in 8.
no matter how good a gc gets, if it does need to do double the work of another gc it is not a problem of the gc. it is that too much allocation is needed even for simple things. java tried to tune their gc over and over instead of just fixing the problem. allocations.
That's true, and while Valhalla will help close the gap, it is important to understand that Java's goal is not to be the fastest language, but to be the language where it's easiest to get to 95% of full performance potential. I don't think any other language achieves that as well as Java. The question, then is how much you're willing to pay for that extra 5%.
In other words, for every Java program X, there exists a C/C++ program Y that's at least as fast. But is the cost of writing Y worth it given the particular performance benefit? The reason Java is so popular is that the answer is very, very often no. And remember that the lower Java costs are not just in the "coding costs" of writing and maintaining, but also in observability. These days, thanks to the Java Flight Recorder, Java gives you unparalleled insight into what your application is doing for very little overhead.
we are basically struggeling to find people in the java domain, the amount of people with c# knowledge in my area is so much bigger.
---
however I'm personally a heavy java user (or scala should I say) unfortunatly the more things I did in c# the more I missed stuff especially in plain java. I thing the oracle takeover of java held the language back a bit for a few years. I mean if I look at the releases, they started to catch up with their new model, etc. which is good.
and I think what java these days are missing is a project like django/ror or aspnet.core which drives the web forward with a fully integrated web framework that does not suck. don't get me wrong spring looks promising, but it tries to fullfil every role by being "flexible" which it probably shouldn't be. enforce something. give us a nice little ORM that is not enterprise, look at linq. and especially don't be so enterprisy.
Granted the parent was anecdotal with their own experiences too, you could at least address the points. E.g. to the point of achieving 100-200Mb daemons, you might reference what RAM footprint you typically expect/experience from a similar program running in JDK 14-16. Rather than just:
> You'd be wrong, because JDK 14 will most likely give you substantially better performance than Go
What helps .NET is the design since the early days to support value types and the introduction of Midori learnings into C# 7 and later for low level programming.
Whomever they were, they were rotating pre-warmed jvm images with disabled GC, and were reaching quite respectable latency figures.
To not have to recycle them quickly, you'll want to not generate too much garbage objects, and that's actually easier than one might think in Java. Especially if you accept to restart the jvm from time to time, as you only need to be mostly statically allocated.
Rare error paths can freely use dynamic allocation as long as most of the service doesn't.
Nowadays you can also get away without using strings in most places, using only char sequence flyweights over "statically" allocated char sequences. Otherwise strings were a pain, especially API's that really doesn't need a string (ownership) but had string method arguments nevertheless.
Used like that, as you would on an embedded platform, theres nothing I know of that actually beats the JVM in raw performance while still being somewhat practical in terms of tooling and hiring. Rust might take that crown, we'll see, but I hope so.
Every single financial firm out there, using Java for sub-microsecond tasks. Really, there is no other way to keep low latencies if you have your GC messing around every few milliseconds.
This may surprise some people, but Java is ubiquitous in low latency environments such as trading firms. It offers performance close enough to C++, but the developer pool is way larger. Also, when one needs to comply with extremely low latency requirements, the way to go is always dedicated hardware anyway.
If you are interested in the topic, there is this project called OpenHFT that aims to provide high frequency trading tools for Java. Particularly, their Chronicle Queue implementation tries to handle the GC latency issue by storing stuff off heap. Its co-founder, Peter Lawrey, has also delivered a handful of good talks about low latency Java.
The only firm that widely uses Java over C or C++ on the hft fast path is Virtu, and the sense I get from some of my friends there is that they regret the decision (and now it’s mostly used to guide fpgas).
There is a TON of Java usage in software that would have been considered HFT maybe 10 years ago, especially at big banks, but the fastest I’ve ever seen somebody make a Java trading program that actually handled the whole tick-to-trade path is 7us, which would be at best ok by C or C++ trading platform standards.
I’m sure you could write some dumb trigger program operating on off-heap bytes that had similar performance characteristics to a C hft program, but unless you accomplish this via code generation then you’re just writing C in Java with a runtime actively battling your goals.
Edit: Jane street uses ocaml + fpgas but they aren’t really in the HFT business in the same way that say Virtu or Tower is.
The amount of bugs springing from just these far outnumbered memory bugs in the C++ and C codebases.
Granted, I have not worked in finance for over 10 years, but I also try to keep up with the times, thanks to some friends, mainly operating in the EU/UK market. But I would not say it is odd to think that Java is ubiquitous, given how technologies often permeate across teams. This is a bit like what happened to Python and FORTRAN, being the latter still widely used in the scientific world, especially physics, but Python adoption is definitely growing, despite its obvious shortcomings.
Now, I understand that the hard requirements around HFT are wildly different, but from the perspective of an employer, having a larger pool of professionals and tools, would offer significant savings. Why wouldn't they take it?
Because the pool of Java devs is only marginally wider than the pool of C++ devs? 5.4M C++ devs vs 7.1M Java devs is not a massive difference in pools:
https://www.daxx.com/blog/development-trends/number-software...
And while I don't have numbers for it, my experience has been that the vast majority of Java devs have very little experience with the kind of specialized use of the JVM that low latency programming requires. On the contrary, I've found it difficult to hire for high-performance Java codebases, because the way they are written is completely different from how "normal" Java is written, so you need to retrain devs used to writing "normal" Java.
That depends on the geographic distribution, though. Also, we are talking of a 25% larger pool.
> And while I don't have numbers for it, my experience has been that the vast majority of Java devs have very little experience with the kind of specialized use of the JVM that low latency programming requires.
You could say the same about C++ developers. Most of the C/C++ jobs are in embedded devices, which differ vastly from HFT environments.
The reason not to use java for low latency latency is that good low latency Java is pretty much only Java in a syntactic and to some degree object sense. Avoiding allocation permeates everything, and is just not worthy the battle.
You basically try to write C++ in Java, except without the language features of C++ to help with memory management and layout, and wit the added burden of having to worry about pools and primitive boxing and the runtime at every corner.
You would also be surprised though at how little hardware has replaced C++ on Linux with good user space networking. Not that many trades are pure latency arb, so getting that last microsecond or two just isn’t worth the effort.
It’s much more common to see hardware in fixed-function locations than it is to see hardware running the meat of a strategy.
Here we go again ...
Neat that they were able to get a dramatic improvement in GC latencies on both G1 and ZGC.
No mention of the Shenandoah GC. Would the same trick help out there too?
Exactly... I thought they may be talking about Project Loom's Virtual Threads (which are going to be true green Threads) which are available experimentally as of Java 15, given they did use Java 15, but nothing in the post indicates they used that.
Yes, but we found that the name "virtual thread" works better. For one, these fibres are actually an implementation of java.lang.Thread, so it's the same abstraction; for another, when people hear "fibres" they sometimes compare it to other implementations of fibres that are implemented very differently, and that causes confusion.
> When you say they aren't cooperative, you mean there's JVM infrastructure that actually pre-empts a fiber?
First, I mean that they don't require any explicit yields. Sometimes people confuse "preemptive" with "time-sliced preemption", so I'd rather just say "non-cooperative".
Second, there is a capability to forcefully preempt a virtual thread even if it's in some computation loop, but that's up to a chosen scheduler, and the scheduler isn't part of the JVM. You can supply your own scheduler, written in Java, and you'll be able to choose to preempt a runaway thread. Currently, this capability is not exposed, but it will be eventually.
See http://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part1.h...
So in this sense, are Kotlin coroutines also non-cooperative? They don't explicitly yield, but do have a mandatory API to use for "blocking" calls.
Yes it can and actually it's quite low-hanging fruit on the JVM. GC safepoints are already there, you just have hook into the mechanism.
try { lock.lock() // long computation here, no interrupts check } finally { lock.unlock() }
What happens when the virtual thread is externally killed in the middle of the long computation? Nothing at all because is not manually checking for the interrupt token? (like Go, unlike Erlang). Or is interrupted but the finally block is not executed and we get a dangling lock? I know Loom is not finished yet, but I would like to know about his prospective.
Arguably, virtual threads also explicitly yield by calling one of the blocking methods in the JDK. This is very similar to putting all the bottom-level suspendable functions into the Kotlin standard library.
>virtual threads, which behave more like Go's goroutines or Erlang's processes.
I think this can be summarized as "Kotlin uses colored functions and Loom uses non-colored ones". This is a well-established core difference, I thought you had something else in mind with "explicit yield".
There's nothing explicit here, or nothing more explicit than the ordinary platform threads you use today -- JDK operations might or might not block an OS thread. There is no way you can tell whether a call does that or not, and that behaviour can change (plus, there's forced preemption as an option).
> This is a well-established core difference, I thought you had something else in mind with "explicit yield".
I would say that an explicit colour for yielding qualifies as an explicit yield, and so falls under cooperative. In any event, virtual threads have neither explicit yield-sites nor special yield-site colours, hence they're non-cooperative, and they support forced preemption.
This is exactly the same as within the "colored" subspace in a language that has this distinction. As long as all the functions you call are suspendable, you equally have no idea which one will actually suspend.
So, without forced preemption based on GC safepoints, and as long as there is any blocking operation left in the library, Loom qualifies as a cooperative multithreading system.
If user code cannot possibly know whether an operation blocks or not, then it cannot "cooperate."
> As long as all the functions you call are suspendable, you equally have no idea which one will actually suspend.
First, this is true hypothetically but never in practice. Consider that if code really didn't care about blocking, then why not colour everything in the blocking colour? The answer is that in the coloured mode, compilation and cost of the two colours are very different.
Second, and more importantly, the reason that there are two colours is exactly to enable a cooperative style. While a "blocking" routine may or may not block, a non-blocking one never does and that is the crucial difference. With cooperative multi-tasking the default mode is that of a critical section -- there is no scheduling point unless you explicitly know in advance there might be one and where. With preemptive concurrency the default is the opposite: yielding may happen at any time unless you explicitly enter a critical section. This results in very different coding styles.
Anyway, we're arguing over definitions, so you may want to consult Wikipedia's definitions [1] [2].
[1]: https://en.wikipedia.org/wiki/Cooperative_multitasking
[2]: https://en.wikipedia.org/wiki/Preemption_(computing)#PREEMPT...
This tells me virtual threads (without forced preemption) are cooperative.
We call them preemptive with or without forced preemption -- in line, I think, with the definitions on Wikipedia -- but whatever you choose call them, the concurrency programming style is the same as that with threads today or Go's goroutines, and is different from the style of C#/JS's async/await, Kotlin's coroutines, or more explicit async code, all of which result in user code relying on knowing where yield points (possibly) are (i.e. "critical section" by default). BTW, even with OS threads, when you run transaction-handling code, as opposed to long-running computation, time-sharing preemption is the exception rather than the rule.
Virtual threads are of the former kind. (At least as long as we don't involve the forced preemption feature).
For the distinction you have in mind, I see the terms "colored" vs. "non-colored functions" to be used the most, and they are both within the "cooperative multithreading" space.
And then say it's comparable. "This basic design is also present in the concepts of green threads and coroutines. In Hazelcast Jet we call them tasklets."
True that one can end up writing terribly inefficient Java code, but one can write terrible code in any language. If I need to write server code where performance is particularly important and I don't want to deal with the cost (in debug time and dev expertise) of C or C++, Java would be my first choice.
Also I'm of the school of thought that performance always matters. Autoscaling in cloud providers sure makes it easy to scale horizontally to make up for slow server code, but once you reach certain size, go have a chat with the finance team about the AWS bill.
What? We've (JavaScript and developers dealing with asynchronous patterns) been able to build scalable (in terms of code and its maintenance) for many many years, probably 10+.
async/wait is simply syntactic sugar and doesn't drastically change anything, you still need to understand the asynchronicity underneath it all, and if you do, you won't have any problems building scalable apps using your knowledge.
It's a complete shift in application design...
If I have to flatmap one more time...
Absolutely, asynchronous and synchronous are two very different patterns when programming with very different trade offs for both of them, but async/await is not synchronous, it only make that particular call _look_ synchronous, while actually being asynchronous.
Failing to understand that async/await is just syntactic sugar for dealing with asynchronous programming will sooner or later bite you.
Here is a gist that shows why async/await is not really synchronous (blocking) programming: https://gist.github.com/matt-mcalister/3f060bf32d292cfebc944...
From https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.ht...
-------------
An alternative solution to that of fibers to concurrency's simplicity vs. performance issue is known as async/await, and has been adopted by C# and Node.js, and will likely be adopted by standard JavaScript. Continuations and fibers dominate async/await in the sense that async/await is easily implemented with continuations (in fact, it can be implemented with a weak form of delimited continuations known as stackless continuations, that don't capture an entire call-stack but only the local context of a single subroutine), but not vice-versa.
While implementing async/await is easier than full-blown continuations and fibers, that solution falls far too short of addressing the problem. While async/await makes code simpler and gives it the appearance of normal, sequential code, like asynchronous code it still requires significant changes to existing code, explicit support in libraries, and does not interoperate well with synchronous code. In other words, it does not solve what's known as the "colored function" problem.
Soon to be almost 20 years ago we pushed >10k messages per second on a JVM, using essentially pentium pro class hardware, and the messages spread over thousands of TCP consumers, yielding average latencies well below 0.5 seconds. Not really low latency, but low enough that we didn't need much lower
This was on a purely blocking implementation, because that was before almost anyone did anything like that on the JVM.
With the advancement in async IO, it's got to be possible to drive many millions of sockets, or have really low latency targets before you have to start being really careful?
So what are you guys doing that seems to need so much async code?
With that I mean actual async code, not code having locks but pretending to be async by using callbacks everywhere, because that's somewhat common.
I'm not trying go be rude, I honestly don't get what people are doing that needs more than the JVM should rather easily provide, unless possibly you have super low latency targets?
There seems to be too many that have to resort to quite cumbersome implementation strategies, so I'm starting to think there's some corner of the industry which I have completely missed, and which requires these strategies regularly?
"Aegis Battleship Weapons System"
http://www.artist-embedded.org/docs/Events/2011/JTRES/Slides...
"French radar system for ballistic missile tracking and measurement"
https://www.militaryaerospace.com/defense-executive/article/...
"NASA Ground Control Station for Multiple UAVs Flight Simulation"
https://www.semanticscholar.org/paper/Ground-Control-Station...
BTW, many if not most of the cutting-edge advances in compilation, low-overhead deep profiling, and garbage collection are done on the Java platform, so it's still the technology leader in those areas.
Ah and the beloved OS around here is reaching 60 as well.
Plenty of legacy love.