Hacker News Reader

Comment by howlgarnish | Hacker News Reader

I used to work on a high-speed messaging system built in Java, and garbage collection was a killer. No matter how much we tweaked the code and the GC parameters, and this eventually got escalated pretty high up in Sun itself, we'd get these occasional multi-minute pauses where the whole thing would just freeze up, buffers would fill and stuff would start dropping on the floor.

This was over 10 years ago, so things may have improved, but it would certainly give me pause about ever implementing anything time-critical in Java.

View on HN

Currently developing a radio communications system using Java, with JNI calls for hardware integration, running on a quad-core 32-bit ARM processor. We have constraints of around 20ms for real-time audio packet processing. Recently some stop-the-world pauses introduced by the GC resulted in calls dropping. A few GC parameter tweaks brought our latency below 20ms and the application no longer drops calls.

This is using the GC that ships with JDK 11. Future improvements, such as Shenandoah and ZGC, may be able to further drop the interruptions to 5ms on average.

The radio repeater is designed for real-time, safety-critical operations dealing with voice communications. Java is up for the job; like others have mentioned, it's more about the technical team and whether management understands what it takes to write and maintain an excellent code base than it is about Java versus C++.

>A few GC parameter tweaks brought our latency below 20ms and the application no longer drops calls.

Tick to trade latency for a modern software HFT stack needs to be under 10 _microseconds_, which is over three orders of magnitude more stringent.

10ms is target at which percentile for HFT? From what I could tell in my limited experience, GC impact is worse at the tail of latency distribution

10ms is milliseconds though not micro, that's 10us

µs

Sorry I meant us - but at what percentile

> 10ms is target at which percentile for HFT?

They already just said they need 10 microseconds. 10 milliseconds is far far too slow.

1000x slow so its unacceptable .

It would be interesting to know what the trade-off for those tweaks was. Typically it's increased memory usage.

Another relevant aspect is what happens when one hits GC pauses again. Will there be any tweaks left? Will they conflict with the previous ones? Using such high-level levers to fix performance problems in a specific component is great when it works, but when it doesn't you're out of options.

Another option is to use a JVM that offers hard real-time constraints to achieve guaranteed performance characteristics. Such JVMs are in use in the automotive and airline industries---JamaicaVM, for example, offers deterministic garbage collection.

You’re coding safety critical systems and depending on severely non-deterministic tweaks to GC to get it to run right?! Where is this for? (Just so I can avoid it)

No need to know where this person is deploying this, if you want to avoid people doing stupid shit with safety critical systems you should just avoid going outside. Actually, avoid leaving the bed, as everything software in this day and age is basically just duct tape and cable ties put together in a rush to hit deadlines.

Have direct knowledge or worked with many critical systems in a few different areas (cannot get more specific for several legal reasons). Can confirm. I'm given to understand that legacy systems (think 30ish years ago) were much better engineered, but there's not enough Proper Engineers(tm)(r) to design (and build) all the critical systems that are necessary.

The biggest issues are that an engineer with the knowhow is expensive, a team of them is prohibitively expensive, a lot of companies outsource this work to different countries and make them sign onerous and dubiously-enforceable contracts, pay little, demand too much, and have incredibly unrealistic deadlines for the type of work that needs to happen.

You know how it is, if it never fails, you're throwing money out. That's the mentality of a lot of management types I've dealt with.

> You’re coding safety critical systems and depending on severely non-deterministic tweaks to GC to get it to run right?!

That's certainly one way to look at over five years of engineering effort, extensive systems testing, countless hours of automated regression test development, independent product quality verification, rigorous code reviews, risk assessment processes, mandatory four 9s call quality requirements, integration analysis of JVMs with deterministic GC, and so on.

Judge not a galaxy from a single star.

> real-time, safety-critical operations

Doesn't Java itself say it shouldn't be used for anything like this?

Most software does. But, let's face it, it's not like C or C++ compilers should come with a manual saying "We strongly recommend using C for safety critical systems".

(MISRA-C and other odd dialects excepted)

Nope, there are even two vendors specialized in real-time Java based solutions.

TRWTF is that Java Specification Request 1 (yes, THE very first one) targeted real-time: https://www.jcp.org/en/jsr/detail?id=1

You should not have any full GC (pause) events, ever, in properly written production Java code.

Source: I worked on several extremely high performance Java systems at Sun and also worked optimizing various Java servers at subsequent jobs based on my Sun experience.

At one company (after Sun) I took on a backend service which was doing GC pauses every 3-5 minutes, the code was a dumpster fire mess. After cleaning things up we didn't hit a single full GC in two months of production use (they redeployed every two months so that's as long as the process lasted).

Extremely high performance properly written Java code has been mentioned so many times in Java performance discussions that it has acquired an almost legendary status, right there with the sufficiently smart compiler.

I have observed the former, written by exceptional coworkers, but never the latter.

Maybe, but that just raises the question...If you're coding your system under the constraint that you should not be having any GC-event, what's the point of picking a GC language in the first place?

GP was not talking about "any GC-event", but about "any full GC (pause) events". That's where the heap has gotten so messy that everything has to stop while the GC sorts it out. It's an absolute worst case scenario. Under normal conditions on a modern JVM, the GC runs in parallel with the application and does not cause any pauses.

They said no full GC. The GC runs constantly in parallel with your program and cleans up what it can, but never has to pause execution of the whole program to reclaim even more memory. Not claiming I know how to do this, but I believe this is what they claiming.

As others said, no full GC events. Young generation GC is cheap, no need to avoid that.

Productivity is the point. Even while paying attention to GC, it's much faster to write good code in Java than to worry about every malloc() in C. I love C, but if I need to churn out high performance code quickly, Java is the choice.

But here we're talking about C++: you don't write malloc, you use containers that encapsulate memory management - likely std::containers with pool allocators. And you put things on the stack. Code is as readable if not more than java - there isn't any "new," anywhere in sight

Modern C++ is probably about as good, with sufficient team discipline. I don't have enough recent experience with C++ to have a strong opinion.

IMO (possibly biased by how much I hated C++ in the 90s) it is easier to maintain clean code discipline in a large team with Java than it is with C++.

C++ after 11 is a very different beast. It's more pleasant to write compared with pre-11. Memory model, move semantics, smart pointers, lots of stuff standardized instead of left to the libraries.

Definitely depends on your team of course.

smart pointer template blah, instead of pointer blah, in a very large code base.. hmm not so pleasant.. the debug level function definitions, adding templated parameters, are literally unreadable. I will agree that the mental model is tamer.

Well, the first place may have predated you and a rewrite is out of the question.

A few other reasons:

The rest of the platform for review and analysis doesn’t have such stringent performance requirements and it would be nice to reuse code.

There also may be some good libraries that are jvm only that your program relies on.

Packing up a fat jar can be a lot easier, less complex and more reliable than building a binary.

Your dev team has a lot Java expertise.

Getting Java to not allocate is not as hard as it seems if you set out to do it from the start. And even if not it’s not impossible because the tooling is so good. You just have to know the right tricks and it can be easier to incrementally learn those tricks for a dev team than learn C++ and it’s ecosystem.

Did you ever detailed on the Internet about those methods to attain no GC pause? It would be interesting to learn from your experience

It's not rocket science honestly. Mostly any Java performance guide will tell you how.

Allocate either locally on the stack or allocate for the life of the process so it never gets GC'd.

Of course, spend time profiling to make sure you got it. Figure out your redeployment cadence and that sets the ceiling for how much data can be an exception to these rules.

(e.g. if you redeploy nightly then it's fine to grow the memory consumption as long as you don't hit full GC in less than 24hrs.)

In a nutshell that's it.

I agree, because I've seen that in my previous jobs...

Programmers used to say that the GC and its fuc*#=+ STW was a pure mess, but under the hood, they didn't realize (accept ?) in fact the code WAS the mess.

Don't (always) blame the tool !

Another user commented here a while ago that they solved this problem by just throwing it on the largest instance by memory (1TB or so), disabled GC and restart it after market closes.

Largest instance sounds like aws. There’s your nr 1 performance bottleneck

well, you choose the node infrastructure by whatever network latency you need to have guaranteed - obviously public clouds have their limits there.

But it’s idiotic to discuss ns-optimizations, when your infrastructure and vms add many times more that.

Aws is slow. Not talking about the network.

ehh I didn't even mention AWS - all I did was using the word 'instance' which seems to have thrown you off.

What else can "instance" mean, other than an instance of a virtual server? Granted may not be on AWS, but still is virtual which is a latency killer.

IMO you can't automatically go from "we have a latency requirement" to "public cloud is out of the question". It depends on what latency requirement. If you can eat an intermittent delay of 20ms, AWS, to me still seems fine, you just need to plan ahead in terms of what resources you rent there. Between that, and the multiple seconds a GC pause could cost you, there is still a space where GC optimization on AWS would make sense. But if you're in HFT and target micro- or even nanosecond optimizations, you sure don't wanna run anything in that critical path on any public cloud.

Why did your system dynamically allocate memory in the first place? High speed messaging sounds like an area where it should be doable to figure out at startup what resources you need to meet your latency and throughput requirements and then allocate them once and basically never gc anything. Was there some fundamental difficulty with that, or was it more that by the time your realized that GC would kill you, all your code was already written in a way that would have made it prohibitively expensive to switch to an architecture that avoids creating temporary objects?

The best part about using a garbage collected language for high performance software is it's exactly the same as a non-garbage-collected language.

Sure, as long as your non-garbage-collected language from ten years ago had memory safety (and general lack of UB footguns), first class IDE support, excellent introspection and profiling facilities and a large potential hiring pool of elite programmers and wide acceptance in the industries you were trying to sell into.

Wait, are we still comparing C++ with Java here?

Yes? If you write zero or low allocation code in Java you pay a price for working against the language and you also lack several tools for writing fast code that C++ provides you with (as well as a bunch of useful abstractions), but your development experience will obviously be much better on a number of other dimensions.

Why is it you think most major programs are built using C++? Most OS's, most drivers, most browsers, most games, most major applications, most compilers, most servers etc etc.

Most servers, definitely not.

Consider that large chunks of Google, Twitter, Amazon, Alibaba etc run on Java.

C++ completely dominates games, browsers (and a few things you don't list, such as competitive programming). It is certainly not marginal in the other areas you list either, but were do you get the weird idea that most operating systems and drivers are written in C++ (let alone most compilers, servers or "major applications")? Unix has completely won the OS market (what percentage of OS installs these days are unix derivatives? 99%+?) and I can't think of a single one that is predominantly or at all written in C++. Apart from lacking a browser, I'd be surprised if you couldn't run a fully functional linux desktop after deleting every single C++ application and library on your system.

Pretty much the latter: it was initially architected for 1x volume, but ended up handling 100x what was originally anticipated. So the engines were rebuilt a few times while the plane was in flight. As a bonus, while in production the system had to run 24x7 and upgrades were a fairly fraught manual affair involving hours of testing on the customer side.

The other major complication was that this was doing a lot of complicated protocol conversion, so it wasn't enough for the core message processing & routing to be solid and leak-free, all the input parsers and output converters had to be too.

It's changed quite a bit, with some of the biggest developments just in the last couple of years. It should still give you pause (depending on how time-critical) but now you have more choices in pause and more pause-knobs.

Cassandra has some of these issues as well, despite arena allocation, buffer reuse, and offheap manually allocated memory.

10 years has seen improvements to GC, but not eliminated the problem.

Cass 4.0 finally enables ZGC, so we'll see how that goes.

If you have stateless, you could do two JVMs and monitor the GC state, and reroute requests to a low-heap JVM and let the other clean up.

Garbage collection is hard.

I'm surprised that there aren't expert-level concepts like weak references and ways to sub-partition the heap, with say a new operator that can target a partition, so major GC doesn't have to do the whole heap. Although that starts getting into rust concepts and analysis.

It definitely improved. The GC is the area where we see more work between the Java versions

Reply on news.ycombinator.com