Hacker News Reader

why not just use C++ or something and never deallocate memory then?

Malloc is usually slower than GC-based allocation (which basically just increments a pointer). Of course, one can emulate this via custom allocators in C++. My guess is that Java development is just easier and quicker, and the JIT may even result in more effective optimizations than AOT compilation.

in C++ they could also try using Profile-guided optimization to optimize hot code paths.

In C++ you can link with your own version of malloc (one which just returns the next consecutive block, and can be implemented simply by adding the allocated size to a pointer).

You generally choose the language which has the libraries and ecosystem which solves your problem. For instance, you'd be silly to use anything but Java to work with Hadoop. This is the pragmatic choice.

But as soon as you make use of almost any library from the ecosystem, you get allocations. Doesn't that defeat the purpose? Why not isolate the Java-using part in a separate process from the non-allocating fast part?

Even if you completely disable GC, Java is still allocating tons of ephemeral objects on the heap. Which in turn is leading to expensive and unpredictable page faults. In contrast, C++'s default is to allocate objects on the heap unless you knowingly call new/malloc.

Of course, it's possible to write Java in such a way to minimize this type of heap-thrashing. But by that point, you're already doing the equivalent of C++'s manual memory book-keeping anyway. It's not like "just provision a ton of memory and turn off GC" is a free lunch.

That's covered in Java by escape analysis and allocation on the stack.

We have written a database in zero GC java and one thing I have not seen any evidence of "escape analysis".

   @State(Scope.Thread)
   @BenchmarkMode(Mode.AverageTime)
   @OutputTimeUnit(TimeUnit.NANOSECONDS)
   public class EscBenchmark {

       Rnd rnd = new Rnd();

       public static void main(String[] args) throws RunnerException {
           Options opt = new OptionsBuilder()
                   .include(EscBenchmark.class.getSimpleName())
                   .warmupIterations(5)
                   .measurementIterations(5)
                   .forks(1)
                   .addProfiler(GCProfiler.class)
                   .build();

           new Runner(opt).run();
       }

       @Benchmark
       public int testEscapeAnalysis() {
           int[] tuple = {0, 2}; // esc analysis? where are you?
           return tuple[rnd.nextPositiveInt() % 2];
       }
   }

And the output of GC profiler:

  Benchmark                                                     Mode  Cnt     Score     Error   Units
  EscBenchmark.testEscapeAnalysis                               avgt    5     8.234 ±   0.029   ns/op
  EscBenchmark.testEscapeAnalysis:·gc.alloc.rate                avgt    5  2647.216 ±   9.275  MB/sec
  EscBenchmark.testEscapeAnalysis:·gc.alloc.rate.norm           avgt    5    24.000 ±   0.001    B/op
  EscBenchmark.testEscapeAnalysis:·gc.churn.G1_Eden_Space       avgt    5  2643.140 ± 177.137  MB/sec
  EscBenchmark.testEscapeAnalysis:·gc.churn.G1_Eden_Space.norm  avgt    5    23.963 ±   1.613    B/op
  EscBenchmark.testEscapeAnalysis:·gc.count                     avgt    5   157.000            counts
  EscBenchmark.testEscapeAnalysis:·gc.time                      avgt    5   103.000                ms

Scalar replacement has several requirements, and escape analysis is only part of the story. One of the restrictions with scalar replacement for arrays is that indexing has to be constant at JIT time, which your example clearly isn't.

The point would be that escape analysis could have proven that the array was not used outside of the function and allocated it fully on the stack, thus not providing any GC pressure. Scalar replacement isn't the point of interest really.

Which version of Java are you using? And what is Rnd? The class included with JDK is Random. We rely on escape analysis to elide such object creation for us and it works reasonably well. If something this trivial doesn't work for you, file bug with them. We have had success with that as well.

This ran on Java 11. This isn't so much of an issue for us. We are trying to avoid allocations, even as trivial as those. There are other examples that allocated where they should not, for example this lambda will allocate.

  () -> System.out.println(1)

I lost hope in escape analysis quite frankly.

Rnd is something I have written because Java's Random is slow and clunky.

https://github.com/questdb/questdb/blob/master/core/src/main...

Escape analysis is very JVM implementation specific, but inline classes should finally sort it out.

The JVM does scalar replacement, not stack allocation. The former is much more limited in functionality, and much more finicky. You also have about zero control over it, compared to, say, C# structs.

If you add -XX:+AlwaysPreTouch to the JVM arguments, the JVM will pre-touch the entirety of the heap at startup time to avoid unpredictable page faults through the life of the application.

I'd imagine other HFT companies may also pay for the Azul JVM which goes even further and comes with a kernel module that the JVM coordinates with for memory allocation. The kernel module pre-reserves x% of system memory at the time it is loaded.

Is the Azul JVM open source? I thought it was just the support which costed money. (I'm just not familiar enough with it)

It is commercial, and sadly their GC algorithms are patented. I'm really anxious for those patents to start running out because the functionality is quite clever.

Basically one main reason you need to stop the world in common GC's right now (and thus cause pauses) is because if you want to compact memory a GC thread has no idea if other threads reads/writes to an object while moving leading to worst case corrupt/lost writes.

What Azul does is to make the pages in question invalid for reading/writing so any access to them would page-fault and then proceeds to move the objects without worrying about other threads. Since the hardware provides the page-fault "for-free" this protection doesn't cost anything in terms of runtime performance in the common case (compared to the new ZGC made by Oracle that has read-write barriers that steals some mutator performance), worst case if another thread tries to read/write memory that is being moved a specialized page-fault handler detects if this was an page that was in motion and can then do a slower read-write but ONLY in the seldom cases where this occurs (compared to all the time of f.ex. ZGC)

why does the hardware provide page-fault for free again?

Page-faults aren't for free (actually kinda expensive) but rather the paging functionality is giving "free" read/write-barriers instead of those being instruction sequences being inside the main program (ie read/write barriers are handled as page-faults by the paging system that the OS already provides to avoid requiring every read/write of the user-code to be aware of moving pointers).

Hey thanks. That was a fantastic explanation.

There's two, Zing (pay for support) and Zulu (pay for access)

seems pragmatic; remind me of this old story: https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...

This is similar to how missiles don't need GC.

https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...

it has, it explodes and the memory is free again. just not for further usage.

The memory is freed all over the target, I suppose

I guess what you mean is the no-op garbage collector which is available in Java 11 http://openjdk.java.net/jeps/318

Even this isn't fail proof right ?

Last-drop latency improvements. For ultra-latency-sensitive applications, where developers are conscious about memory allocations and know the application memory footprint exactly, or even have (almost) completely garbage-free applications, accepting the GC cycle might be a design issue. There are also cases when restarting the JVM -- letting load balancers figure out failover -- is sometimes a better recovery strategy than accepting a GC cycle. In those applications, long GC cycle may be considered the wrong thing to do, because that prolongs the detection of the failure, and ultimately delays recovery.

So essentially you need to know the memory footprint of your applications. To err on the side of caution just get as much memory that is available in the market.

At this point like another reply mentions here aren't you just better off writing C++ code ?

Depends if your returns are based on writing faster code or shipping correct code faster. Once the advantages of the former are eliminated, optimize for the latter.

> * I guess what you mean is the no-op garbage collector which is available in Java 11*

I think I heard of the same pattern before 2018, so I guess there are other ways to do it. Perhaps Java has some option like memory usage value at which the GC is run.

I wonder if it would be cheaper for them to have a hot standby system, and swap them every few hours.

Speaking from experience with JVM HFT applications (we used Scala).

There are a lot of tricks though to not require 1TB.

And allocation in general is a bad idea even if you don't collect because it scatters stuff all over memory and messes up cache locality. You really, really don't want to allocate in a performance sensitive jvm application if you can avoid it. It's the opposite of a lot of what I was told and taught (e.g. never do object pooling), but empirically, in my experience, allocations are the biggest slowdown. You can get an application a lot faster just by opening up the memory allocation tab in a jmc flightrecording and refactoring the biggest allocators, usually there is a lot of easy to optimize low hanging fruit that will give good performance improvements, even better than focusing on hot spots in code (in my personal experience).

By far the biggest allocator in trading is going to be marketdata and calculations on it. For reading marketdata from the exchange it's best to leave raw data in memory and access it with a ByteBuffer / sun.misc.unsafe. Under this pattern classes have 1 value, the memory address to pass into sun.misc.unsafe, then everything from there on is done with offsets onto that address. For calculations it's better to write things as static functions, or use object pooling.

In the course of optimizing a trading engine I wrote lots and lots of code to get allocations down to zero. It's definitely doable, but best done from the start, I refactored an existing trading engine to do that, it was not very fun.

1TB of RAM is cheap though, versus spending engineering hours.

Reminds me of the classic WTF "That would've been an option too" https://thedailywtf.com/articles/That-Wouldve-Been-an-Option...

agree. hardware is getting cheaper and cheaper

Except Memory / DRAM.

This was a great read, thanks for that! Never heard of The Daily WTF before.

Congratulations!

https://xkcd.com/1053/

Of course there’s an XKCD for this! Ah, the wonders of the internet.

The article in question was written in 2008: I was 11 years old. I guess I now fall in to the category of “heard of it by the age of 30”.

> 1TB of RAM is cheap though, versus spending engineering hours.

It's not about the amount of RAM, it's how fast and predictable the overall system is. Note in particular the remark about cache locality; that can be the difference between nanoseconds and microseconds.

What the GP is referring to (Misc.Unsafe) is basically going back to manual pointer reads/writes (IE writing more or less C code in Java), the benefit is you get C speed/memory layout/cache-locality but with the downside of writing it in a less suited language (Java).

HOWEVER If you DO _allocate_ much then 1TB seems like a great choice since GC pauses are killers w.r.t. latency in comparison to cache issues. A cache line (often 64 bytes) would probably only hold at most a couple of Java objects (minimum is probably like 16 or more bytes for each small object) so cache locality won't be improved much by a GC (Yes, the G1 GC in newer JDK's does neighbour compacting iirc so you get a little cache locality but only if the patterns were bad from the start, but avoiding GC entirely is better)

Can you explain why you would code java this way instead of dropping down to C?

Probably because they started out in Java and don't want a full rewrite once they've gotten this far (might be that they don't have many C++ devs anyhow). I remember working on similar codebases 10 years back when J2ME games were still a thing (J2ME runtimes usually had horrible GC's and some people really went overboard in trying to avoid them) and it wasn't that fun at all (Even if the questDB codebase that is linked elsewhere in this thread seems to be saner).

Also JNI for calling C/C++ code is fairly bad and error prone (if you want to combine code), JNA is better but i think this is one of the bigger reasons that Oracle is funding GraalVM is that it promises "seamless" interoperability and breaking up things into Java and C/C++ parts might be a good option in the future.

JNI replacement is called Panama it doesn't have anything to do with GraalVM.

GraalVM is the new name of MaximeVM one of the JVM meta-circular JVMs. Other well known ones being JikesRVM and SquawkVM.

What Oracle and others in the Java community are doing is reducing the need to drop down to C with support for value types and more fine grained control over native memory.

Java 16 will have the first preview release of the new low level memory APIs.

The meta-circularity is part of the impl details, cross-language calling (Java/JS/LLVM) seems to be the main selling point of it, so if you want GC-less logic then Clang->LLVM then running on the JVM is an option, no idea if it also simplifies native bindings though (that is more of a Panama issue and improving that is a good idea aswell).

Productivity and safety of Java.

Those unsafe bits are a tiny portion of the overall code.

> ...a new intern named Bob. Bob raised a finger and said "Yes, I have a question...

Damn those smart-ass interns!

Do you know of any public examples of how this sort of Java looks? I imagine you lose out on being able to take advantage of much of the JVM ecosystem and I struggle to see what using Java even adds anymore.

here is one: https://github.com/questdb/questdb. Disclaimer, I work on this project. Main reason we use Java is speed of development (which increased with amount of base libraries written) and ease of testing.

Would you do it again? Coming from C/C++ first, Java then and coming to C# it just feels so much more pragmatic with f.ex. struct(s), slices and stackalloc (and unsafe blocks with pointers in a pinch) allowing for GC less programming w/o resorting to turning pointers to integers and using function calls for memory access all around. (Noticed that you do i guess query compilation via the ASM toolkit?)

It is a good question. I feel there is a happy medium where boilerplate is in Java and more intricate data processing routines are in C++. Makes both worlds simpler. I like C++ better. Those things you mentioned - Java truly sucks at indeed, but you don't always need them. When it comes to IDE, testing, compilation speed, cross platform code and finding talent - Java is way easier than C++.

Yeah i see that outlook, my comment was actually mostly about C# (been using it for the past year and these things above are in it as well as all the base being Java-ish)

Ha, these are many of the same things one might do to make a chess engine fast on the JVM. This only strengthens my belief that the best computer science education you can get is studying chess programming.

I had a side project of 'flattening' classes at classloading time into large mappedbytebuffers, removing all allocations needed for ser-des and recursing on belongs-to references, copying buffer slices from sockets to the buffers (and reusing the socket buffers), and using netty for sockets (selector allocates crazy). Very fun project with javassist...

RAM is not the only aspect of a Garbage Collector....

Over a decade ago, we used a third-party pre-trade risk system that was implemented in Java. Since it was a "service" (we connected to a TCP port), the underlying tricks they used to make it "fast" were transparent to us... until it was not.

They highly tuned their GC to where there was seldom any GC. One day, the third-party made a change to a supervisory service to generate more periodic monitoring emails. The file handles from this actor were apparently not GC'd and held by the process until the system ran out of file handles. That made the service stop working properly.

But, in addition to alerting, that service had a more important job: it was a post-trade risk "watchdog" to the pre-trade risk gateways, which we sharded across.

The various pre-trade risk gateways, upon not hearing from the watchdog authority then 1) began cancelling outstanding orders and 2) not allow new orders. We saw this happening haphazardly over 10 minutes and had little time and capability to recover; this also happened near the end of the trading day when some orders (MOC, LOC) are not cancellable. This was across our whole organization so it affected many strategies. So it wasn't as simple as "stop all" and although we had many checks and recovery procedures, this was a pretty special Chaos Monkey.

We ended up with a >$1B basket of random stocks that cost >$10M to liquidate over the next few days.

$10M evaporating in 10 minutes because of "GC optimizations" and poorly considered OS settings.

The #1 risk in algo/HFT trading is not financial, but operational.

[Some of the technical details may be slightly off, as it was third-party; my outlook is pieced together from post-mortems. Also, no other party (e.g. broker, SIPC, market participant, the third-party) was financially affected besides our firm.]

Game devs face similar challenges, especially on phones. Though they have to be less radical in their solutions—which afaik usually boil down to ‘preload the level into memory, use object pools, and don't do any new allocations’.

Why don't they use fixed size arrays like in embedded systems and not allocate memory dynamically?

How was these achieved before Epsilon? http://openjdk.java.net/jeps/318

Also ZGC / Shenandoah can help a lot otherwise