Hacker News Reader

Comment by viraptor | Hacker News Reader

I know this isn't the point of Folia's test, but:

> During the period when 1,000 players were online, we reached a maximum of ~7.9GB/s heap allocation and our GC was hovering around 2-3GB/s when averaged over a minute.

As someone who dealt with soft-realtime telephony stuff, this makes me want to scream in horror. It seems like the platform really hurts the performance here. In an average second with that many players, most (all?) of them will not do any action apart from changing their position. (Just moving around would ideally only use preallocated per-player space only; extra in-game events, npc AI, etc. could still use some allocs) It's great Folia can get more out of the bad situation, but... oofff.

(not a criticism of Minecraft by the way, it's better to be popular and inefficient than not exist - I just didn't realise quite how much overhead there is due to the runtime)

View on HN

> In an average second with that many players, most (all?) of them will not do any action apart from changing their position

The world is not static though. Each player loads in a lot of living entities (monsters/animals/...) and block entities (furnaces, redstone, ...) that all need to update. There's some overlap of course, though in a game like Minecraft where there's a near infinite world to explore that overlap is smaller than you would think.

Well yeah but that's a process that can run independently from player movement and (I guess?) can be a highly asynchronous and paralellised process.

(note: armchair remark, I honestly haven't a clue)

Not really.

Block entities (those with complex data, like inventories) can be read/interacted with by redstone (a regular-ish set of blocks in terms of data), which can then trigger a piston or dispenser (which changes the world), and said piston or dispenser can then interact with the players movement, pushing them, or hitting them with an arrow, or dispensing water that changes their movement.

Same again for non player entities, but also factor in their AI having to adjust pathing based on the world changing.

> can be a highly asynchronous and paralellised process

It really can't, though. Parallelization is the enemy of consistent game mechanics, especially in a complex sandbox game like minecraft. There are tons of things that interact with each other in minecraft's world - if you just tick everything at arbitrary times, those interactions cease to be deterministic and might break entirely unless you write a ton of spaghetti code to deal with every edge case. Redstone in particular is the best example of a mechanic that heavily relies on the synchronous nature of the game logic loop.

> Each player loads in a lot of living entities (monsters/animals/...) and block entities (furnaces, redstone, ...) that all need to update.

Sure, but that list of entities in the area is close to static (apart from crazy redstone magic). One would expect them to be pooled and not have many allocations for each tick.

But each has to do a lot of stuff. People on such big servers build farms with hundreds of mobs dropping thousands of items which then go through redstone sorting systems.

> People on such big servers build farms

~Normally, yeah, but Folia is based on Paper which, AFAIK, is known for its unreliability when it comes to the technical aspects of Minecraft (ie redstone and farms.) Even your bog-standard simple item sorter is apparently a bit hit-or-miss on Paper.~

Edit: Possibly outdated info, things sound better re: technical Minecraft support on Paper now according to 'ocelotpotpie.

The hardcore technical folks seem to prefer things that don't alter the vanilla mechanics quite as much as Paper, it's true. But FWIW there are lots of farm and redstone designs that work reliably on Paper.

> But FWIW there are lots of farm and redstone designs that work reliably on Paper.

Ah, good to know - been a long while since I last tried Paper.

Its fine for most stuff the average player does. I wouldnt run a redstone computer on it but I personally never had an issue with my farms.

> Even your bog-standard simple item sorter is apparently a bit hit-or-miss on Paper.~

My simple sorters have been working just fine(tm) on Paper in the last (>5) few years.

I do hear co players having occasional problems with fancier set ups, but most automation just works.

The kind of mental gymnastics that people do here to justify needing 8 GB of RAM with a turnover of 3 GB per ~minute~ second to handle 1000 users.

Per second.

It is definitely more than required, but let’s not go to the other end, it is neither a static html page.

I wouldn't say "each has to do a lot". There's a few trivial behaviours usually. Especially if you know what behaviours are common, you can optimise entity lists for that.

But that's not even what we're talking about here. This was a short-lived test server with new environment, not huge farms. The super high GC stats we see here are the baseline behaviour. More complex scenarios will push this even more.

Thousands of items is small. Factorio doesn't need that amount of RAM when it routinely handle dozen of thousands of items every ticks.

Remember Minecraft is written in Java, not C++. The Bedrock version is written C++ but has only experimental, very rough native server software that is almost never used.

> Factorio doesn't need that amount of RAM when it routinely handle dozen of thousands of items every tick.

It might not inefficiently allocate all that RAM every tick, but it still has to scan a lot of RAM, and write to a bunch of locations to update the game state.

It loves CPUs with huge caches because of that.

The world is 100k blocks to a side, with players split up in groups of 20 over the whole thing. Since allocation rose steadily from 35 min and peaked from 47 to 63 minutes, I'm guessing they were spread out pretty widely.

There were 113 regions in one of the pictures- call it 9 players per region, with ~927 live unique chunks (8x8 chunks for 9 players ~=566, plus a 19x19 area around the world origin thats always loaded). From what I can tell a Minecraft block is a bit over 10 kB. So ~10 MB per region.

If each thread is allocating 10 MB for each region on each tick (which the screenshot shows is 8 ticks per second) *that would work out to 8.6 gigabytes per second*. IMO, that's too close to be a coincidence- I'm thinking there are a lot of shared chunks per region, but the threads are copying significant amounts of data by value or they just have an absurd overhead.

The Minecraft Java version is notorious for thrashing memory. It allocates and discards objects at an incredible rate, even when the player isn't doing anything. The server has a GUI where you can watch the memory allocation with its distinctive sawtooth pattern.

I'm excited for Project Valhalla to affect Minecraft performance; a lot of the temporary allocations are things like BlockPos objects (a Vec3i, basically) and VoxelShapes (a tree of axis aligned bounding boxes; so, like, an array of a struct that's six f64s) which seem ripe for becoming value objects that are inlined by HotSpot instead of living on the heap

Why not pool these objects? I'm primarily an Android developer and it's a well-known easy optimization on Android to avoid short-lived objects as much as possible, but especially during drawing and other actions that run every frame. You just don't use the word "new" in onDraw and other related methods. Android Studio would even warn you if you do.

But then the new APIs in JDK itself are designed such that you have to allocate loads of short-lived small objects. I was told that HotSpot does deal with them reasonably well to avoid them degrading the performance, but apparently it isn't very good at it?

Allocation is typically really cheap, maintaining pools for objects would likely have more overhead. And while collecting garbage takes resources too, it's heavily concurrently, especially in GCs like Shenandoah and ZGC. So instead of more overhead due to pooling on the thread that uses the objects, you have more overhead on a different thread during garbage collection.

So while it makes sense to avoid unnecessary allocations by using different APIs (e.g. not creating Streams in hot paths), pooling brings far more new problems with it. It might make sense for large objects, but generally requires in-depth analysis to make sure it actually helps.

Also, when plugins come into the equation, you need to make sure that those can't modify objects they aren't meant to modify, which involves copying of objects. Additionally, some objects have different representations in the API (what's used by plugins) vs the implementation (what's used by vanilla minecraft), so converting between those representations is another source of allocations.

Pooling would have been more overhead than allocation + deallocation? Do you have any relevant readings?

No idea how to do it in java but a few pointers ought to be enough. You can also omit clearing the memory area between allocations for things that aren't security sensitive, if that is done in java, which I would assume.

Allocation + deallocation might have more overhead together. I'll try to rephrase: Requesting an object from a pool might have more overhead than allocating a fresh object, and deallocating the object doesn't happen on an application thread but on a GC thread (depending on the GC, obviously). Alexksey Shipilev has good resources how GCs in HotSpot typically allocate objects (https://shipilev.net/jvm/anatomy-quarks/4-tlab-allocation/).

There definitely are scenarios where pooling might make sense, but basically the low hanging fruits in that area in minecraft are already reaped.

I got that, and thanks for the link. But I'm not convinced a pool would have more overhead. It is basically the same thing (not really, but then again, pretty much) just you can encode more information about the usage than a general GC can possible do.

For sure there are other tradeoffs, and whether it is worth it. But when we actually see many GB/s that is not cheap even if you are able to offload to other cores.

The article also concludes:

>It is funny to consider that having TLABs is the way to experience more frequent GC pauses, just because the allocation is so damn cheap!

Sounds like a nightmare, the problem just snowballs, because now you'll be tempted to get into GC tuning.

Seems like there should be a performance benefit from not constantly blowing out your L2 cache. If you can keep your object pool hot there should be quite a bit of performance to be gained. The downside is that this would require active memory management (explicitly freeing the objects when you're done with them) and if you have that why bother programming in a GC language in the first place?

For short lived objects, heap allocation is probably about as fast as allocating on the stack. By pooling you can end up moving objects out of the fast eden space into older generations.

I worked on optimising a java library a few years back and one of the bigger speed ups was removing all the object pooling code.

But that means that the pools still use the GC? If so then I fully agree. But the point of a pool in my mind would be to reuse the memory rather than allocate+deallocate it.

Agreed. Allocation is pretty cheap, but it can still be slowed by an order of magnitude by two things:

1. by using finalizers

2. by using object initialization blocks

If you avoid these two problems you get nano-second performance, because the JVM does not need to run code during object creation and GC.

> not a criticism of Minecraft by the way, it's better to be popular and inefficient than not exist

I'm glad you said that, it's a very wise statement. It's the origin story of so many working-but-inefficient projects. We just don't see the graveyard of perfect-but-never-finished projects on the other side of the scale.

This is not an issue with the runtime. If anything, the JVM is handling this really well.

The problem is code like:

while(game_running) { position = new Box(x,y,z) }

instead of reusing objects:

while(game_running) { position.set(x,y,z) }

plus, Minecraft is not static! There are many thousands of other things moving and changing state in the world.

An age old question... Is the code wrong for not being optimal, or is the compiler/JIT wrong for not optimizing it? I've seen optimizing compilers like clang, gcc, rust optimize constructs similar to this to the same machine code. Ideally you want your high-level stuff that makes code easier to understand to be completely transparent and not have a performance impact when the code actually runs.

For my simple case, it would. Generally, the code around the allocation is complicated so maybe it's not getting optimized for some reason. Or maybe the object is passed through some complicated path... or maybe it's added to a list and then cleared at the end of the frame...

Neither is wrong, and there's very likely a lot of low hanging JVM performance fruit here still to claim.

Right now they're in a bit of a transitional period. Since a few weeks ago the most obvious thing to try is using the latest Oracle GraalVM with ZGC. ZGC is a pauseless GC like Shenandoah and the Graal compiler is a lot better at escape analysis and removing allocations than the stock C2 compiler especially now Oracle made the enterprise compiler free to use.

The main problem is going to be that ZGC Generational is not quite launched yet. Without generational GC it may not be able to keep up with those very high allocation rates, even with a better compiler reducing them down again. Generational ZGC should be in Java 21 but then you'd have to wait for GraalVM to catch up. So, probably it's worth trying that combination in the next six months or so.

Whilst I don't know about ZGC vs Shenandoah, the performance improvements from using Graal EE over C2 can be large even for Java (the wins are much bigger still for other higher level/more dynamic languages). But the Minecraft community isn't really known for adopting the latest JVM tech. They're pretty conservative.

Edit: someone downthread linked to https://github.com/brucethemoose/Minecraft-Performance-Flags... which talks about all of that so I guess they're getting more experimental!

> This is not an issue with the runtime

Yes. But no. But yes.

The code couldn't been written better to allocate less. But the runtime encourages cheap allocations you don't think about. But the runtime could provide/encourage tooling that makes it hard to make that mistake. But...

It all overlaps. Sure, JVM is cool and handles it, but also what JVM is influenced how people instinctively used it.

Most people don't write Java games this way. If you look at any libgdx code examples people cache objects and use polling - you don't know how modern a VM the user will have especially on Android.

Minecraft Java used to allocate over 200MB/s per player when moving, and just 50MB/s when static. A lot of it stems back from Minecraft 1.8, where code was updated to take a BlockPos(float x, float y, float z) class instead of just (float x, float y, float z). Pretty much every single object in the game has to deal with its position. Millions of objects allocated per frame, only to be discarded later. Along with a looooot of other questionable choices. It's not even a runtime problem: you'd have the same issues free()'ing that memory on your own. Just stop allocating so damn much.

But then again, Notch never claimed to be a great programmer.

the post-1.8 codebase awfulness has nothing to do with the source programmer. hell, by 1.13, the game had practically been rewritten at least once over.

Just use a struct... Oh wait, Java does not have structs....

Yet.

IIRC the introduction of BlockPos post-dates Notch. The style of just passing three floats around was his original style.

Notch was well aware of the issues around allocating a lot of small objects in Java. He used to write 4k games with a byte code assembler. Implying technical knowledge of the platform was not the problem.

Notchian code ran way better than modern MC - even though it used completely obsolete OpenGL 1.1 fixed-function calls.

They've upgraded to OpenGL 3 and shaders now; but the game's memory and CPU usage is through the roof with inconsistent performance and huge lagspikes.

Most of this is caused by utterly spurious and unneeded allocations; stream code on hot paths and complete overengineering and abstractions totally unfit for purpose. By that I mean heavy usage of inheritance and indirection in rendering code, and horrible memory wasting with vertex data. (Most things are full floats even if they could be packed and in some cases, even duplication of the same data due to bad data design)

Do you have any source for that? It seems questionable at best. Also, depending on where it is used, Java deals very well with indirect calls, it doesn’t have the same cost as, say, a C++ virtual call.

Source for which section? I am basing this on mostly my experience and codedigging - I don't think there are many written sources about observing this other than anecdotes on forums or reddit or whatever.

With that being said, I'm happy to investigate in detail if you can tell which parts you are interested in.

You are right, the JVM has many optimisations built in to de-virtualise calls and avoid allocations and the such, but if you abuse it, the JIT will give up. I don't think any JIT will de-virtualise or perform escape analysis on 5-6 layers of indirection with branching.

Do you think a minecraft server (and multiplayer game servers in general) can benefit for running on a soft-realtime OS like Linux with the PREEMPT_RT patch?

Games often use their tick rate (iterations of the event loop per second) as a quality metric, and an rt OS should favor preemption and low scheduling latency over throughput... so maybe that should give a more fluid experience?

But the bottleneck for minecraft seems to be in memory allocations, so an OS that can schedule threads rapidly may not change much after all.

First, Minecraft would have to actually support multithreaded work properly. Here the issue is just plain throughput / computation. RT usually makes things slower, but more predictable. MC needs to go faster first.

it all depends on if you want the tick time to equal the wall clock time.

Has anyone already tried to use GraalVM with it? Cool would be trying to do a native compilation and compare the performance

That's a sample size of one, but I did try it once and it ended up being basically the same. HotSpot took a measurable amount of time to reach GraalVM performance (in the minute if memory serves), but once the hot path got JITed, they were basically on par. Since a minecraft server is (usually) a long-lived process, it does not matter in the end.

I've tried different JVMs, different garbage collectors, etc, etc. They all make negligible difference.

Well, Java isn't really used for perf critical workloads to be fair.

I'd rephrase that as "Java usually isn't the first choice for perf critical workloads". It's still a very performant language if used correctly (outperforming C in some benchmarks where the JIT can shine), and used for lots of performance critical stuff.

Java is absolutely used for perf critical workloads, hell, I am fairly sure that there are no runtime in existence that would handle that load of memory pressure better - of course the problem is sort of self-made, not every allocation should have to be done in the first place in this case.

But Java is heavily used in high-frequency trading, is the backbone of many cloud infrastructures and it is such a large platform that there are plenty of small niches where it is used and performance is important.

True, though in any competitive shop it's still second to C++, as they can afford it.

Isn’t the niche where truly insane latencies are required absolutely dominated by ASICs and a general purpose CPU is way out of the picture?

I believe so

Reply on news.ycombinator.com