Folia – Multithreading Coming to your Minecraft server | Hacker News Reader

Folia – Multithreading Coming to your Minecraft server

paper-chan.moe

125 pointsthunderbong69 comments

The modding/custom server situation with Minecraft is so exhausting. I can't believe after all these years there is still no official modding or plugin API.

So now we have Folia, which is a fork of Paper, which is a fork of Spigot, which I believe was originally a fork of Bukkit. Nice. I guess people don't want to upstream patches for some reason.

We kinda do... in a limited fashion... for Bedrock.

Java MC's development had always been extremely conservative, and it always will be under Mojang. Its both saved it from going downhill and weighed it down.

It's a shame Bedrock modding (behaviour packs?) is so limited. The performance is so much better than the Java version. It's also a shame it has Xbox shit and microtransactions shoved in.

Well the Java version is so moddable because its Java, thats almost impossible for Bedrock unless it goes open source or moves to unobfuscated C# (which is impossible because of the MTX of course).

Bedrock has IAPs now? I mean it was useless anyway for running a private server, and only runs on Windows, and automation is limited because they crippled the world update distance to have the omg raytracing! or whatever.

And now it has IAPs too?

Bedrock has had IAPs for many years. It's fine for a private server and they offer Windows and Linux versions of it (client side it supports pretty much every platform except Mac/Linux, including oddities like ChromeOS). Just don't go running it expecting to set up what you would on a Java server - it's two different environments. World update distance default is lower in Realms but can be configured on a private server or client hosted session. It's not really related to raytracing, simulation range has little to do with that.

Both clients/servers have their ups and downs. I do wish they'd release the Bedrock client for Mac/Linux though. Then all platforms would have an official way to interop.

> It's fine for a private server

Land claim / griefing protection? Multiplayer sleep? Custom recipes? Dynmap? Multiverse? WorldGuard rules? :)

Edit because I can't reply any more:

> to use a 3rd party bridging solution with it instead like GeyserMC+Floodgate

All that for the privilege of running the Bedrock client on a Mac with Crossover, I guess. Nah, they can keep it. I didn't even claim the free Bedrock when they were giving it free to java owners.

On the server I play we're looking forward to testing Folia so redstone addicts will only lag themselves and not everyone btw.

> Just don't go running it expecting to set up what you would on a Java server - it's two different environments.

That said multiplayer sleep, custom recipes, bedrock-viz, and others exist. Customized protection I've found to be lacking though.

But honestly if you're looking at that many 3rd party things you might find it easier to use a 3rd party bridging solution with it instead like GeyserMC+Floodgate. It still has limitations compared to a native java (client, server, mod) tuple but it's probably easier to adapt to if you're used to java server.

Sorry I just can't figure this out, what is an IAP?

In-app purchases

Does modding need to mean actually modifying game files? When we say modding, we generally mean adding new blocks, items, and behaviours no? Surely this could be done with a public API? It might not give you quite as much freedom as existing mods but you could get most of the way there.

I'm not sure you understand how mechanically complex MC modding gets.

https://www.youtube.com/watch?v=rR8W-f9YhYA

... and that is just one mod. There are mods for interplanetary space travel, mods for modular portals and matter-energy conversion, mods for full z height terrain gen and massive premade structures, mods for virtual NPCs and towns, "backend" mods that overhaul the render, light engine, chunk loading and such. You cant get that with simple block APIs (which Java and Bedrock have to a limited extent).

And we do have modding APIs, like Fabric and Forge. They are just unofficial.

That being said, there is a better middle ground between the current extremely limited official APIs and the very permissive unofficial modding APIs, but it will never happen because it would interfere with Bedrock's MTX, and Java's development is too conservative.

You seem to underestimate the amount of modding that vanilla Minecraft servers have, such as Wynncraft or Hypixel Skyblock. They turn Minecraft into a completely different game and most of the changes are server side.

Also, Fabric and Forge are not modding APIs. They sometimes provide extra APIs on top of the base game to make mod compatibility easier, but you are very much supposed to just mess around with the Minecraft codebase directly and directly reference Mojang classes in your code. If all you want to do is add a new block in Fabric you don't touch any Fabric API.

Yeah no question that server only modding is huge too. Its like 2 completely different ecosystems.

Also Java performance doesn't have to be so horrendous, that is Mojang'a fault. They could iron it out with a tiny fraction of their sales.

That being said, the low hanging mod fruit they could incorporate is often poisoned by GPL or some other weird license.

Why isn't there a mod to iron out the performance if its so trivial?

There are many such mods. There have been for years.

https://github.com/TheUsefulLists/UsefulMods#performance-and...

See https://github.com/Fabulously-Optimized/fabulously-optimized...

There is no reason for Mojang to copy mods. It is better for them to develop their own solution so that they can maintain parity between Java and bedrock edition.

Well they have had the opportunity to (for instance) overhaul the renderer or thread dimensions since 1.0 in 2011, and we are still here tapping our feet.

They have tried to pull in mods before, but unfortunate drama that ensued probably soured them on that idea.

I remember, back before Mojang was acquired by Microsoft the devs mentioned working on a modding API, and how it would allow for all kinds of things.

I have a feeling such plans were killed off during the acquisition, as it has never been mentioned since then. Even now it would be simple for the devs to allow players to add new blocks and items via server sent resource packs without needed a modded client, and yet for some reason they do not.

I still have to thanks Bukkit/Spigot for getting me in to Java in the first place tough, and thus my current position as a Java dev. I don't follow it as often but I still maintain a small server my friends play on.

There's a lot of drama there.

Basically, there was Bukkit. In 2011, Mojang hired a bunch of Bukkit's developers, but didn't tell anyone that they also legally took control of the Bukkit project. This was because Mojang didn't really want Bukkit to be their official API as much as being a temporary API for the time being (and also didn't want to have anyone assume they would be responsible for it), but from that point, Bukkit was legally their project and the community didn't know.

Then, Mojang changed their EULA to ban Pay-to-win servers. This caused a lot of controversy, and one of the Bukkit developers left. Mojang then revealed that, yeah, they owned Bukkit for years, so they didn't need him. Another developer, feeling betrayed, who had contributed to Bukkit, submitted a DMCA takedown against his own code bringing Bukkit down with it, angry at having done "free labor." (You might say, "but the GPL!," but being tricked into a contract under false pretenses can legally invalidate contracts, making the GPL-license for his contributions potentially invalid.) Mojang panicked at the takedown and a ton of developers abandoned ship, working on Bukkit's community fork Spigot. Mojang then 2.5 months later brought Bukkit back and that time really just threw it into the community's hands.

The result of the aftermath was everyone had restructured around Spigot, based on Bukkit; and Mojang wasn't going to deal with APIs and open-source stuff for a long while afterwards. (They did later try purchasing parts of OptiFine, but the developer wanted compatibility to be all-or-nothing which Mojang wouldn't accept.)

https://www.reddit.com/r/Minecraft/comments/6g1tqn/psa_no_bu...

https://blog.jwf.io/2020/04/open-source-minecraft-bukkit-gpl...

https://www.reddit.com/r/Minecraft/comments/2fk5nn/my_respon...

The DMCA was since Bukkit is an API while CraftBukkit is an implementation of Bukkit. Bukkit is GPL meaning you can't make a combined work with proprietary software - such as Minecraft, which is what CraftBukkit is.

There is a 9 year old song that was making fun of the modding API taking seemingly forever back then already.

https://www.youtube.com/watch?v=G-zthYFjk6s

Actually, this is absolutely possible today using data packs. They're the successor to the modding API project and they provide a very simple declarative JSON-based syntax for adding new blocks, items, customizing world gen, adding new commands, advancements, and even custom entity behavior (using the flexible tag-based entity data system).

Did a quick google search, it still does not seem to be possible to add using the namespace system they implemented in 1.13 (supposedly for such a reason) and datapacks. Currently, it looks like 'custom blocks' involve using a hacky method involving invisible armor stands and item frames holding items modeled as a block. Which means they count and behave as entities instead of true blocks, and this can be an issue since too many entities will lag a server.

Data packs and resource packs certainly seem very promising as a modding API, but it still seems the devs are artificially restricting their usage.

Ah, that's fair, I knew lots of datapacks used custom entities but didn't realize that custom blocks were entirely impossible. I agree it seems like a strange limitation. I guess the biggest issue is that while custom blocks are easy to make, custom block behaviors are much more complicated to implement outside of code compared to custom entity behaviors, which are much more customizable as a tradeoff for being much less performant.

It didn't die, it evolved. Mojang began taking cues from the modding communities, improving the internal APIs, making the overall codebase more generalized and robust, moving model data to declarative JSON, adding registry systems instead of hard coding, introducing additional resources namespaces by default, resource packs, data packs, simpler creation of custom blocks, biomes, and most recently opening up Mojang's own class and method name mappings so that the insane process of mapping obfuscated Java class and method names can be stopped, allowing mod frameworks like Forge et al to have consistent API making and to reduce the previously huge amount of time needed for mods to get ported to new versions.

The biggest problem with the "mod API" is that, effectively, you'll need to open source Minecraft entirely to satisfy the kinds of changes mods make to the vanilla experience, or else everyone will keep using the patch systems we are building on now. It is discouraged unless absolutely needed, but mods can even apply bytecode level runtime patches on the vanilla classpath to enable functionality that the mod framework hasn't yet exposed.

As an addendum, other games get away with a "designed" modding API because the alternative is too difficult, as most games are built in C/C++ which is nowhere near as malleable and discoverable as the Java runtime. It's also why mods for Bedrock (the C++ implementation of Minecraft) is barely a thing.

Isn't it also that Bedrock is typically a Windows Store App, which are very very locked down?

This exists, it’s how the much more popular Bedrock edition’s addons work.

- "Folia, which is a fork of Paper, which is a fork of Spigot, which I believe was originally a fork of Bukkit"

We're not old, the forks just fly by so very fast

Yeah, but Bukkit is ancient - so ancient, upstreaming is basically impossible and even discouraged because it would break compatibility with the few people who still need it. It's really only maintained as a reference architecture by Spigot because it helps with initial Minecraft release reverse-engineering.

Spigot, meanwhile, is run by one developer who only posts his code within a few weeks of a Minecraft launch after working on it completely privately. Paper has examined taking over that role, but has decided against it for the time being until they are ready to fully take on that job because... it's not an easy job and he's basically mastered it (Paper would be much slower if they were doing it themselves for the first few releases). Also he runs the plugin repository and Paper isn't ready to launch their own yet. Also, Spigot and Bukkit have this ancient API that not many plugins use, but a few do... and it'd be nice to not break them.

So Paper is the first remotely reasonable downstream with a big open-source community for it, which then makes it more like... one fork. And even though Paper is technically reliant on Spigot (and Spigot on Bukkit), Paper can really cut them off whenever convenient. And there's a good reason to make Folia a fork - the way it works under the hood is very different than a traditional Minecraft server, and it basically breaks all the Spigot and Paper and Bukkit plugin compatibility.

In other words, grass roots open source development is hard :)

Minecraft server forks almost seem to develop as fast as Javascript toolkits. It's impressive, in a way.

I think this has to do with the way that ultimate, true upstream involves the closed source Microsoft components. None of these servers seem to be built from the ground up, each extending an existing package (closed or open source) in one way or another.

What exactly would the point of an official modding API be, even? Anything Mojang could provide would inevitably be inferior to what can be done by directly modifying the game's code via existing third party mod APIs.

Folia is a complete re-write of Minecraft's core game loop to support "true multithreading and regionalized ticking" ("ticking" being the term for a single turn of Minecraft's core game event loop). Even the most ambitious official modding API in the world would never ever contemplate supporting something as thorough and involved as this. Minecraft does in fact have an official plugin API, known as data packs—they're obviously more limited then the ability to completely rewrite the game's code, but they can create new blocks, items, entities, customize their behavior, change worldgen, add achievements, and more. Here's an example of a "bendy pistons" mod that was built entirely using the datapacks API, with no mods required: https://www.youtube.com/watch?v=_Wk7ZUYkrpI (and 7 more features from the original video: https://www.youtube.com/watch?v=2LRvq6H11bs)

Oh my god, bendy pistons are ridiculous. Thanks for that.

> The modding/custom server situation with Minecraft is so exhausting. I can't believe after all these years there is still no official modding or plugin API.

> So now we have Folia, which is a fork of Paper, which is a fork of Spigot, which I believe was originally a fork of Bukkit. Nice. I guess people don't want to upstream patches for some reason.

Time and effort you put into upstreaming patches is almost always wasted, either you get it included and then have to maintain it forever or it will get removed due to bitrot or developers will just say no and all your effort was wasted. Don't forget that developers might also want to share their opinions about you based on your commit. There is no better way to build community than saying that no one cares about you inside of it.

> I guess people don't want to upstream patches for some reason.

My insanely biased impression based on my experience is that this is just the norm for Windows-centric development. Generally you have no expectation of being given access to source code at all; if software is free then it's "freeware", but for whatever reason the developers don't generally want you modifying it. So many power user tools just died out when their developers quit working on them because the source code was not available, at least not under a free license.

I remember back in the early days of Minecraft (beta+ era) I'd just go on the forum and download random jar files uploaded by whomever, and mod Minecraft with them. Nobody asked for source code. Or thinking about doom9, for years and years the norm for tool development was that you'd download a random exe from someone on a forum, with a single thread for "support". If they disappeared one day, that was it. That's also the way a bunch of Android stuff is/was developed on xda-dev.

The more Linux-centric set of norms has expanded outward into a pretty robust open source ecosystem, but it's by no means the universal norm for free (as in beer) software development. Even in cases like Paper / Bukkit where the source is free, building the software yourself is very much not the norm, often is not even supported.

I've gotten the same impression. So many Windows tools are given away for free, no hint of ever monetizing or anything, just a hobby / service by the developer, and yet no code is provided.

I remember hearing an explanation from one such solo developer with a relatively recognizable brand that he doesn't want people to be able to easily replicate his tools with added malware / adware. But I feel like download sites have been doing that for years anyway.

He has released at least one tiny but useful piece of code as public domain.

Also he does have one product that charges for. I suppose one potential reason to keep code private is to maintain more control of the free tools for purposes of branding and bringing attention to the pay tool? That's just a thought.

Folia most likely breaks compatibility with plugins so forking is to be expected.

If you ignore server plugins, the only effort in actually trying to build a modding API is Quilt and it's Quilt Standard Libraries, which you probably guesses further splits the modding community into Forge, Fabric and now Quilt but that is on the Forge and Fabric guys for not even trying.

Bukkit was in fact preceded by hey0's mod, the project of a 15 year old kid who was largely responsible for bootstrapping the entire line of Minecraft server mod APIs.

After all these years though it is extremely exciting to see this advancement, I'd expect this to allow extremely large servers. Makes me want to start playing again ^_^

Mojang releases the deobfuscation mappings for each version nowadays.

This is, for those who know a little bit about Minecraft hosting, extremely impressive stuff. Stuff I never thought I would see for years. Minecraft has been notoriously single-threaded and breaking it apart into regions, even though discussed for years, has always been a crapshoot whenever attempted - so getting it even close to finally possible in a reasonable and self-adjusting way is astounding.

> provide a no-compromise solution

Except for fabric/forge compatibility. :(

Tick threading with server mods is just a mad dream anyway. Even dimensions threading is apparently impractical.

Paper and Spigot, the parent projects, have never been compatible with Forge and Fabric, and are more popular for servers.

Of course, though there have been some mad projects to try an merge the server implementations with fabric/forge.

Paper + Fabric is currently dead though, so not much hope there.

I wonder how it handles the seams between the regions. How does a player interact with another player on the other side of this border.

If I read the documentation, it appears to be that it will expand the region to accommodate both players or merge the regions, so that you will never have a situation where two players from different regions are visible to each other.

Makes sense, also very cool.

So worst case is you're essentially back to single threaded. With a slight overhead of tracking connected regions.

I guess that makes sense.

I wonder what this looks like on practice, what kinds of play styles will see the biggest boost from this.

I also wonder how this plays with things like hyper long range TNT/Ender pearl cannons. Entities are their own pile of worms, but distant regions can suddenly influence each other in some cases.

My hope is this allows expanding the player count on servers like 2b2t. The normal play style here is small groups of players spread out massively over the map. It would be incredible if you could effectively have each user on their own thread and mini server with it all just seamlessly working.

Would be outstanding to see a server with something like 4000 players online all on the same map.

My mind went to 2b2t as well. Unfortunately players still tend to congregate somewhat around spawn. Perhaps Folia lets them raise the server cap to a degree, but it appears here that you still can’t have more than a couple hundred players in the same area. Allowing more than that limit on the server invites a mess in the event enough of those players end up in the same place for whatever reason.

I wonder if the result would be that it would just lag out for spawn but players outside the area would enjoy a full performance game. That seems like an acceptable penalty.

Check out 6b6t if you haven't already. It's an anarchy server that runs on multipaper, which is a distributed Minecraft server. The owner claims it can support 1000 simultaneous players.

2b2t is a great use case.

Imagine being able to moderate lag machines on a per region basis, it would make it so much easier to track problem players.

I wonder how "bubble merging" would look like. Some hiccup?

I used to play on civcraft and at some point they decided the lag was unbearable and restarted the map. The new map was a number of map shards connected with portals (bungee?). It fixed the lag but getting around got more cumbersome.

Yeah would you roll back the smaller bubble and propagate the larger one? Try and merge them? What happens with 3 bubbles? Is the worst case here much worse than single threaded?

At the moment there are some weird nuances around update order based on world location and direction that the more technical players have learnt to work with/around for the most part.

Predictable update orders are a big deal for those super technical players.

I have thought about this a lot already and basically came to exact same conclusion with the "bubble" system.

But don't the bubbles have to tick at the same time to be 100% vanilla? They still compute their ticks independently but only "roll over" together. Which would still increase performance extremely, don't get me wrong. But will not stop lag machines, since global mspt would be the min of all bubble mspt's. Although it would make tracking lag generation a lot easier.

Stuff is already used to ticking at a different pace because only parts are loading in the world at a time. Lag machines don't work as it will only lag your region, not all regions have to tick at the same pace.

Regions would need some level of synchronization to allow the merging / splitting algorithm to work, I think is GP's point.

So does time advance differently in different regions? If two regions start redstone clocks and one of them has a lag machine, will they desync? Will it get to a point where one is day and the other is night at the same time? What happens when the regions are merged or a player crosses between them?

If I understand correctly, there’s still a global thread for things like day/night cycle, and then region threads handle their own redstone ticks and can indeed fall out of sync.

I see. I wonder how they sync up again and how dynamic the splitting and joining is. I guess long distance instant wire would be the worst case scenario for this system.

That's nice... when you like to use the Spigot API (which, of course, is used by the majority of servers)

Something like this for Fabric would be great though. Server-side modding with Fabric offers a lot of opportunities you can't do with the Spigot API.

In fact there have been multiple mods that tried to implement this for Fabric or Forge. It's keeping everything in sync with Mojang's codebase that's the hard part.

Right now I just use DimensionalThreading. Putting different dimensions on different threads is quite easy to do.

> It is recommended to have a minimum 16 physical cores in order to fully benefit from Folia.

I didn't expect this. I wonder how many Minecraft servers are using more than 16 cores in production today.

I'd assume minimal numbers since the existing software is mostly single threaded. The comment seems to suggest that to make use of the new multi threading, you have to first have some large number of threads to actually use.

This has been attempted by many people in the past, including myself, but I'm sure that if anyone is to have a shot, it's SpottedLeaf. He's done a lot for Paper performance and is very familiar with the workings of Minecraft.

Reply on news.ycombinator.com