Bcachefs Status Update

I worked for a time at a game studio (many times actually, but we’re just talking about this one time specifically) between 2016-ish and early 2020, and got involved with a project that had absurdly long build times – 7+ hours per build per SKU. Woefully under powered equipment to run the CI/CD on. Performed obvious h/w upgrade, but build times were still measured in 2+ hours. It was Unity3D, and it will do mysteriously inscrutable lengthy builds because some asset changed, which causes a cascade of other assets and modules to be rebuilt. These builds are too big to fit purely in to a RAM disk on any reasonably priced h/w I could convince the powers that be to obtain, at least at that time, given that I was a fresh face at the game studio and didn’t have the political capital built up yet. Also, they were tight fisted when it came to spending money on h/w.

Using initially bcache I was able to get the build times down to 35-ish minutes per SKU, some would still take 90-ish minutes because of a big asset check-in from the art team. I was mostly using bcache as a smart RAM disk – files permanently stored on HDD, with an SSD tier (256GB or 512GB IIRC), and then a RAM tier. It was a little brittle at times, we lost a few builds, but for the most part everything worked, and losing a build due to running in to an issue with bcache (very rare, usually attributable to PEBKAC) and with more tweaking, got the build times for each SKU under 20-minutes. We had about 30 SKUs for iOS, Android, Windows & Linux and almost everything was Unity3D for the game engine.

From there I started working with bcachefs; same tiering setup: RAM -> SSD -> HDD. Unity3D, as explained, does mysterious builds, and is single-threaded, and needs to grind its way through 15GB of high resolution art assets in PSD, MB, AI formats, run custom build scripts, etc. I have a few machines, with a little digging in to the Unity3D project indexing format I was able to make it so that a machine could take a few assets, fire up an instance of Unity3D, “compile” the assets, then send them back to a common cache. This is technically how Unity3D works – cache assets previously compiled – it’s really bad at it most of the time though, and at the time, when switching between target platforms, it would blow away its own asset cache. I could use bcachefs, because of the CoW, to keep a “safe copy” of the Unity3D cache on a HDD, make the entire project cache rapidly available to a Linux build agent to determine what assets needed to be rebuilt, and then when that was done, move the project cache to a MacOS or Windows machine for final assembly. And with more tweaking, and some horrendous Bash scripts, got our average build time under 7 minutes for our project. This was a multi-vendor project, and our studio became the studio that mastered all the builds and produced them sooner than anyone else.

Because of that work, the Director of Engineering said “can you do that for our other teams?” and that began another epic quest, but those other teams, they obviously had their own way of doing things and were resistant to change. I was first able to move them to their own individual installs of TeamCity running in a Docker container for doing the CI/CD, and then VMs (Linux, Windows, MacOS (shhh! Don’t tell anyone I virtualized our build Macs for our non-release builds on non-Apple h/w)) to build each game for each target platform with each game’s odd environmental quirks, e.g. MacOS for iOS and Android, Windows for PlayStation and XBOX, Linux for some other thing. Well all this is great but I was tasked with trying to squeeze a gallon in to a pint pot – “We should have a dedicated build server” said I. “These are the specs we need with many cores and much DDR4s.” And then a bean counter said “naw dog, I get by with this little 13” laptop for running my Excel, how about a nice quad-core with 32GB of RAM, and we’re being generous here. We’ll even throw in an SSD if you behave.”

Eventually, we got a machine with hundreds of cores and hundreds of gigs. And I couldn’t figure out why it didn’t perform how expected until I looked at the SSDs and talked to people and they said “We saved $50 buying our M.2 from Amazon, you should be happy, it was _twice_ the size of that one you specced.” And thus began a lengthy discussion of why I specced a U.2 enterprise drive with a fucking huge queue depth and no write-cliff. Once I got the h/w I wanted, well, we cannot run RAM disks in each individual VM because that will eat all of our RAMs. So I looked around and found a company that had a tiering driver, RAM -> SSD -> HDD and I said “hey, that sounds like bcache!” and I reached out to the company and they said “sure, it’s a $1,000. Per year. Per CPU.” Well that ain’t gonna fly, I just convinced the company to buy me $12,000 of h/w and two VMWare ESXi licenses, they aren’t going to spring for $4,000 a year in software licenses. So I did what any delusional developer with balls bigger than his brains would do – I said “I’ll write my own! It should only take a weekend or two!” Hahaha! Ha!

I started with the bcache source, and over several weeks of late nights and weekends was able to build it as a storage driver for VMWare’s particular flavour of Linux with all of its badly documented internals. I cribbed a lot of notes with the help of BinaryNinja and that other company’s bcache-like driver.

With bcache now loading as a storage driver in to VMWare, I could set up a group of VM’s and point them at a bcache (RAM -> SSD -> HDD) storage area that contained all the data each VM, in whatever client OS and target platform would need to create a binary build. And because it is bcache, well, we’re both caching an awful lot of very similar chunks of data, and also effectively giving each VM its own resizable smart RAM drive, without having to setup a RAM drive inside of each VM. The downside of this is that if my hacked up bcache vmx driver were to crash, or the SSD go offline, every VM using that for storage comes to a screeching halt and loses whatever data is stored. But that didn’t matter, because these are build machines – OS and tools stored on one virtual drive – build directory containing source code, assets and binary output stored on a less reliable, but far faster virtual drive. If my driver crashes, we simply restart each VM and the start-up script for the VM will recreate the bcache’d storage tier.

Right around this time Apple was dropping hints that they’d be moving away from Intel h/w, and also, for final builds on Apple h/w, things ran as slow as molasses in a Minnesota winter – mostly due to the fact that these are final builds and nothing to do with the fact that the bean counters decided that an underpowered five year old Mac with a 5400RPM HDD and 8GB RAM was sufficient for a critical piece of company infrastructure. 8GB isn’t enough RAM to create a RAM disk to hold the entire project in, nowhere close enough, but MacOS has this concept of a FusionDrive, whereby they would sell you 1TB HDD with a 32GB SSD glued on to it, and unlike several of the hybrid drives that are available in non-AppleLand, Apple had the “wisdom” to build the ability to make a FusionDrive right in to the OS – it was the OS file system driver that directly handled the gluing together. I was able to use that built-in facility to make a FusionDrive that glued a RAM Disk and an external SSD together which in any sane environment would be an absolutely fucking terrible idea. But for a machine that just spits out builds all day long, and doesn’t care if the entirety of the SSD contents are wiped out due to someone tripping over a power cable or unplugging the SSD or turning off the machine, it didn’t matter, just run this FusionDrive script at start-up and everything is back as it should be.

FusionDrive script here: https://github.com/JustinLloyd/fusion-drive

There was other fun things around that CI/CD project, such as how to get Linux to use a VPN where the client IT department has determined that only macOS, Windows, iOS and Android are permitted to connect. "Well we just send the magic strings that tell the remote VPN h/w that we're really an iOS device and we don't need any of their spyware installed thankyouverymuch." Interesting side-note, got a C&D for sharing that information from "large vendor of network h/w." Further interesting note: Engineer working at large vendor of network h/w posted a script on github on how to do that exact thing under their personal account about two months later.

Checking for a lock ordering violation is simpler because you only have to look at data structures for the current thread - you're just looking at what other locks the current thread is holding.

To check for a deadlock you have to do a full DFS of threads blocked on locks held by other threads, looking for cycles. Quite a bit trickier, but it was well worth it :)