Hacker News Reader

Comment by cosmic_quanta | Hacker News Reader

It's not super relevant, but I was poking around their CI infrastructure, and I noted the use of a temporary RAM disk to speed-up tests:

https://github.com/pypa/pip/blob/master/.azure-pipelines/scr...

I'm very surprised. Is this common?

View on HN

Yeah, I've seen that pattern around and also used it myself. Useful if you're using the disk a lot, for any purpose.

Let's say you have 200 JS libraries that you need to install dependencies, build and test for each commit. Making the JS dependencies install into a common cache directory into /dev/shm will easily speed up the install phase (and make the projects share dependencies cache fairly easily), and you could also make the test data (like fixtures) go through /dev/shm if you need it to.

Another cool trick - use ephemeral memory SQLite database during unit tests:

conn = sqlite3.connect(":memory:")

But also run against your prod-like database (Postgres/MySQL etc.) later in the CI pipe to catch idiosyncratic database problems.

For Postgres, there is also disabling fsync delays (20x speed claimed) and even running it all in memory.

https://dev.to/thejessleigh/speed-up-your-postgresql-unit-te...

It's a fairly straight-forward optimization if disk is a relevant part of your run time and you don't need terribly much of it. Probably not exactly common, because many people don't bother, but also not extraordinary.

I'd bet its becoming increasingly popular/convenient to do so with the growth of container-based test suites - really trivial to include a "postgres-ram" image for your test stack instead of one that uses disk, and you don't even need to know what tmpfs is to do so.

Maybe even more so on cloud instances?

If the tooling you use lets you create tmpfs mounts for your container its very easy, yes. AFAIK those are RAM-backed by default. I think from inside a container it could be difficult to have the permissions necessary.

Pretty common if you run a lot of builds, have a big test suite or pay per minute.

In my environment unpacking a 200MB .tar.gz containing node_modules from previous runs takes half the time on RAM disk vs. SSD local storage. Python doesn't gain as much, packages tend to have fewer small files but there still is a small gain.

How much tests benefit from using a RAM disk depends heavily on what they do, I have seen 10% for React apps.

We also use other simple tricks like running databases with disabled fsync, mv instead of rm, cache whatever can be cached between CI runs, shallow checkouts and so on. Nothing ground breaking but it all adds up.

However, for pipelines which run for five minutes twice a day I don't even bother.

Yes, it has secondary benefits like not burning out SSDs and wearing out HDDs.

The drawbacks are that memory is kinda expensive in high amounts, and compilers love memory.

These days, “high amounts” is a lot more than most projects need - even ultra-portables have 8GB or more. Most projects don’t have unit test suites limited on file I/O measured in gigabytes unless they also have a budget.

Was very common in the spinning rust days to speed up io bound ops, and mentioned in yesterday’s Amiga thread as built in. Could still be useful if tests use a ton of disk.

Yikes! I'm surprised by the number of people who think temp RAM disk for QA testing came from using containers or the cloud. It was actually in use prior to that technology. And, it was popularized in the use of remote PXE installation which required modifications to the OS after the fact.

Where do you see anybody thinking that "came from using containers or the cloud"?

My contribution is correct. I'm not a parrot. Temp RAM disk started in Linux 2.3 (possibly older than that), it didn't come from builds inside containers or instances in the cloud.

Reply on news.ycombinator.com