> A potential way around this is virtualization, to run multiple operating systems on the same machine.
What’s the actual issue? Linux can have trouble servicing many page faults in parallel in a single process (technically mm) due to lock contention. Multiple processes should reduce this contention.
Do you have more details?
As you wrote in the article, you still have the old server that can support your current load. That likely won’t be an option in the future as your load continues to grow.
I'm also feeling out what's a way of working with this machine that isn't a huge pain in the ass. When you've got one instance running on one machine, manual deployments is fine, but I think something more CI-driven is probably going to be necessary to keep sane with 8 index shards and a test environment as well.
Kubernetes would be an option but I have bad experiences with that too. A bit too much spooky action at a distance for my taste. Whole ecosystem feels very fragile and churny in a way I'm not very happy with, and the abstractions designed for hiding away the complexities of dealing with a cluster make running it on a single machine where those abstractions aren't necessary just pointlessly awkward.
Congrats on the recent success by the way ;)
They’re not, though. There’s a tree of “page tables” (that is, a tree with branching factor 512 or so, in the format used by the CPU) per process. Also, per process, there’s a tree of VMAs (the logical maps from contiguous virtual address ranges to whatever logically backs them) — these are created by mmap and friends. And, regrettably, a lock, also per process (although this lock is a read-write lock, and page faults are reads).
If you have a whole bunch of processes mmapping the same file and thrashing against it, you could end up with contention for that files’s data structures that track mappings, but that seems unlikely. Mappings of different pages of ordinary memory should scale well.
And a VM, for this purpose, is more or less like a process. QEMU (or whatever other userspace host you use) literally maps everything that the VM logically maps, and VM faults are handled as though QEMU triggered a page fault.