A 40ms Go garbage collector pause caused by swap
frn.sh
frn.sh
I believe that's actually the same reason why Apple stopped using GC in their frameworks in favour of automatic reference counting.
Many make the mistake to think there is only one way to do a GC.
One of the authoritative books on the subject, https://gchandbook.org/contents.html
And a quite well known paper on the matter as well, https://dl.acm.org/doi/10.1145/1035292.1028982
That is a tracing GC by the way.
There are also tracing GC implementations with deterministic resource management APIs, .NET and D have them for example.
Anyway it doesn't matter any longer, AI is going to replace most of us, and it comes with automatic everything.
I prefer to consider GC only the methods of memory management where reclaiming the no longer used memory is done either asynchronously with the main program or as late as possible, i.e. when new allocation requests cannot be satisfied.
In the normal implementation of reference counting, memory is freed as soon as possible, i.e. exactly like stack memory, when blocks are exited, so I do not consider reference counting as GC.
The problem with GC in the strict sense is that you cannot predict when it will happen. With both stack memory and reference counted heap memory you know that whenever you exit a block, some time will be spent with running destructors and for freeing memory, but such interruptions will not happen in other points of the program.
Pretty much all the high-performance GC/refcounting algorithms are hybrids in one form or the other; it's a spectrum of choices. https://dl.acm.org/doi/10.1145/1028976.1028982 explores this in some detail.
However, if you have distinct names it is efficient to use them with distinct meanings.
Making "garbage collection" synonymous with "freeing memory" is bad, because it eliminates a means to distinguish various methods for freeing memory.
Like I have said, I consider useful to define "garbage collection" as any method of freeing memory where the memory is not freed as soon as possible (i.e. when a block is exited), but freeing is deferred to be performed at a later time, even as late as possible (i.e. when new memory allocation requests cannot be satisfied).
Indeed, many garbage collection algorithms use reference counts, where memory deallocation is deferred, but when I use the term "reference counting" without any other qualifier, I mean it in the sense in which it was originally defined in 1960, where the time when memory deallocation is run is predictable, exactly like for stack-allocated memory.
I prefer to write programs with well-defined worst-case behavior, so I normally prefer deterministic algorithms. Thus I always prefer to use reference counts instead of GC. I have never encountered a case when avoiding reference cycles was difficult.
Which in an industry where some folks call themselves Software Engineers after a bootcamp, without any kind of accreditation, I rather stay with the definition from those that do language design and compiler algorithms research.
The paper "A unified theory of garbage collection", which has started the fashion of considering reference counting as a kind of garbage collection, only shows correctly that both tracing and reference counting are complementary implementation techniques for a garbage collector.
It does not mention anywhere the essential practical difference between the traditional standalone memory management with reference counts and a garbage collector, which stays the same regardless whether the garbage collector also happens to use reference counts for some purposes, which is the difference between predictable and unpredictable times when memory reclamation is done.
Many of the modern authors of academic papers are a poor model of using computer terminology (or for the terminology in other domains), because very frequently it is obvious that they have not read the old works where such terms were introduced for the first time, even when such works are cited in the bibliography.
Unlike them, I have done an extensive research to find when and where various computing terms have been used for the first time, and I strive to use most terms with their original meanings, not with corrupted meanings, even in the cases when the latter have become more popular lately.
Cedar was already combining reference counting with a cycle collector, as one of the very first systems programming languages with automatic resource management.
It’s the same with ARC. You also don’t know when the counter will reach zero.
In the normal implementation of reference counts, counters can be decremented only at block exits and not at any other program point.
At a block exit some of the local variables that are freed may contain references, so freeing them will decrement some reference counts. Then some counters will reach zero, triggering other deallocations and the decrementing of other counters. This will repeat until no other counters reach zero.
All the memory deallocation happens predictably, only at block exits.
If a variable is not freed immediately when a counter reaches zero, but the deallocation is deferred for a later time, which is not predictable, that is no longer classic memory management with reference counts, but it is a garbage collector, which happens to also use reference counts, probably in combination with some tracing algorithm.
When reference counts are implemented, manual memory deallocation, like with C free() or C++ delete, should be forbidden, but even if it were used that would just introduce other program points besides the block exits, where it is known that memory deallocation will happen.
Yes, that's expected and no not "regardless of implementation": the GC implementation CAN be improved.
See this discussion on reddit: https://old.reddit.com/r/programming/comments/1wf2fei/40ms_g...
Copy/pasted here: >>
I remember a research paper about swap and GC, where the GC cooperated with the OS to avoid this kind of issue. AFAIK it went nowhere, too bad.
[–]andreiross[S] 15 points il y a 21 jours
Are you talking about this one? https://cse.buffalo.edu/\~mhertz/bc-pldi-2005.pdf. If so, yes. Too bad. I don't know the repercusions this paper had in the past, though, in the sense of pros and cons of the bookmark collector. Don't know if anyone tried to actually implement it or design it at some point.
[–]renozyx 9 points il y a 21 jours
Yes, congratulations for finding it. And I don't know either,. Except that they did implement it on Linux (of course) https://plasma.cs.umass.edu/emery/cooperative-memory-managem...
<<
I would go further - I think the whole swap lifecycle needs to be communicated. Before the OS swaps out a page, if it could invoke the GC which would clean up that piece of memory so that we dont end up writing garbage to swap.
The OS should also allow marking pages as piority to stop them from being swapped out.
This is IIUC possible using the mlock(2) family of syscalls: https://man7.org/linux/man-pages/man2/mlock.2.html. (On Linux, though I'm guessing other UNIXen and operating systems have it or something equivalent.)
Any program that has a rarely accessed, but vital chunk of memory is vulnerable to this sort of issue.
"Stop doing that"
If you care about latency, disable swap. System wide or for the specific the cgroup.
If you care about latency, mlock() your memory, do not disable swap. Swap is good and gives the kernel an equal opportunity to evict data and code pages.
I'd rather have applications be oom_killed than having them swap out, the former is rather obvious and demands action.
Disabling swap will just moves pressere elsewhere: to code pages. And evicted code page is no better: full stall while kernel loads that page from disk.
Under memory pressure, the kernel evicts less popular pages from memory. If a page has been mapped from a file, it is dropped (if dirty, then it is written out first). If it is needed later, the kernel can read it back from the file. If a page is anonymous (read: heap page), then there is no backing file and the kernel copies it to swap before dropping it. This is swapping.
So, what happens if you disable swap and the kernel is low on memory? What can it evict? Anonymous pages cannot be evicted: there is no swap to put a copy in. The only choice the kernel has is to evict pages that are mapped from files. Those include pages mapped from the executable. You don't eliminate stalls by disabling swap, you just move them elsewhere: the kernel will page out code and your app gets paused whenever the execution flow hits such a page.
Your userspace early OOM killer triggers and lets you know you're trying to run more than will fit in memory so you don't do it again. (In my experience, the kernel OOM killer can't be trusted to kill processes soon enough.)
I haven't had to analyze the performance of no-swap processes before. My assumption is that code is hot enough to avoid eviction and that evicted code pages are rather the exception. To strong-man the argument, I can imagine long running complex (bloated) services could have parts that are not touched unless a specific request comes in.
When the kernel needs memory, it goes hunting for a page it can discard. But since that's transparent, the kernel can only discard a page if it knows it can get it back (after all, it's still got valid data on it, and maybe you'll try access it again later).
If there's swap, a page full of stale/unneeded data can be written out to swap. But if there's no swap, your page of "dangling data that you'll never use, but is still valid & referenced" can't be discarded; the kernel doesn't know you won't want it later, and it can't recreate the page if it throws it away.
So like sibling said, at that point it has to find other pages it can evict from memory, ones that _do_ have somewhere persistent they can be written out to. Pages loaded from binaries on disk satisfy that, so those will get dropped instead.
You can tell an average Golang user to use mutexes, you shouldn't be telling them to lock memory manually.
This is not exclusive to GC, any program that reads a rarely accessed piece of memory is vulnerable to this.
For example better using the stack, or pulling out the big gun of manual memory management.
I'm sure they had reasons to choose Go when they first designed this project but they don't go into them at all.
Feels like they just wanted to play with a new toy.
Generational GC can avoid some of it by ignoring the old code entirely, but Go does not implement generations because its relatively strong ability to stack-allocate generally replaces the nursery, and for most work loads you don’t recoup the costs.
They explained in the post why this wasn't an issue: they were producing very little garbage, but there was a very large object graph.
> manual memory management
If you need to do manual memory management in a GC language with no builtin support for it, like Go, that's probably a sign that you should switch to a different language.
FTA:
“These latency spikes definitely smelled like garbage collection performance impact, but we had written the Go code very efficiently and had very few allocations. We were not creating a lot of garbage.
[…]
the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references”
In general when you tune knobs for GC, you pay for benefits in one area with sacrifices in another. Two big knobs to turn are pause latency and throughput. You probably wouldn’t want to go full “optimize for latency” because you’d end up with poor throughput. Also vice versa. Java’s reputation for poor GC performance is partly due to historical defaults that tune it for throughput.
Go’s GC is already a “concurrent mark-sweep garbage collector” and already has “extremely low mutator pause times, on the order of tens of microseconds”. It sounds like on-the-fly is just a different flavor of what Go already has.
It's a well known algorithm. Folks who do GCs for a living know about it. The folks who work on Go are surely aware of it. I'm assuming that they do not use it for a good reason, hence my question!
Fil-C's GC (Fil's Unbelievable Garbage Collector) uses an alternative on-the-fly algorithm, which I call Phil's Concurrent Marking.
I've documented it here: https://fil-c.org/fugc
Here's the source: https://github.com/pizlonator/fil-c/blob/deluge/libpas/src/l...
Phil's Concurrent Marking differs from DLG in that it only requires a Djikstra barrier and uses a permagrey stack (something that Go used to do).
However, FUGC does clever things for coroutines (as in ucontexts, which Fil-C supports) - they are not permagrey; they only become grey if they execute. That's relevant to Go because Go moved away from permagrey stacks because of coroutine scan overheads, which the FUGC coroutine strategy might avoid.
But even if Go could not go back to permagrey, then the answer would be to use DLG, which would involve using the combined Yuasa+Dijstra barrier, which Go uses today anyway
If it's only a single fault I'd expect it to be to be dominated by IO latency, which is pretty good on NVMe. As the article shows those 40ms were accumulated over hundreds of pagefaults. The problem there was that there was a stop-the-world pause stalled by all those pagefaults together.
A fully-concurrent and swap-friendly GC you could maybe define that it only increases the active working-set by x GB on top of what the application itself (and the rest of the system) is actively using and will only page-in y GB per second. And it'd do so on GC threads, not the application threads. Whether that would be sufficient to not cause a swap storm would still depend on all the other stuff happening besides the GC.
I'm sure an agent can work on this and get some numbers with a day's worth of tokens.
You could MADV_WILLNEED the GC metadata when you start the GC process hoping they’ll have been paged in by the time you STW, but assuming that area is not massive it’s probably a better idea to just prevent it being paged out.
On a lower level, the OCaml and cpython GCs use a prefetch buffer during marking to schedule around the cache.
For example, why not swap out not by LRU page but by dense node clusters on the heap graph, maintaining in-memory summaries of inbound and outbound edges for liveness? If you do this, you don't have to swap the cluster in to do a GC involving it.
If the whole cluster becomes unreachable, you wouldn't even have to swap it back in to get rid of it: you'd just drop the swap reference and deem the swap space free.
I don't see anything this deeply integrated happening near-term, but it's fun to think about.
I use Rust where i need low latency.
Databases can do this via the page cache since that's basically an implementation of swapping where the DBMS has full control.
Edit: anyway, the problem in TA was that GC metadata had to be accessed. The GC cannot delay this. It might decide to abort the stop-the-world phase so the application can continue while GC metadata is swapped in, but then one runs into liveness issues since next time the GC runs it might already be swapped out again.
(though of course a swap is a swap - but you can "trigger" it depending on your memory or file access pattern)
Not really, reference counting can cause a single object deallocation to trigger an arbitrarily long chain of deallocations.
Variable time yes but you know when you're going to pay it
(I mean yes you can put your gc.run() there as well, but it might not give you the results you want)
The predictable runtime performance is also a myth, because they never take into account the use of NUMA memory, lock contention, possible stack overflow and stop the world in the case of cascaded deletions in naive implementations.
I'm not sure most GC implementations worry about the rest neither (as by the several complaints we see going around)
(makes me wonder who's buying those - things like Azul, etc)
There is nothing "special GC" about it, the failure is to assume there is only one way to implement GC algorithms, as if there is only one way to implement hash tables, tree re-balancing algorithms, .... and then place all languages into the same bucket.
Then we have the modern times with AI driven code generation, where no one cares how their agents are actually doing the work, with what kinds of resource management approaches.
I still think that reference counting is promising though. First because it meshes well with static analysis memory-management techniques such as inference of uniqueness and borrowing -- that can optimise away RC altogether. (and Swift's compiler already does some of that). I have not seen any work that could optimise away tracing GC in a similar way. Second, because I believe that it would be possible to design hardware with object-memory addressing that would performs atomic reference counting with no additional runtime cost.
Such hardware has been designed in the past, Lisp machines, Ada machines, the famous iAPX 432 Intel's failure.
I'm being downvoted, but honestly, is anyone always testing their software while swap is active?