Hacker News Reader

Comment by flohofwoe | Hacker News Reader

> ...and want to control exactly where when and how memory is allocated...

Isn't this the whole point and the reason why manual memory management is still a thing - and actually getting more important with the ever widening CPU/memory latency gap?

There simply is no silver bullet for memory management, you can either have high performance or automatic memory management, but not both.

(vastly simplified of course - but in languages with automatic memory management - no matter if garbage collection or refcounting - you need to spend so much effort to reduce memory management overhead that it is usually less work to do it all manually in the first place)

View on HN

There's the third option, which is what Rust and C++ do by default, where types have destructors that are run when the value goes out of scope, but also have mechanisms to be activated manually at a precise moment if you know what you're doing and have specific needs.

RAII is not a silver bullet. RAII wants to have neat recursive ownership and cleanup of everything, which is undoubtedly a nice model, but sometimes in low-level code you have to choose between nice recursive ownership and performance.

For example, Protocol Buffers are trees of objects. Originally they were simple and had recursively-owned, heap-allocated nodes. But it turns out that traversing and deleting a big graph of heap-allocated objects is expensive, and arenas are more efficient. But arenas make you give up recursive ownership.

Which is why low level languages like C++ and Rust make RAII the default in their standard library (easy, simple), but it's still completely possible to manage the memory on your own if that's necessary for the task at hand. Having easy defaults with the capacity to do everything by hand if you need to is the most flexible system.

The last time I looked (which has been a while, I admit), that meant that you had a very hard time using any of the standard library types with an arena or whatever form of management you used for your specialized code.

What does that look like now?

If you're willing to write Nightly Rust, you can use the allocator_api feature to have the types which allocate take an allocator parameter so e.g.

Vec::with_capacity_in(10, arena_allocator)

... obviously user-defined types don't know these allocators exist and so may not provide a way to pick which allocator is used, whereas in Zig this is "always" how it worked.

For stable Rust, until/ unless allocator_api is stabilized, you will need to replace the global allocator for your code with one that has the desired behaviour. That applies to everything using an allocator, but on the other hand you lose considerable flexibility.

There is no incompatibility between RAII and arenas. RAII doesn't require heap allocation.

That's not really the point. If you're using an arena it means that you plan to reset it at some point and erase in bulk everything that was stored on it all at once. This is the opposite of letting the destructor of every object run.

Not every object in an RAII language needs a destructor, e.g. objects that don't manage their own memory because they came from an arena

In Rust terms, the arena should own the data and each object should just borrow from the arena. In Rust, when you drop a borrow it doesn't deallocate the memory; only owned values deallocate on drop (that is, in the destructor). Besides, borrowing enforces that the arena won't drop while the borrows are still outstanding.

https://manishearth.github.io/blog/2021/03/15/arenas-in-rust...

This is by definition manual memory management.

I would argue RAII is still a type of automatic memory management, but it has the determinism and ability to reason about what's going on the same way fully manual management does.

From the Garbage Collector handbook I got a very great quote about what automatic memory management really is. I will butcher it up, but the basic essence is that automatic memory management is above all a software design tool — well designed programs are built from components that are highly cohesive but loosely coupled. What it means for memory management is that a component shouldn’t have to care about how other modules handle memory for correct working. This is possible with a GC, but isn’t with manual memory management, and RAII doesn’t change it — function signatures leak memory management concerns in both Rust and C++.

Ongoing refactoring and maintenance costs are also higher in low-level languages due to this reason.

> you need to spend so much effort to reduce memory management overhead that it is usually less work to do it all manually in the first place

That has not been the case in the vast majority of applications I've written across a number of domains using garbage collected languages.

Yes, there is runtime overhead to GC. But for many many programs, the cost of the overhead is acceptable and you can spend all of your time worrying about the application domain and not worrying about allocation.

Not allowing you to talk about allocation in any great detail certainly on the surface seems to simplify things, but as with most things of that nature it tends to complicate things in edge cases (or just entire parts of industry). What the GP is alluding to, it seems to me, is that in those cases the effort to remedy the problem tends to become a constant hunt and guessing game of "Where are we accidentally allocating and ruining everything?".

Inevitably some kind of object pooling comes into play in most of these cases and this is arguably a much worse solution to the basic problem. If you know you're going to be landing in this space I don't think there's much purpose to pretending you're better off using a GC that you'll be trying to avoid anyway. It's far easier to sketch out your allocation strategies and do things deliberately.

Fast software allocates and deallocates in bulk, so the straw man of individual `malloc` and `free` isn't exactly relevant. Arenas, bump allocators, etc. are what is actually being discussed as an alternative.

(As a side note I think this makes RAII a somewhat peculiar solution to the issue as well. RAII implies objects themselves controlling allocation and deallocation and this is just not very useful at all if you want to make something that behaves properly.)

> Where are we accidentally allocating and ruining everything

This is trivial to profile under any platform that has remotely sane tooling, and usually it is trivial to fix as well.

Object pooling is a very last resort thing to do, I really can’t think of any project I have worked on where there wasn’t another solution to performance problems.

"and you can spend all of your time worrying about the application domain and not worrying about allocation."

This has never been my experience in 10+ years of middleware development in GC languages in large teams. The problem is just deferred to production where you need to analyze heap dumps and run a profiler to hunt down excessive allocation issues and why the GC is running all the time.

Sometimes, troubleshooting and correcting many of these issues take multiple more man hours than coding the service in the first place! The usual solution of "throw more hardware at it" only works until mgmt starts complaining about costs and uptime.

I have spent more time debugging memory issues in Java/C# projects than in my first large scale project in C++ where the lead architect strictly mandated the use of a custom allocator for everything.

I'm not sure why you trimmed the beginning of my sentence.

The parent comment says: "(vastly simplified of course - but in languages with automatic memory management - no matter if garbage collection or refcounting - you need to spend so much effort to reduce memory management overhead that it is usually less work to do it all manually in the first place)"

That suggests they are claiming that all users of GC languages will eventually spend more effort dealing with memory than if they'd skipped a GC in the first place.

I am making an existence proof counterargument: I have worked on many many programs in GC languages where at no point in the program's lifecycle did I need to spend much time worrying about memory.

I believe the claim "all programmers will spend more time dealing with memory in GC languages" is false.

I have absolutely not made any claim that "no programmer will spend more time dealing with memory in GC languages". There are certainly programs that are better written with manual memory management. I've written plenty.

My point was only that there are also plenty of programs where that's not true.

> There simply is no silver bullet for memory management, you can either have high performance or automatic memory management

That’s only true for workloads where you can optimize the memory layout/allocations. This may not be true in general, where you will end up implementing a shitty GC that will definitely perform worse than a properly written one.

Also, the cost of a proper GC is heavily overblown in my opinion. Especially when value types are available.

GC is not a solved problem and can be costly. We clearly see in a production server written in Go spikes in latency and memory usage. So most likely for the next server-type application we will use Rust.

Then there are benchmarks where Nginx is faster than Caddy by factor of 3.

Go doesn’t really have a good GC for what it’s worth. And of course there will always be domains where the tradeoffs of a GC doesn’t worth it, but those are very very rare.

The problem with GC is that when it does give problems, there are no good choices. One either spends time trying to tune available knobs to hide the problem or rewrite application to try to reuse objects etc. turning the initial nice architecture into spaghetti mess.

I very much prefer for this reason reference counting. Yes, time can be wasted fighting leaks through cycles, one still have avalanches of releases problems, the performance can be bad with certain usage patterns. But solving these problems is at least local and does not required rewriting the whole application.

The real downside to reference counting is concurrency. If the Reference Counter lives in one Core A's L1/L2 cache, then Core B needs to write to the reference count, the cache line is invalidated, and Core A needs to re-fetch it again. You get two cores fighting over the cache line. Everything still works correctly, but it causes some delays if each core is constantly updating the reference counter.

One workaround is giving the threads their own private reference counter (making sure they're on different cache lines), and updating the shared reference counter only when their private reference count reaches 0. Not only do you stop the fighting, you don't even need atomic operations to update the private reference counts anymore.

Apple mostly solved that problem in their silicon. But thread-private counters is a good solution for other platforms.

What kind of workload does the server run?

It serves as a proxy or manager for other components. It is IO bound. As such Go should be an ideal language for it. But GC kicking in a wrong moment still brings troubles.

So Go GC just need a api that pauses GC when it wants to run

Reference Counting (including a secondary counter for internal cycles) is your silver bullet.

The only hairy issue then becomes multiple processors needing to update the reference counter, where the count has to go in an out of the various processor's cache. One possible way to resolve that is to have a per-thread reference count, and only change the globally shared reference count when a per-thread's reference count would reach 0.

> There simply is no silver bullet for memory management, you can either have high performance or automatic memory management, but not both.

RAII is the way.

RAII for memory management implies that destructors free owned memory, which can quickly cascade out of control for complex objects (e.g. you still need to think very carefully about memory management).

You are right in the sense that for high performance applications you still need to explicitly take into consideration memory allocations, allocators, indirection, into consideration. RAII helps you implement your preferred design, but it doesn't make you ignore the details.

Reply on news.ycombinator.com