Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?
(Assuming that the article incorrectly used Mib when they meant MiB. Used correctly b=bit, B=byte)
The only explanation I can see (if their conclusion is accurate) is that the end result of the symbolization is more than 400MB additional memory consumption (which is a lot in my opinion), however the process of the symbolization requires more than 2GB additional memory (which is incredibly a lot).
The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting.
Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling.
Regarding the increase of 400MiB, yeah it is a lot, and it was a surprise to us too. We were not expecting such increase. There are, I think 2 reasons behind this.
1. This service is a grpc server, which has a lot of code generated, so lots of symbols
2. we compile the binary with debug symbols and a flag to compress the debug symbols sections to avoid having huge binary. Which may part be of this issue.
How big are the uncompressed debug symbols? I'd expected processing uncompressed debug symbols to happen via a memory mapped file, while compressed debug symbols probably need to be extracted to anonymous memory.
cargo build --bin engine-gateway --release
Finished `release` profile [optimized + debuginfo] target(s) in 1m 00s
ls -lh target/release/engine-gateway
.rwxr-xr-x erebe erebe 198 MB Sun Jan 19 12:37:35 2025 target/release/engine-gateway
export RUSTFLAGS="-C link-arg=-Wl,--compress-debug-sections=zlib -C force-frame-pointers=yes"
cargo build --bin engine-gateway --release
Finished `release` profile [optimized + debuginfo] target(s) in 1m 04s
ls -lh target/release/engine-gateway
.rwxr-xr-x erebe erebe 61 MB Sun Jan 19 12:39:13 2025 target/release/engine-gateway
For this particular service, the size does not matter really. For others, it makes more diff (several hundred of Mb) and as we deploy on customers infra, we want images' size to stay reasonable. For now, we apply the same build rules for all our services to stay consistent.
Smaller executables are more for things like reducing distribution sizes, or reducing process launch latency when disk throughput is the issue. When you invoke compression, you are explicitly trading off runtime performance in order to get the benefit of smaller on-disk or network transmission size. For a hosted service, that's usually not a good tradeoff.
For your comment, I think you are right regarding compression of debug symbols that add up to the peak memory, but I think you are misleading when you think the debug symbols are uncompressed when the app/binary is started/loaded. Decompression only happens for me when this section is accessed by debugger or equivalent. It is not the same thing as when the binary is fully compressed, like with upx for example.
I have done a quick sanity check on my desktop, I got.
[profile.release]
lto = "thin"
debug = true
strip = false
export RUSTFLAGS="-C link-arg=-Wl,--compress-debug-sections=zlib -C force-frame-pointers=yes"
cargo build --bin engine-gateway --release
When compiled with those flags
export RUSTFLAGS="-C force-frame-pointers=yes"
cargo build --bin engine-gateway --release
So the peak is taller indeed when the debug section is compressed, but the binary in memory when started is roughly equivalent. (virtual mem too)
I had some hard time getting a source that may validate my belief regarding when the debug symbol are uncompressed. But based on https://inbox.sourceware.org/binutils/20080622061003.D279F3F... and the help of claude.ai, I would say it is only when those sections are accessed.
for what is worth, the whole answer of claude.ai
The debug sections compressed with --compress-debug-sections=zlib are decompressed:
At runtime by the debugger (like GDB) when it needs to access the debug information:
When setting breakpoints
When doing backtraces
When inspecting variables
During symbol resolution
When tools need to read debug info:
During coredump analysis
When using tools like addr2line
During source-level debugging
When using readelf with the -w option
The compression is transparent to these tools - they automatically handle the decompression when needed. The sections remain compressed on disk, and are only decompressed in memory when required.
This helps reduce the binary size on disk while still maintaining full debugging capabilities, with only a small runtime performance cost when the debug info needs to be accessed.
The decompression is handled by the libelf/DWARF libraries that these tools use to parse the ELF files.symbols are usually included even with debuglevel 0, unless stripped[0]. And debuginfo is configurable at several levels[1]. If you've set it to 2/full try dropping to a lower level, that might also result in less data to load for the backtrace implementation.
[0] https://users.rust-lang.org/t/difference-between-strip-symbo... [1] https://doc.rust-lang.org/cargo/reference/profiles.html#debu...
I don't think the article is misleading, but I do think it's a shame that all the interesting info is saved for this hackernews comment. I think it would make for a more exciting article if you included more of the analysis along with the facts. Remember, as readers we don't know anything about your constraints/system.