1: Erlang GCs a process' heap whenever that process' heap gets full, or when you call erlang:garbage_collect() explicitly.
2: Erlang stores most data associated with a process on the process heap, there's one such heap per erlang process. Binaries are a special case. If they're large (> 64 octets), the heap only contains a reference to the binary, the binary itself is stored in an area specifically for binaries.
1+2 can result in a lot of binary garbage left lying around. Here's how it can happen:
a) A process creates a relatively large amount of unused heap space. This could happen by temporarily using a large amount of heap, e.g. by calling binary_to_list() on a large-ish binary, doing something with the list then dropping it. For arguments' sake, let's say we made a heap (for one Erlang process) with 30M free.
b) Now the process moves on to a new phase: creating large but short-lived binaries. Let's say 1M each. Those binaries don't live on the heap, only a reference to them does. So they only consume 8 (?) octets on the heap.
c) If the references are the only thing using the heap, then you can make 4M of them before filling the process heap. But since they're 1M each, they'll eat 4T of the binary heap. i.e. you'll run out of memory.
As you found out, setting 'fullsweep_after' to 0 doesn't help, since a GC is never triggered. But explicitly calling erlang:garbage_collect() does.
You can investigate a bit more using tracing. erlang:trace/3 can generate a message whenever the target process is GCed (the garbage_collection entry in flaglist). If my guess is right, then you should see that your processes holding all the binaries are never (or rarely) GCed. Tracing the GC is cheap, you won't notice any performance difference if you only do it on a few processes.
The process_info BIF can also tell you quite a bit, e.g. the process heap size.
Disclaimer #1: my knowledge about the details above may be wrong or out of date. But the mechanism is known. I've seen it in embedded systems and handled it in similar ways to your approach.
Disclaimer #2: obviously I'm taking a guess. It's possible you've run into something else entirely. That's why I suggested some things to look at to confirm or reject my suspicion.