Most userspace memory allocators are based on mmap(). Using sbrk() is legacy. The slab allocator (Bonwick, 1994) is the dominant algorithm (both in userspace and the UNIX-like kernels), although there are various flavours and somewhat different implementations of it. Since it uses fixed-sized allocations under the hood, they are nicely packed in pages. Those pages can be released with munmap() once there are no used blocks. Sure, depending on the application and workload, allocations can be become quite distributed amongst the pages over the time, but that is a separate problem.
Also, as another comment states: madvise() is also an option, but it doesn't reclaim the virtual address space. On 32-bit systems VA space exhaustion can be a problem, especially with "modern" applications.
Also you can verify in the code, e.g. https://git.musl-libc.org/cgit/musl/tree/src/malloc/expand_h...
I believe OpenBSD's malloc only uses mmap and also aggressively munmap's memory--to better catch application bugs and to keep addresses randomized.
Facts and to the point.
madvise( start, size, MADV_DONTNEED );
this informs the kernel you're done with some part of the memory. mapped things will be re-read and others will act as if zero'd when later accessed. both can be safely dropped without taking space in the swap, instead being newly created when next needed.
[1]: https://pubs.opengroup.org/onlinepubs/9699919799/functions/p...
I guess I didn't see how long that operation takes though, but it could certainly be useful for long-running processes like web browsers to call it every once in a while...
What you can do is search through all allocated memory for full 4K pages which are unused, which can be "given back" to the OS.
However, it's unclear how much benefit there is from doing this. If you give back memory too eagerly, and then need to request more later, you end up thrashing the kernel. Assuming you have some swap space, this unused memory gets swapped out.
It could be (hard to tell, but I do this in my own memory managers) that the memory is technically still assigned to the program, but the pages are marked as unused, in which point there is no "memory" there at all, just address space which you can start using again later. However, it can still look like my program is using something crazy like 1TB of memory.
Glibc malloc can return memory to the OS. This is what the trim that people are talking about does.
But I think malloc can move memory allocations - it can move objects out of a page and return it to the kernel by copying objects into an existing page with a gap and then remapping the old addresses onto the copied page. Then the memory has moved but retains the same address and the old page is returned. See for example the Mesh allocator which does this and still manages to expose a conventional malloc interface.