Compiling new Android systems requires 16 gigabytes of RAM at compile time. Trying to compile it with 6 gigabytes will end up swapping so much that the system becomes unusable. I have 12 gigabytes and it's a bit slow and painful.
I have no idea why it takes so much memory and I consider it to be somewhat a problem. However, it's not a problem I can do anything about but I still have to live with it daily and get my job done.
Linux has excellent caching for I/O operations, so having more memory makes things faster, so even if something does not explicitly need the RAM, having it there will make thing s go faster.