Yes, messages keep piling up and memory usage keeps increasing until the Erlang VM goes out of memory.
To debug, there's a max_heap_size parameter which can be configured to log an error message when heap size of any process goes beyond a threshold, or it can even kill the process[0]. Although I don't know if this takes the "off-heap" messages into account.
Also, there are lots of libraries which can provide a "top" like view of the running system.
[0] - http://erlang.2086793.n4.nabble.com/Max-heap-size-td4716974....