Then what happens if the GC keeps taking longer than the allowed time? Will it then not keep allocating more and more memory that will never be freed?
I haven't looked at Nimrod's GC, but I believe it's indeed incremental; it won't take more time than you give it, but it will leave things uncollected if you don't give it [enough] time.
Also the GC is only triggered on a memory allocation. It doesn't run in a background thread or anything like that. So if the GC fails, then the allocation of memory fails (as I understand it) which means your scenario would be caught early.
Does this mean that they use different heaps?