This is definitely very significant, you just took away 6% of the CPU time available to the game to calculate everything necessary for that frame. And that's if there's only one pause inside each frame. And if you are running close to 100% CPU time, and the pause happens right before the vsync, you might miss the vsync and you will drop a whole frame.
This is not only a problem with GC, even with more deterministic memory management, like plain malloc() and free() in C, you can have pauses if the operating system needs to go and find some free memory, update page tables, flush TLB caches and so on. So for games and anything real-time, you probably want to avoid memory allocations as much as possible, and certainly don't want your whole process being paused at random. So if you are stuck with GC, then one question would be: can GC be done for individual threads without blocking other threads?