Here's some free marketing, I am not affiliated with any of these co.s: The best tool I've used for multiple users was CoScreen, because it shows everyone's cursors and allows simultaneous sharing of windows from multiple coworkers. So per that point in the OP that the major benefit is "trivia" knowledge, if you have one coworker who can quickly solve subproblem X from their terminal, and another who can quickly solve subproblem Y from their terminal, both terminals can be shared simultaneously and everyone can gain that context.
That makes sense, the "constantly moving around having side-conversations and whiteboarding things" being a good part of it.
After reading your description, I could see myself liking in-person mobbing too, and it really being a different thing than "pairing" (which I don't love)... but trying to do it remote sounds nightmarish to me personally.
I definitely think remote group programming is possible, but I reckon it'd be really hard to get right _starting_ remote, and it'd never be quite as good as in person is.