Is this some kind of mathematical thing based on how orbits work, or is it more like natural selection, in that any moons that don't avoid each other just aren't there anymore.
It would be if we saw many colliding orbits and only those that had resonance surviving in long run.
But resonances are perfect ratios and no random orbits will ever have perfect ratios of orbit times.
Instead what happens is that collisions seem to be extremely rare occurrence. Once two moons find themselves on close orbits they will tend to exchange energy until they find some kind of resonance that will keep true for a very long time until disturbed.
Resonant orbits might not be natural selection, but they might participate in a form of thermodynamic selection: two simple forms and structures selecting for and reinforcing one another.
They are both subsets of the same thermodynamic processes of structures/arrangements sticking arouns when they mutually support one another creating persistence in a chaotic system
My understanding is that some folks believe there are similar mechanisms underlying the emergence of both -- a generalized form of selection between mutually reinforcing structures that encode elements of prediction of the chaotic environment they reside in, be that the periodicity of orbits (that "know* a stable path that will reasonably persist) or life itself (that have similar baked in prediction abilities we perceive subjectively as consciousness).
Disclaimer: biochemist, technologist and armchair follower of complexity science
We are just trained as children to see the small things independently orbiting the big thing, but in reality all bodies orbit each other.
What would this animation look like with one of the moons at the center with Jupiter in a fixed position?
These simplifications are often conceptually not that much simpler, they just have much simpler math. But saying "x is what's happening, but in these circumstances we can treat it as y" is an important skill in science and life, we should teach it more (and benefit from fewer misunderstandings caused by teaching only the simplification)
Nothing orbits anything: everything moves in a straight line; space is curved