Thanks to Captain Disillusion I know how this is done (obvious in hind sight, but cool to learn):

The movement of the marbles is pre-rendered and you can color the marbles in the end result right so at the beginning the distribution looks random, but everything falls perfectly into place.