The solution to this is higher refresh rate and/or VRR.
VRR would allow you to sync to any needed timing (regardless of VRR range, as multiples of fps are fine), dynamically, with no need to modeset to fixed supported refresh rates.
120Hz not only happens to be a multiple of both 24 and 30, but judder from such low framerate input becomes irrelevant at these rates, with VRR instead serving high-framerate content that is slightly out of phase.