The popularity of pentatonic music also suggests divisibility isn't unmissable.
Why do the divisors of 12 matter in music? I'm not aware of anything using the divisors of 12.
12 is mainly important in music because the (3/2)^12 is a power of 2. We divide up the octave into fractions of frequencies, not multiple os steps (a.k.a. fractions of number-of-steps-per-octave). The harmonious way of stepping through the 12 notes is in steps of size [2,2,1,2,2,2,1], which is importantly not by divisors of 12. (Stepping strictly in any arithmetic progression hits at least one low-harmony tone and misses high-harmony tones.).
Stepping in the sequence [2,2,1,2,2,2,1] leads to frequency ratios in these low-complexity (==harmonious) fractions: [1 9/8 5/4 4/3 3/2 5/3 15/8 2]