I've heard that special forces guys tend to be smaller, but I'm not sure that's true. It seems like there could be a few tactical advantages to being shorter, less likely to bump your head maybe, but who knows.
Historically, military selection emphasized calisthenics and load carrying capacity (body armor/weapons/rucks) + endurance. Although modern assessments have somewhat shifted towards weightlifting components, calisthenics/rucking are still a major focus. There is generally a disadvantage with height, where longer limb length creates more leverage to overcome, which is a disadvantage in both calisthenics and moving external weight around.
Average size is ~5’10” @ 180lbs at selection
(1) https://sofprepcoach.com/special-forces-selection-body-compo...