I took ML in person at UIUC 4th year / graduate cross-listed level, and the 3rd year Stanford algos course on Coursera (supposed to be identical to the material in the actual course, yes yes though feel free to laugh b/c it's Coursera). These classes were interesting, but about as challenging as typical second year math/physics classes. They are peanuts compared to senior math/physics classes.
That said I still think CS is more challenging than most majors, it's just that the "few" is much more highlighted because there is very high demand. Certainly way higher demand than math/physics, and more demand than other fields that are still reasonably challenging like engineering, pre-med, etc. It's definitely the field where some reasonable increasing function on both demand and difficulty is maximum.