Panning shots probably look awful if you're watching 24 fps content on a device running at 60Hz (many consumer devices, computers, and Blu Ray players do this by default). Since 24 frames can't be evenly spread out over 60 frames, you'll see motion judder, and it does look terrible.
However, 24 fps running at a true 24Hz is perfectly fine, and you will not see flicker - at least not if produced by competent filmmakers. That's because the camera's shutter angle is normally set such that motion blur makes the motion look smooth. Real cinematographers also know exactly how fast they can pan with a given lens without causing any noticeable strobing motion.
Comparisons to video games completely miss the point. Most video games are unable to properly simulate motion blur (some try, but it doesn't usually work well), so you have to have high frame rates for things to look smooth. Since you need to react quickly in video games, high frame rates (combined with the lack of motion blur) are also helpful in preserving crisp edges on moving objects (so it's a practical advantage). And finally, video games generally try to simulate reality for the player, so players are more concerned that the technology makes the experience believable rather than intentionally suspending disbelief (as in film or theater) to passively unpack the narrative nuance of artwork unfolding on a screen.
For those same reasons, sports also do well with higher frame rates, but when fiction (which is obviously disconnected with the present reality) tries to use high frame rates, it falls into "uncanny valley" territory - much like the computer-generated humans of movies like "The Polar Express". As others have noted, a few directors have really tried to push HFR film to the public, but it has never been received well (whether or not HFR is recognized as being responsible for the uneasiness felt by audiences watching HFR content).
As for HFR content being "easily" downsampled - it really isn't, at least not with respect to motion blur (which is an essential point that many people miss in these discussions).