That's most certainly the display sequentially flashing red, green and blue sequentially in time to approximate shades and hues. When you stare right at it, your eye averages the photons with a time constant of 1/30th of a second or so, and so you don't perceive the flicker. When you move your eye or blink, though, individual portions of your eye are exposed to only relatively narrow time slices, and so re only exposed to the red, green or blue photons.
Cheaper DLP projectors use a single light source and mechanically spin a color wheel with alternating red, green and blue filters. They look great when staring at it, but if you move your head or wave your hand in front of it, you can easily see the three color channels.
Perhaps those stadium displays are DLP projectors based, or maybe they're RGB LED and are simply PWMed at a relatively slow rate. Most LEDs can be switched very fast, at say 10Khz, but maybe there's electrical limitations of building such a large high brightness display. If it's only PWMing at 100-200Hz, you'd see similar effects. In particular, each color channel will be on for different duty cycle durations, and LEDs are very fast to turn on and off. So, when you move your eyes or blink, you'll once again get separation of the channels in your vision.
You can do a similar trick with your smartphone camera. Record video, and point it at the display then wiggle the phone up and down and side to side. The phone most certainly has a "rolling shutter" which means it captures an image sequentially in lines either horizontally or vertically. It does it quickly, but slow enough that different lines should be able to pick up colors. You may not even need to shake the camera up and down to see a funky image. It's the same reason why CRT monitors and TVs look funky on video but not film.