This is pretty much what happens.
There's an IOC subsidiary called Olympic Broadcasting Services (OBS), which has crews/cameras/directors at every session of every event. If you're watching an event happen (as opposed to an interview), there's a 99% chance it's coming from an OBS camera. If the shot is coming from a site-specific camera - like the track-side camera at speedskating or the overhead curling shots - it is coming from an OBS camera. OBS cameras are the only ones allowed to go into the field of play; all other cameras have to stay on specified media risers outside the competition area.
When you watch an event on CTV, you're actually watching the OBS feed (with OBS graphics) overlaid with a CTV-supplied commentary track - the only sound OBS supplies is the ambient environmental ones, and it doesn't cut to any interviews or anything. Those of you who watched the Beijing Olympics on the CBC website may recall that there were eight or nine feeds available, but only one of which had CBC commentary. The commentary-free ones (which also didn't have commercial breaks or explanations) were the raw OBS feeds, without any additional CBC editing/directing.
When a broadcaster pays for exclusive national rights to the Olympics, one of the main things they're paying for is exclusive access to the OBS feeds. As for why they don't sell that feed to multiple networks? I assume they've worked the math and determined that they'll make more money by selling it exclusively, as opposed to cooperatively.