Only a select few have any leverage at all to bargain for more money (think Husain Bolt and others at his level). But these select few are already profiting big-time from the Olympics via endorsements, etc. So the last thing they want is to put the games at risk.
The rest of the athletes are there because they love the competition. And there's nothing wrong with that! It's truly a win-win that hundreds of athletes are living out their dreams by competing for a gold medal in canoeing or archery or what-have-you, and that somebody will even be watching them do it.
Stepping back a bit, there's no reason that anybody should be getting a salary simply because they are one of the very best at their chosen sport. Sports stars command large salaries to the extent that they have dedicated fans. If you compete in a fringe sport, you're not going to have many dedicated fans.
(This is why every top-tier athlete makes more from product endorsements than from their team salary.)
((This is also why NBA players earn by far the highest salaries among U.S. athletes - basketball allows for much more player/fan connection than any other major sport.))