So, at the start, if someone is ladder-climbing-panic-drowning, then they are floating, have air, and have energy ... why are they drowning? What triggered the ladder-climb-panic mode?
You aren't unconscious yet because you still have plenty of oxygen, but your instincts have been triggered on a more primitive level -- possibly initiated as the lungs failed to function as expected.
It takes continuous, prolonged effort to stay afloat, and a fair bit more to move forward. The less you move forward, the slower your progress and the greater the energy you need to reach shore. You start drowning when you can no longer stay afloat - that is, when you can not quite move enough water downwards to support your weight. When you have no air, you have no patience. Water is quite difficult to lean on, and you will still easily climb out of the water if presented with a grip (but will may not be able to crawl far on the ground).
Swimming is excellent exercise. It will tire you out thoroughly and quite unexpectedly, but you will want to keep close to firmament.