I've wondered about this before, and had to explained to me. Basically, RMS only considers software "eligible" to be free if it could, in theory, be moved to other hardware of the customer's choice. It is the ability to put the software somewhere else that is the "freedom" being abridged by closing the source. On the other hand, software that is fundamentally tied to hardware (such as firmware or infrastructure control) couldn't reasonably be moved somewhere else, so close-sourcing that software is not impinging on any freedom that would otherwise exist.
It's a fine distinction to make, but I get it. I don't necessarily agree with it, and I think it's a strange place to draw the line, but I can understand the logic behind it.
If I'm understanding this right, then the only part of Skype that RMS objects to is the desktop client. The network and infrastructure being closed-source does not remove freedom. (NSA is a separate consideration)