Yes, the "them" refers to the users. If you have users of a binary server, you must allow those users to download the source code. You do not need to allow anyone who is not a user to do so, though.
Edit (programmer needs sleep badly): Clearly it doesn't have to be a binary. It's any executable over a network. But you don't have to give the source cod to everyone, that's really my only point :-)