I don't think there's a setting for "only allow people from X server", but you can make the room private and only invite people from that particular server.