There is no downside to using it. However, keep in mind that the key file is shared across all clients, so you probably want to change it periodically depending upon how much you trust your clients not to become hostile. You could run multiple server instances on different ports/IPs to subdivide your client base, or when rolling out a new key.