Make a separate ssh key for your phone with restricted sftp-only access to a phone-specific backup directory on your server.
It's worth pointing out that this will preclude use of rsync and similar tools that require running a server on the remote machine. I'm not aware of robust alternatives so if anyone else knows one please tell me! (and I consider rsync-restricted shells and similar approaches a hack)