You don't have to if you use EncFS. All encryption is done client-side.

The only thing your VPS provider could do is delete your files, but Unison's backup feature should protect you from losing your files (in a way).