Also, once you've got per-user encryption on ~/.ssh/authorized_keys you'll need some other mechanism for users to log in.
And don't think of having the users do their own backups - locking users' home directories mean they won't be able to run their own cron jobs.