There are ways of locking down an incoming ssh connection to scp-only, you can even filter the files it has access to. I looked into this once, but never implemented it b/c we went elsewhere with the project.

Note: This is by making the handling script the thing that runs for a certain ssh key, I don't think it works with a password though.