SFTP/SCP works if you use the ProxyCommand, or agent forwarding. Clients such as WinSCP support agent forwarding, at least. Not sure about FileZilla. I'm not a Windows user, so can't say. :/
Maybe I should just install and try this for myself...
If I get around to it, one could wrap sftp completely, so that the available servers simply show up as root-level directories.
EDIT: Opened an issue about it.
That wound be amazing! I'll see if I can check if ProxyCommand is doable with FileZilla later as well.