Hacker News Reader

Comment by dagss | Hacker News Reader

An alternative to doing file based SFTP is to just treat SFTP like an API.

A company I work for implemented an SFTP service where every operation simply translates to some SQL DB lookup. And a file download kicks off a larger SQL query and generates the report on the fly, streaming the result straight through to the SFTP client.

Works great! SFTP can be an API just like HTTP. Under the hood the protocol is reasonably contained and doesn't require a filesystem backend at all.

Depends a lot on the usecase of course.

See https://github.com/pkg/sftp

View on HN

I did something similar using the apache ftp client, as long as the resources are identified by a path it's very convenient and extremely easy, just implement a FileSystemFactory with apache mina and off you go.

What if you open an SFTP handle, and then write 5 bytes halfway through a 20 GB file and close the handle? How do you translate that?

hint: he doesn't.

You could. I mean, at least with OpenSSH you can specify a byte range. That is how lftp is able to chop up files into many streams on SFTP. I can't imagine anyone doing this with a database however, at least, not for writes.

Similar to how you might handle writing 5 bytes 20gb offset into /dev/null?

Seems similar to if someone sends an HTTP method to an URL that doesn't support it. Just return an error.

From parent: "Depends a lot on the usecase of course."

The usecase that I see most often of SFTP (and hinted at in the parent's problem description) is generating one-off reports for third parties, or passing data to vendors who are stuck in the 90s, like financial services companies.

It's almost always read only (or read and delete), in which case implementing an API like this is pretty straightforward. Log unsupported commands perhaps and decide if you want to implement them later.

You need to make it transactional. Upload to a temp file name (something easily ignored by whatever backend processes are looking at the files) and then do an atomic rename once the transfer is complete.

I think this implementation uploads it to memory before going to S3. It usually won't handle 20GB files (unless you have like 20GB of RAM) and in this case, were it a smaller file, it'll just never upload.

https://github.com/moriyoshi/s3-sftp-proxy

Yes the usecase was for a readonly connection.

Reply on news.ycombinator.com