Hacker News Reader

Comment by avolcano | Hacker News Reader

Huh, I thought it already had this! Must have mixed it up with a different object storage service (maybe DigitalOcean?).

I've been using B2 for backup storage for some personal projects. It doesn't necessarily do anything "better" than S3 from what I've seen, but never having to log into AWS's dashboard is a reward enough on its own.

They do have a command-line client that's a quick PIP install, so you can do something like:

  b2 upload-file bucket-name /path/to/file remote-filename

Which is, of course, nice for backups.

View on HN

I really wish the B2 client support uploading file from Unix pipe. It would be nice to be able to archive a huge directory into a tar.bz archive and directly pipe the result into the B2 client without having to save the archive into disk first.

Currently I have to save the tar.bz archive to disk first before uploading to balance. Took several hours to do so (huge spinning disks array, not as fast as ssd), while uploading to B2 is blazing fast. Saving the archive to ramdrive essentially solved this, but as the data grows I don't have enough memory to spare anymore for a ram drive that can fit the whole archive.

Can you use process substitution?

    b2 upload_file bucket <(tar -cj huge-directory) archive.tar.bz2

The argument the command sees will be something like "/dev/fd/42", and the shell will provide the output of tar through that file.

Thanks for the idea! I'm going to try this.

Does process substitution actually wrote the content to disk first or not? The information on internet I found seem to be conflicting on this. If it's actually writing the data into disk first, then it's probably won't solve my problem (limited disk i/o). Afaik writing to pipe won't result in saving the data to disk temporarily. I guess the only way to know is to try it out on my system and see how it performs.

If process substitution doesn't work, shouldn't /dev/stdin work? I haven't tried it, but as long as b2 doesn't try to check the file size before uploading I don't see why it wouldn't work: b2 upload_file bucket /dev/stdin < file

It definitely doesn't write it to disk first. It's basically a pipe() under the hood, but exposed as a file descriptor. Downside is that seeking doesn't work, but that shouldn't affect your case.

So apparently process substitution doesn't work. The b2 client is probably trying to read file size or something, keep failing with "ERROR: Invalid upload source: /dev/fd/xx". Maybe their api requires knowing the filesize upfront instead of allowing "streaming" upload.

>Huh, I thought it already had this!

Same since it seems to be on the new storage service launch checklist right after "buy hard drives".

Reply on news.ycombinator.com