Hacker News Reader

Disclaimer: I work at Backblaze.

> Does Backblaze really want this sort of traffic profile?

Oh heck yes, we would very much like your business! :-)

Backblaze only charges you $0.005/GByte/Month so your $436.77 bill would go down to $94.95 and we would be happy to have it. That is profitable for us. (Since Backblaze doesn't have any deep pockets or VC funding, we have to stay profitable.)

So your pricing is aligned with the storage cost. But youre explicitely not pricing in write throughput/IO access, deletes, or similar caused by lifecycle events? How does that square with long term trends to greatly increase density while IO remains flat for the past decade?

> IO remains flat for the past decade

Although I agree that storage density growth for HDDs has greatly outstripped any I/O growth, I don't agree that the latter has been flat (i.e. no growth).

Are you sure you're not doing something like comparing 7200rpm drives from 10 years ago to 5900rpm (or slower) or variable-speed "green" or even SMR drives from today?

That said, I think what many people forget is that for "cloud" storage, the I/O bottleneck is almost certainly going to be the network and not the backend storage, especially for mainly sequential access.

If each of their "pods" holds 60 drives and each "vault" holds 20 pods (17 of which are non-parity data), that's over a thousand drives per vault. If each drive is 7200rpm non-SMR, it can saturate a 1Gb ethernet with sequential I/O, and random I/O would divide that by 10 or so. That's the equivalent of 100Gb/s, per vault.

That kind of bandwidth is possible and even affordable to provision inside the datacenter without metering and charging for it and is likely to dwarf the size of the connection to the Internet.

Oh, Backblaze B2 DOES charge for transactions, but it is not intended to make Backblaze a profit, only to recoup the cost of the transactions. You can see a complete list of what each and every API call costs here:

https://www.backblaze.com/b2/b2-transactions-price.html

Their bill was $1885.43, not $436.77.

This is because they were pushing an average of 112 requests per second, the entire month.

> Their bill was $1885.43, not $436.77

I understand, I was comparing the storage cost.

Backblaze publishes a cost calculator here: https://www.backblaze.com/b2/cloud-storage-pricing.html (scroll down, it is half way down that page). But it might not contain transactions.

Backblaze does charge for transactions to recoup costs from that sort of thing. But it isn't intended as a "profit mechanism", we believe we approximately break even on the bill for transactions.

Oh I see, transactions do cost. $0.004 per 10,000 for Class B and C, which seems to boil down to delete and upload calls being free but the rest are paid.

In this case their bill would have had $115.89 for the 290 million requests.

Glad to see it's not a free lunch, I'm sorry for assuming otherwise.

And thanks for b2, it's a great service :)

I run a service which writes around 50-80 million small documents into S3 per month and, like you, we spend more on PUTs than storage and retrieval. So reducing PUT costs is something we've looked into a few times.

We do make heavy use of the S3 versioning API to let us roll back files, lifecycle rules for past revisions and have never been able to find any service which offers the same amount of functionality with reasonable pricing.

One thing we have done which has saved us quite a lot is adding a buffer between our service and S3 which delays file writing by up to an hour in case an updated version of a file is uploaded. This lets us limit our PUT requests but still gives us hourly revisions of files if we need to roll back. Without this system we'd be looking at over a billion writes a month. Certainly not a strategy suitable for everyone though.

Disclaimer: I work at Backblaze.

> We do make heavy use of the S3 versioning API to let us roll back files, lifecycle rules for past revisions and have never been able to find any service which offers the same amount of functionality with reasonable pricing.

Backblaze B2 has both lifecycle rules ( https://www.backblaze.com/b2/docs/lifecycle_rules.html ) and file versioning ( https://www.backblaze.com/b2/docs/file_versions.html ). I would LOVE for you to evaluate it and tell us what we are missing to win your business!

The Backblaze B2 system is still new and we are actively expanding the APIs all the time, so we're always collecting customer requirements looking for common issues we can tackle next.

Depends on the size of the file. If you're below 4 KB, the main competing system is databases, probably cassandra or DynamoDB for scaling/availability. You should also look into redis/memcache to provide the caching/buffering.

Beyond that, S3 is pretty good and it surely makes sense to model the content as files rather than blob.

>>> Probably there are some things we could do to reduce the number of PUT requests.

You can zip very small files into bigger archives. It will save a lot of requests.

Generally speaking, it's very inefficient to deal with numerous small files because of latency and overhead. It's not a specificity of S3.

> Generally speaking, it's very inefficient to deal with numerous small files because of latency

I want to second this. Backblaze's Personal Backup uses the same storage that B2 uses, and what we see is that tiny 10 byte files transmitted from "far away" (New Zealand to the United States) can be as slow as 15 kbits/sec transmission rate (yes, dial up modem speeds) because throughput is slaughtered by the latency to push each file.

Due to internet philosophies such as "TCP Slow Start" we do not see full throughput rates until after 40 MBytes in a single HTTPS POST if a customer is as far away as New Zealand.

One way we circumvent TCP Slow Start is to launch multiple threads. If TCP Slow Start results in only 1 Mbit/sec of throughput, we launch 20 threads to achieve 20 Mbits/sec of throughput. But you have to ask yourself what is going so horribly wrong on the internet if EVERYBODY has to write multiple threads on the same identical path through the internet in order to take advantage of the bandwidth available. Why not just allow 1 thread to get the full 20 Mbits/sec of throughput?