atYevP

Yev from Backblaze here -> We work with the drive manufacturers and their distributors! One of the reasons we think it's important to state the drive model is so people know exactly what is getting deployed versus just using ___ 14TB or ___16TB. But we source our drives just like any other large cloud storage provider would!

Yev from Backblaze here -> rest assured that we do read what you're writing on these posts and they've spurred some internal process discussions. I believe the bugs you mentioned were cleared/fixed with version 7.0.0.439 which was released in Q1 of 2020. We did leave HackerOne and switched over to BugCrowd to handle our bug program. It's private at the moment, but easy enough to get invited (by emailing bounty@backblaze.com). While we spin that program up (it's a new vendor for us) we may stay private, but hopefully that's not a permanent state.

Edit -> I just noticed the Daniel Stenberg libcurl citation. Oof, yea that certainly a whiff on our end. Luckily though we were able to make up for it (he has a write-up here: https://daniel.haxx.se/blog/2020/01/14/backblazed/).

Yev from Backblaze here -> sorry about that Thom - it's one of the things we're definitely working on hammering down. Right now we're growing quite a bit on our engineer and infrastructure teams and one of the projects we'd like to see is more automated status updates. We typically will throw updates onto Twitter or our blog if it's a large outage - or affecting many different people, but totally recognize that process can use some sprucing.

We do have some drives that we use as boot drives and for logging. We write about them a bit in the post - they're primarily SSDs, so not included in the overall mix of data drives!

Yev from Backblaze here -> We just started deploying more of them! We added almost 6k WDC drives in Q4 - so we're getting more of them in the fleet! They have a pretty low AFR - but haven't been deployed for too long, so they'll be interesting to follow!

No, that comment was edited after I had responded - we're not against status pages - they are on the roadmap, but unfortunately not implemented yet.

Well, hopefully there won't be a next time - but also, bothering me doesn't help debug or resolve the problem, our Ops/Tech team do that :D

Yev here - we're investing the S3-endpoint issue - apologies for delay in response here, it took them a bit of time to wake me up.

*Edit -> we've identified the fix and are rolling it out, you should see S3-Endpoint service resuming shortly!

Yev here -> Hey there! We've just signed with a new program provider and hope to be rolling out the private program in early November with a public rollout to follow. For now though we ask folks to reach out to bounty@backblaze.com directly with any bugs they find so we can track them internally!

Yev here -> I should add that we have a Q&A webinar going live tomorrow with Andy the author, you can join the Q&A here: https://www.brighttalk.com/webcast/14807/446738?utm_source=T....

Yev here -> I LOVED the OkCupid blog posts. So much good stuff in there. <3

I have a great relationship with the Code42 folk - they're good people and do good work! Their Small Business service is a bit more expensive, but yes at that increased rate they've been able to provide service to a few more platforms. I have no idea, but I do hope it works for them! I send people there all the time as an alternative if they don't want to use B2 Cloud Storage w/ an integrator (which is what I normally tell folks to use if they have Linux/Server/NAS but not a ton of data).

Feelin' good!

Yev here -> Good question! The ops team may have a different theory, but I think it's when they die! We still have 15k 4TB HGST drives in our fleet that are crushing the game, and have been for years. That said, we'll eventually migrate off of them to get more density, but so long as we're not cramped for space, we tend to keep high-performing drives in our fleet for as long as we can. There is a math equation that flips after a certain point where we can get a lot more density by replacing those smaller-capacity drives - so that tends to be the other main driver for migrating the fleet.

*Edit -> For giggles here's an OLD post where we dissect drive failure and talk about migration a bit -> https://www.backblaze.com/blog/3tb-hard-drive-failure/

Yev -> Hey there! At the moment I don't believe we have any of those in our fleet (some MD and MG but no MN) so I don't have any data on them or how they'd function in our environment at the moment!

Yev -> that's really interesting. Do you want to publish your data and maybe we can compare notes ;-)

Yev -> Great question. The problem does lay with the abuse of the unlimited services. We've been offering our unlimited backup for 10+ years now and our only price increase was from $5 -> $6/month after 10 years of keeping them steady. It's possible to provide the service, but we have some natural "gates" that were built into the product to help mitigate abuse. Primarily our unlimited backup service is for Mac and PC computers – we don't back up NAS/Servers/Linus machines.

That helps quite a bit because it reduces the outer-bound of how much data can store with us from typical Mac/PC devices. Granted we do have people with 80+TB backed up with that service, but that VAST majority store A LOT less. The average is low enough that we can continue to provide the unlimited service - kind of like a buffet model. Some will eat more, most will eat less and buy a drink (Extended Version History).

Keeping it to Mac/PC only does make Linux, NAS, Server folks unhappy, but thus far no one has been able to crack the code on an unlimited, fixed-rate service that can back up those devices economically, but if we figure it out we'll let folks know :D

Yev from Backblaze here -> Most of the comments have it. They tend to be more affordable and the failure rates aren't that much different from their counterparts. Since we account for failure with our software layer a small failure rate delta isn't a big deal and is eclipsed our cost savings which we pass on to our customers!

Yev from Backblaze here -> no, any drives that come DOA wouldn't have made it past burn-in and into our fleet, so they aren't represented.

Yev from Backblaze here -> price plays a large factor in our purchasing decisions, but you'll notice if you look at the failure rates that they're not THAT much different across the manufacturers. For home usage, you're likely going to be fine regardless of what drives you buy. Though those sweet sweet 4TB HGST drives are getting harder and harder to find!

Yev from Backblaze here -> we do have SSDs in the fleet but they are primarily boot and data drives, so they're not really an apples:apples use-case. We're constantly looking for interesting news from our fleet though, so if there's something that pops out, our default approach would be to talk about it :D

Yev from Backblaze here -> what a fascinating question! You need storage to "generate" the data, and you need storage to "store" the data - so this is a fun thing to ponder!

Thanks for the kudos! We've been seeing a lot of talk about this today - I'm glad we can be there for folks now that they need us :D

Yev from Backblaze here -> We're definitely a good repository for all of that data. Happy to have you :)

Yev here -> It's less of an algorithm and more of a purchasing department with lots of spreadsheets that tie into the AFR data as a data point - but yes, it's possible we could automate it, but there's some factors that our supply-chain team would need to be involved in anyway (like vendor diversity, no large batches of single drives etc...) that make outputting that algorithm a bit harder.

Yev here -> Yes we want to mitigate large batch failures by adding some drive/vendor diversification, that's definitely part of the equation!

Yev here -> Thanks! We love producing them - they are a lot of fun!

Yev from Backblaze here -> Some of the responses down below are correct - it really does come down to price. We run a fairly lean ship and so the price per gigabyte is weighed pretty heavily when we make our purchasing decisions. We have a post where we go into the cost of hard drives over time, it's about 3 years old now, but still a good read -> https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/.

Yev from Backblaze here -> Yes, we tend to weigh heavily in favor of price and Seagate thus far as been affordable with performance that meets our requirements, so that's why you'll see those drives in higher numbers. We're also trying to diversify our fleet a bit, so it's not purely one vendor, but we do try to keep an eye on cost since we run a pretty lean ship!

Yev from Backblaze here -> thanks! We're pretty excited about this, there's never really a better time for AWS users to diversify their cloud storage vendors and keep a working copy somewhere else. :D