Wasabi is a good option. They’re S3 compatible and don’t charge any egress or ingress fees. Been using them for a few years. Great speeds and customer support.
edit: perhaps their RCS option would be cheaper if you know exactly how much data you need to store in advance.
600x WESTERN DIGITAL Ultrastar DC HC550 18TB (10800PB in total) - $500 each
~$350k in hardware, up to 20kW energy consumption, should fit in two rack towers. You can host it for about $1.5k somewhere. All assuming no redundancy :)
So toss in at least one SRE type person. Say $200k/year.
Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one....
Since redundancy is a thing, double that $350k. And 10pb is what they have now so double it again for 20pb. Add in $10k per rack for switches, routers, wires, etc.
So probably you are looking at a million dollars of capital plus labor to actually execute on this. And don’t forget the lead time might be a month to get the hardware and a week or two to install it. Plus all the configuration management that needs to be built up. Not to mention monitoring. So maybe a quarter of work just to have it functional.
I haven’t even factored in opportunity costs. What could this business be doing that adds more value than building out a little data center?
I dunno. Maybe it does make sense to manage your own hardware. But it helps to calculate the entire cost of ownership, not just the cost of the servers.
This person's entire job is managing a few racks of hard drives? How often do you think they're actually going to get called in?
> Since redundancy is a thing, double that $350k.
True, but you can do redundancy for cheaper with parity or tape.
> And 10pb is what they have now so double it again for 20pb.
> So probably you are looking at a million dollars of capital plus labor to actually execute on this.
You can go a couple PB at a time if the upfront cost is daunting.
> Add in $10k per rack for switches, routers, wires, etc.
Yep, though that's not very much in comparison.
> Plus all the configuration management that needs to be built up. Not to mention monitoring. So maybe a quarter of work just to have it functional.
This is the one I'd really worry about.
> I haven’t even factored in opportunity costs. What could this business be doing that adds more value than building out a little data center?
You always have to keep opportunity costs in mind, but something like this can pay for itself in under a year if there's significant bandwidth cost too, and that's an amazing ROI.
Not often. But the server gods are a cruel mistress and it will definitely shit the bed when you are on your honeymoon, or maybe the day after your first kid is born.
At this volumes you probably do want a carbon copy at another site to mitigate disasters like datacenter fires.
How about every day?
Quick guess how often disks need to be replaced when there are thousands of them. ;)
Also for one or two thousand disks I would expect less than one failure per week.
- network switch Juniper EX4600 (10Gbps ports) + 3rd party optics ~$11k
- cheap 1Gbps switch for management access <$1k
- some router for VPN for management network - $500
- 1Gbps (not guaranteed) internet access with few IPs ~$350 / month
- 100Mbps low traffic internet access for the management/OOB network.
Time to get the hardware - 2 months. Time to rent and install hardware in rack - about 1 month. I don't count configuring the software.
This setup is full of single points of failure so I would consider it one "region" and use something like CEPH + some spare servers in each "region". That way you don't need to react immediately to hardware failures. Just send a box of hardware from time to time to the DC and use ~$20-40h/h remote hands service to replace the failed drives or whole servers. You could also buy on-site service from the hardware vendor for 1-3 years adding some cost.
I think the most important thing would be have a cleaver person who design a fault tolerant system, automatic failover, good monitoring and alerting so that any on-call and maintenance job is easy and based on procedures. That way you could outsource it. Only then it might have some sense.
Also, the hardware can be depreciated, which reduces its net (of taxes) cost dramatically over time.
Five years (probably the useful life of the equipment in general) of $210,000 per month is $12.6M. That's a lot of savings.
Regarding accounting, the AWS monthly charges are also net of taxes so it makes no difference.