Highly Available Block Storage
digitalocean.com
digitalocean.com
root@ubuntu-1gb-nyc1-01:~# time dd if=/dev/disk/by-id/scsi-0DO_Volume_volume-nyc1-01 of=test.dat bs=1024 count=10000000 10000000+0 records in
10000000+0 records out
10240000000 bytes (10 GB) copied, 58.0655 s, 176 MB/s
real 0m58.248s
user 0m2.608s
sys 0m41.604s
Some quick observations:
* Easy to add one when creating a droplet; by default they let you create volumes with these sizes: 100GB, 250GB, 500GB, 1000GB, 1.95TB; it's also really easy to create your own size.
* You can resize in any increments; took about 4 seconds to go from 100GB to 110GB with no downtime; you obviously need to resize/manage the mounted volume yourself.
* [Edit 1] Deleting the droplet does NOT destroy the volume. Worth keeping in mind when you spin them up/down.
* [Edit 2] Remounting an existing volume to a new droplet was quick and painless.
I can't help but I note that you benchmark the drive with a block size of 1024. Minimum block size of a modern SSD is 4096, anything lower would just cause unnecessary load. Also if you want to benchmark a drive/block storage I highly recommend using fio.
fio --direct=1 --rw=write --ioengine=libaio --runtime=300 --bs=4k --numjobs=4 --iodepth=32 --size=2G --name=4096-direct --filename=/dev/vdb --group_reporting
Also experiment with --rw=rw|randrw|read :)
(and make sure your device is large enough to hold numjobs*size)
There is also the problem about i/o syncing. By default OS buffer the data before writing it to disk.
In the case of dd, the real reason why it's often a misleading benchmark is that it's the best-case scenario: full-size streaming I/O where each block is only accessed once. Some applications do have that kind of operations pattern but most don't and can see very different performance characteristics because they depend on things like file creation/deletion overhead, random I/O within an open file, read and write contention if the same file is being accessed by multiple processes, etc.
we use fio with a standard test config across systems for meaningful results.
http://motherboard.vice.com/read/nra-complaint-takes-down-38...
Get out and vote for the right people this November.
If you chuckled at it, you should check Gary Johnson out. For real.
"We received notice on behalf of a trademark holder that a customer of DigitalOcean was hosting infringing content on our network. DigitalOcean immediately notified our customer of the infringement, and the customer was given a five day period to resolve the issue. The infringing content was not removed within the specified period even though several notifications were issued. Per DigitalOcean’s terms of service, a final reminder was issued to our customer and, when no action was taken, access to the content was disabled. The infringing content was subsequently removed by the customer and all services were restored in less than two hours."
> The infringing content was not removed within the specified period even though several notifications were issued.
You don't have to remove content under the DMCA, you can also file a counter-notice which gets the content host off the hook and then the matter goes to court[0].
But that also assumes DMCA which, if memory serves, was not in play here. It was a trademark complaint, which DigitalOcean has no responsibility to resolve.
Ultimately DigitalOcean's response, even with that statement, seems at odds with how the law is actually written. The other party also claimed they did respond to DigitalOcean, they just never removed the legal parody material which is their right.
DigitalOcean's understanding of the NRA's rights is more expansive than the law itself. Effectively their trademark policy is to automatically side with the trademark holder, irrespective of fair use[1] (see page 9+).
[0] https://en.wikipedia.org/wiki/Digital_Millennium_Copyright_A...
[1] https://apps.americanbar.org/litigation/committees/intellect...
Did you miss that? It's not infringement if it falls under fair use. They were not following their TOS because they did not confirm the content was infringing a copyright
The vast majority will not unless you are a large customer with your own legal staff on retainer to provide the appropriate legalese/notices/etc.
That said, the original reporting on this and the statement from DigitalOcean are at odds (Motherboard's update with DO's statement), and since I haven't verified either, I'll retract any specific support for either side of this particular instance.
That requires money for a lawyer to evaluate it. If the customer has their own legal staff that does this and relays that opinion to the host, as well as being large enough to cover any legal costs DO might incur, DO would be fine with it.
You are basically saying you are entitled to using DO's legal staff and financial resources in addition to the hosting you've paid for.
No, what I'm saying is that DO must already do this to some degree if they are handling requests, as otherwise I could send letters claiming trademark/copyright infringement for any number of things and get many customers shut down. If they have internal guidelines for what they do in cases when trademark/copyright infringement, I expect they follow those. I also expect that those policies do the minimum legally required of them. That's not because it's cheaper and garners good will from customers (it does), but because to do otherwise is taking sides in a legal situation without being an appointed arbiter of the law. Not only is this excessive, but it's anti-customer.
If DO is doing what they think they must by law, I have no problem with that, as long as that is clearly explained. In the case we were previously talking about, the statement from DO (at the motherboard article) is somewhat ambiguous as to why they did what they did. Per DigitalOcean’s terms of service, a final reminder was issued to our customer and, when no action was taken, access to the content was disabled. Was the take down required by law, or was DO overly aggressive in handling it? Without a statement as to why, (and I think that given some people's assertion that they went beyond what was legally required of them), their reasoning is somewhat ambiguous, and harder to call into question. If they clearly define they enforced their TOS based on what they believe is a legally required of them, then we can look at the law and their actions and evaluate whether that's true, and if it's not, DO can learn from the experience or be called out as a company that is capricious in their execution of the law.
What it boils down to is that "We received a complaint infringement. We enforced our TOS and shut down access to the content in question." leaves a lot open for assumption. I would be much happier if it was "We received a complaint infringement and as we believe is legally required of us we enforced our TOS and shut down access to the content in question." It's a small change, but it allows customers (and critics) a much clearer view on how DO handles situations like this, and allows for the public to make an informed choice on whether they think DO was correct in their actions (whether they really were legally required to do so). It's subtle, but I think it's a very, very important distinction.
It comes down to them making a risk assessment on whether they're more likely to incur cost from taking down versus ignoring. In some cases the balancing act is due to the nature of the outcomes: on the one side if you do nothing it's unlikely (low probability) but you could land-up in court and incur millions, versus on the other side refunding a customer their annual subscription and taking a reputation hit - it's the asteroid crash problem. It's notable that they asked their customer for a response and didn't get (what they consider) a satisfactory one. Who knows the specifics in this situation, but they won't have done it just on a whim!
i'm under the impression that "fair use" is a defense and/or judge's assessment, rather than something "civilians" can factually declare.
https://news.ycombinator.com/item?id=12010521
"""
11:02am - "you need to remove the content or fill out a counter claim"
12:47pm - "we were wrong, no counter claim can be filed and you now have 39minutes to remove the content or we are disabling the network"
"""
Then in the comments:
"""
5 days ago we got forwarded a forward of a takedown request that was sent to CloudFlare from a representative of the trademark holder. We responded to that forward within 22 minutes of first receiving it and have been in communication with Digital Ocean ever since right up until about 1.5h prior to be shut down.
"""
Believe who you will.
Edit: added 2nd quote from comments
As evident in the comments beneath your link, they had been in contact with DO for ~5 days at that point. Your quote leaves out important information and is misleading at best
I also haven't forgotten about their private blog censorship debacle.
DO is not a serious player and should not be trusted.
Unfortunately this means the pricing comparison is just wrong.
[1]: https://www.joyent.com/blog/on-cascading-failures-and-amazon...
[2]: https://www.joyent.com/blog/magical-block-store-when-abstrac...
[3]: https://www.joyent.com/blog/network-storage-in-the-cloud-del...
We've not had anything like those dark days with Persistent Disk. It's still true that having your storage across the network opens you to networking failures taking out your storage, but the gain in durability and maintenance pays for it (in our case, live migration would just be crazy with local spinning disks, we tried it didn't work).
Disclaimer: I work on GCE, and we want your business ;)
It looks like Exoscale does live migrations with locally attached SSDs.
Been there, done that, don't want to sit staring at consoles waiting for live disc migrations ever again.
It doesn't take doing this very often to make you realise that this is fundamentally backwards: you want the disc data already present on more than one storage server so that if one goes pop, you're not stuffed. Once you've done that, you can make the observation that hardware RAID is no longer necessary, and save yourself a layer of complexity.
[1] https://cloud.google.com/compute/docs/disks/#data_persistenc...
> Your instance might experience a short period of decreased performance
and how tolerant they are of unplanned reboots. I suspect what it means is that they'll throttle your VM (or maybe pause your IO) so it can't interfere with finalising the migration.
That being said, this is Google. They've probably thrown more man-hours at this than anyone else would think sane. I'll note that this is in the context of automatic migration away from "maintenance events" (whatever that covers) - it sounds like they think they've automated away a lot of the reasons we were having to keep an eye on things, but they're still vulnerable to hardware failure (obviously).
tl;dr: Yes, we do this for our scratch disk SSDs, but you wouldn't want to rely on this for a guaranteed durable storage (as it'll take a really really long time to move 100 GiB of HDD).
As soon as this rolls out to the region I've got that droplet in, I'm going to pull the trigger on it. I might even spend the effort to migrate my droplet to a supported region just to get this.
What are us data nerds supposed to do? We want to take 10 terabytes, run a batch process on it, keep the 20TB, then continue with about 5GB of working data until the next month's terabyte comes in, then we want to batch through the 21TB. Right now the price slider doesn't even go up to 21TB, and clicking on the "need more storage button" doesn't go anywhere, but I'm assuming it would be $2100 / month which is more than 3x as expensive than vanilla S3.
Are you going to keep each months 10 TiB forever? Something like Nearline is going to be a much better fit even if you process the data and have to pay the $.01/GB retrieval fee.
Disclaimer: I work on compute engine (so of course I want your money), but I'm honestly curious.
But yeah, we basically look to SSD for everything as that continues to take over market share from spinning disk and when you have to support a product 5-10 years, it's necessary to look ahead, so as SSD prices continue to drop we will eventually switch to SSD for everything and spinning disk will become the equivalent of tape drives in the old days.
At Hetzner, you can rent a dedicated server with 2x3TB drives for about $25/month ($0.005/GB, to scale to multiple servers, use Ceph) and a few larger machines for under $100. At OVH you can buy object storage for around 1c/GB and rent a few dedicated servers for less than $100/month, or use their cloud.
If you go to the _really_ low end, time4vps will give you a 1TB VPS for 2EUR/month, if you pay for 2 years (and they give you 4x the storage as bandwidth).
I don't work for any of these companies but I have services with all of them.
Price calculation is straightforward:
- $0.005/GB/month for storage
- $0.05/GB download
- $0.004 per 10k downloads
There is a base free tier as well.
The reason we focus on SSD exclusively for most of our services is that there are obvious performance benefits and as we look into the future the prevalence of spinning disk will continue to decrease over time.
That being said, if we are only providing SSD there are going to be customer use cases where it's not going to make sense, but that's also part of looking into the future.
So you should be able to create a larger striped volume to run through your batch processing and then depending on how you plan to access that data afterwards you can ship that off to an object store if it's going to be accessed infrequently.
You've got to remember that block storage is >> faster than s3/http. Its also has significantly lower latency.
Also if you want fast processing you need to start looking at shared filesystems/NFS
Disclaimer: I work at Joyent.
[1]: https://www.joyent.com/manta
DigitalOcean may have some value at the basement level of compute, but as a professional in this area there is literally no situation where I would use DigitalOcean right now because I value my time. AWS is already laughably cheap, the tooling is overwhelmingly superior, the resources available are better, and you aren't duct-taping together half-solutions and reinventing every wheel. (This is less an endorsement of AWS and more an endorsement of Not DigitalOcean; GCE is more than fine, Azure has some ugly bits around autoscaling that I don't like but you can get by.)
Anyone see a flaw in this? (I know there are other ways to achieve similar benefits - my files could be on S3 and the database could be a separate droplet etc but these introduced various drawbacks and added complexity)
> Anyone see a flaw in this?
If you're rebuilding from scratch because you're not sure that you can update things, then you're probably in need of a configuration management tool (I'm a big fan of saltstack[1], mostly because I don't like Ruby or DSL's, but there's lots of options out there[2])
If you're worried you're going to lose transitory data, it sounds like you don't have a trusted and tested backup/archival/recovery process in place. So having it stored on a single EBS / DO BS / etc means you're still exposed. If you're rebuilding and rolling data over, in this scenario, I'd be copying, rather than relocating, any precarious data repositories.
[1] https://saltstack.com/ [2] https://en.wikipedia.org/wiki/Comparison_of_open-source_conf...
But they'll only be in a known state in that case if your setup is extremely comprehensive.
In practice I've seen too many config management tools where long running servers have ended up in unknown states because changes have been applied, and subsequently changes have either been made outside of the toolchain, or changes have been done to the config in ways that doesn't let the tool know what has changed, or the tools simply doesn't have a way of comparing machine states without comprehensively enumerating everything on the server (e.g. people running Ansible playbooks that adds X, subsequently removing the requirement X from the playbook, and going about without considering whether or not X will interact with Y which they've added later).
As a result, I see rebuilding from scratch as largely orthogonal to whether or not you use a configuration management tool or e.g. build VM images that you replace wholesale, or whatever you do: You should rebuild from scratch regularly, as coupled with a test-suite it's the only realistic way of knowing whether or not you've left anything out of your build process.
My biggest caveat with config management systems is that they tend to end up encouraging live changes to a setup, instead of a build-test-deploy cycle. Sometimes that's necessary, but to me that's a last resort.
1. I am already using Ansible - but I want to run my "build from scratch" playbook every time I push a change instead of my "git pull etc." playbook.
2. I'm not scared of losing data because of backups or similar - it's because I want to start from a clean droplet every time I push a change.
I do know in theory that playbooks are idempotent and running the 'build from scratch' playbook on a running instance should result in the same state as a genuine 'build from scratch' but there are many places this can fail:
1. Someone has inadvertently modified the environment at some point. A quick-fix or accidental change or an alteration in my playbook that isn't clearly reflected in the running instance
2. Not all ansible modules are perfectly idempotent. It's a fairly leaky abstraction.
3. If you never build from scratch apart from the first time then you're not really testing your automated deployment. The next time you really need to build from scratch you might get a shock.
> 1. Someone has inadvertently modified the environment at some point.
> A quick-fix or accidental change or an alteration in my playbook that
> isn't clearly reflected in the running instance
> 2. Not all ansible modules are perfectly idempotent. It's a
> fairly leaky abstraction.
> 3. If you never build from scratch apart from the first time then
> you're not really testing your automated deployment. The next time
> you really need to build from scratch you might get a shock.
Your original post suggested the problem you had was with transitory data - evidently your concern is more to do with the ability to consistently and reliably rebuild a known and trusted environment.
Persistent block storage won't help you with that.
The reason I want a single droplet is mainly cost. I use one droplet per client rather than a multitenant setup. I could use S3 and a separate hosted db but for various reasons I'd prefer not to.
That said, how do you prevent a rogue droplet from going crazy and hogging up all of the SSD I/O?
If DigitalOcean isn't doing something limiting, they'll have to in order to prevent the situation that you're describing. You don't have to charge for it in order to prevent bad neighbors.
https://www.digitalocean.com/community/tutorials/how-to-use-...
"He was a bold man that first ran a production database on a brand new block storage service!"
If I had to pick two, those would be them!
With DO, at least there is an additional option available for users.
With GCE and AWS, the outbound bandwidth is expensive. 1TB = $90 (AWS, GCE) vs 1TB included (Linode, DO).
When they launched they just felt so far ahead of everyone else (1/4 the price of linode for SSD!). But price has been static for 5? years, so feeling lessing exciting these days.
This is EXACTLY the thing I need for some stuff I'm working on!
Will post results.
Disclaimer: I'm a Gluster developer, which isn't quite the same as being a Ceph developer but we do talk to one another. ;)
Block storage (RBD) is probably the most widely used interface (judging by mailing list and IRC activity).
Since I asked what the "backend" is, my question is still valid.
I did not get good speeds and I am wondering why that may be...
I think you might have misread the units: locally attached 50105 us (50 ms) vs block 546 ms.
The throughput numbers are on par with AWS and GCP block storage. This seems reasonable aside from the high latency.
I use DO mostly to compile stuff on Linux when i don't have access to a physical server, and storage size is always a problem.
This a main reason I still use Amazon AWS: I can create an instance and if it doesn't perform, upgrade it until it does. Then when I'm finished, kill the instance and save the volume. Next time I need it, just create the instance for the job, perhaps at spot pricing, then kill it again.
---
TWENTY times (edit) the price as B2 from Backblaze ($.10 vs $.005 per GB per month). It is one of the more expensive ones. But that gets you two things:
* (moot See edit 2) SSD! (significant iops improvement)
* (moot See edit 2) No transactional costs! (not sure if just between digital ocean instances, but they say none)
Improved performance and no transaction costs MAY (edit) be worth it for some applications.
Edit1: made it an order of magnitude cheaper in my head after looking at backblaze. It is no where close to the same price. Thank you for the catch, scq!
Edit2: and I'm just all kinds of off on this. Block not object storage. Essentially storage you can mount and move between digitalocean instances. That makes no transaction costs moot. You still have to get data out of the instance.
Thank you all for quickly catching how backwards this post was! I need coffee.
I believe B2 is an object store, not a block store (like this). The AWS analogy is block store is EBS, object store is S3.
(The TechCrunch article also got this confused.)
Not more like amazon EFS?
We'll, "expensive" is obviously a relative term, but why store data and not access it?
https://www.online.net/en/dedicated-server/rpn-san
There are several places you can get ~5 TB for +/-10% of the 1TB price at DO.
DO is offering a SAN at Object Store prices. :/