Replacing Dropbox in favor of DigitalOcean spaces
mitjafelicijan.com
mitjafelicijan.com
For your secure backups, I can suggest Tarsnap by Colin Percival, who's a security expert and a frequent contributor on Hacker News. I don't get anything for recommending Tarsnap; I'm just a customer. https://www.tarsnap.com/
https://blog.digitalocean.com/an-update-on-last-weeks-custom...
I now use Linode for most of my stuff, and once they have cloud firewalls available in Dallas I'll be moving the rest of infrastructure over as I can.
I really need to write up my story into a blog post I can link to, but until then, I put a recap here a few days ago for anyone interested in the details. Mine wasn't as bad as some people's, but pretty scary personally: https://news.ycombinator.com/item?id=25806086
Given the sheer volume of dodgy packets that originate from DO network space, I find this surprising.
I get so much rubbish from DO space, nonstop port scanning at absurd volumes (sometimes totaling 6 digit+ pps), from customers that haven't been removed in years. massscan/zmap from people with a fake opt out page (that I shouldn't need to be opting out of), etc.
Someone once used a DigitalOcean droplet to run an aggressive layer 7 denial of service attack against a service I host. I notified DO via their web form, didn’t get a reply for three days, then got told to email to their abuse@ email address instead.
Dutifully, I mailed abuse@ and included the relevant server logs, gzipped, as an attachment. I got a reply that they won’t open any attachments, please copy and paste the logs directly into the email.
As an outsider, it seems to me as if they have few people with the appropriate technical background staffing their abuse department, so legitimate reports are left uninvestigated due to dumb technicalities like not being willing/able to open attachments, and bogus reports are accepted without thought or understanding as to the appropriate course of action.
Furious the second time, I sent a crafted image of an invalid ID to check if they actually even care. It seems they don't, as the account got unlocked all the same.
Of its own the blog post won't do anything, but the next time they're being overeager and shutting down a legitimate customer, you can link to that and corner them to fix their procedures.
On the other hand, my account is suspended from DigitalOcean, I have multiple droplets they refuse to give the data to, and they even had the audacity to keep billing me until I did credit card chargebacks. I still have no idea why I was banned.
Maybe because I used the GitHub student pack promo but I had already finished uni? I don’t know.
Backup of personal data is often a 1-way endeavor — here’s a dump of photos I scanned that I don’t need to look at anytime soon. But with tarsnap I can’t do that, nor can I then have the ease of browsing to the file and just opening it while it transparently downloads on demand.
From their site:
"The Tarsnap service is built on top of the solid platform provided by Amazon Web Services."
Tarsnap runs on Amazon.
Decent write up on the cryptography in restic here: https://blog.filippo.io/restic-cryptography/
Some tarsnap issues detailed here: https://www.kalzumeus.com/2014/04/03/fantasy-tarsnap/
The only reason I stopped being a customer was because another rapacious racket of a business, Comcast, introduced miniscule but enforced data caps in my area, so online backups aren't attractive anymore and I've gone back to external drives and offsite rotation. When I cancelled, Colin sent me a personal e-mail to make sure it was alright to delete my backups. It was probably the best exchange I've ever had with a service provider.
I don't doubt your happiness, but I confess that I'm having trouble reconciling "running it more like a utility" with charging 25¢/GB per month for storage. That is just staggeringly high. What I'm paying $6/month for with Arq would be over $160/month if I were using Tarsnap, and I'm getting end-to-end encryption, deduplication, and versioned file backups. What advantages does Tarsnap bring to the table that justify such a tremendous cost?
I'm not a tarsnap customer, but I think what you're paying for is a service built by a literal obsessive genius that will 100% work when the chips are down.
I believe this is the ticket that would add support for this to restic: https://github.com/restic/caddy/issues/2
In short, give rclone your keys, (small ~/.rclone config file) tell restic to use an rclone backend and add an extra argument when running restic. (See the blog link above, all explained)
There's an open issue for this which doesn't seem to have moved: https://github.com/restic/restic/issues/2134 - it does have a couple of PRs linked on there which implement proper soft-delete.
Looks pretty straightforward. I am surprised it is not more of a priority to get merged though, it is so essential.
Have they been around for long, and you know of any external public audit that has been done on them?
They are DIN ISO/IEC 27001 certified: https://www.hetzner.com/unternehmen/zertifizierung. Of course if you only store encrypted data you don't really need this auditing.
If I'm a normal end user, I will probably get a vastly easier to use product at about 1-10% of the price from Backblaze. And if I'm a serious business that can easily afford the > 10x premium and engineering to configure the backup I probably wouldn't want to entrust it to some company with what looks to be a bus factor of one and the apparent technical limitation that a restore might cost me a few DAYS of unanticipated downtime [1].
I mean, sure, if you are some unix nerd wanting to backup your dotfiles and a few small documents (or repos) for maybe around ~$50/year, why not go with some artisanal backup service for HN street cred (and a laudable open source donation policy)? But what other good use cases are there?
Am I missing something obvious?
Also, good encryption and security; e.g. Tarsnap is trusted by Stripe (https://www.tarsnap.com/testimonials.html).
Tarsnap does have real downsides - restores can indeed be slow, bare per-GB cost is high, and "like tar" is not a user interface that everyone will like - but there are definitely upsides, too.
I looked into setting up something of the sort for myself a little while ago, but unfortunately the pricing just isn't feasible if you have a large amount of data (say a few hundred GBs). You could easily end up paying more than $100/mo for storage + compute + bandwidth, while Dropbox and others charge ~$6-$10/mo for like 2TB.
I've been thinking about this kind of setup for my and my family's data, at least.
Furthermore self hosting a backup server at home doesn't protect against burglars and fires. You'll probably lose both your data and their backup.
That's why I prefer a local backup plus a remote one (a combination of git repositories and file storage. )
3 backups 2 local 1 remote
For example, https://www.server4you.com/dedicated-server has 2x 2TB for $25/mo in St. Louis, MO and unmetered bandwidth.
OVH Cloud in the U.S. has servers with 32GB of RAM and also 2x2TB drives starting at $59/mo and unmetered bandwidth. (both of these are French companies that are expanding into the U.S.) If you are interested in other locations (including Canada), check out ovh.com, kimsufi.com, and soyoustart.com (all part of the enormous OVH global footprint).
If you're in EMEA, or don't mind an extra 100 or so millisecond (SSH works fine), check out OVH and Scaleway (France), and of course Hetzner (Germany). Each of these offer dedicated servers for very reasonable prices, and copious amounts of bandwidth with reasonable pings to almost anywhere.
For example, https://www.kimsufi.com/us/en/servers.xml currently has 500GB hard drive servers with Intel Atoms available for $5/mo in both Canada and France, or bigger hard drives for just a few bucks more. This is OVH's cheapest line of servers, but OVH does a pretty extensive burn-in test before deploying them to you, and they tend to be pretty solid. (keep backups in case a hard drive fails, but that's always a good idea anyway.)
It says “currently being replenished” for the 2 lowest price packages in Canada
NextCloud and ownCloud can be set up to utilize Amazon S3 or Wasabi storage. This works really well and is very inexpensive. Both my GFs business files and my personal/work files all are backed up Dropbox style for less then $10 a month including web and mobile access.
I found NextCloud a little buggy compared to OwnCloud but it has more features and unfortunately I'm stuck with it for the time being and can't easily switch back to NextCloud. The sync works fine though.
What happens when s3cmd fails and after two months you discover the "Vault sync succeded" emails you have been getting were all illusory?
If the author is reading: please take the time to update your example, including proper error checking (at least "set -eu"), otherwise the people on the internet that are going to copy your script are in for a hard time.
Skeleton of the "incriminated" script:
#!/bin/bash
[...]
s3cmd sync ...
[...]
notify-send [...] "Vault sync succeded [...]"I will add what you proposed.
Thanks for your input.
set -e
You'll never regret it and might be very very very happy it was there. And if you start with that, then as you work on the shell script, you'll be more likely to make the script be idempotent (checking for files before copying, checking for lines in files before appending/sed'ing etc.) Idempotent >> non-idempotent for "alter the state on an end point" things.
#!/bin/bash
set -e
(
echo "Start of Subshell"
(exit 1)
echo "This shouldn’t run, right?"
) || echo "Subshell failed"http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -euo pipefail
IFS=$'\n\t'
for name in "${names[@]}"; do
echo "$name"
done
What's more, the double-quoted array solution will work even if the elements contain newlines or tabs.
Dropbox's killer feature is 'Online-only' storage IMHO, it allows me to store and access 300GB+ of music and other projects on my laptop without carrying an external HDD with me all the time. Every file is saved as a zero-byte file, which makes the files discoverable even though it's not stored on my disk. When I try to open the file it downloads it automatically, which works surprisingly well. And it's way faster than mounting DigitalOcean Spaces using s3fs for example.
Also Dropbox is cheaper: 12eu for 2TB, compared to DigitalOcean Spaces: 5eu for 250GB. And I can even access my files through the Dropbox API.
Sadly, for me, it is too expensive. It is 240 euros per year for a family of 6.
For comparison, the same plan in Google One costs 120 euros per year. Same for Office 365.
If they did 120 euros per year for the family pack I will migrate to them because I like Dropbox better at syncing.
But somehow I have the feeling that Dropbox > Google Drive > OneDrive at syncing. Anybody can confirm?
1) Get a Synology NAS.
2) Install the 'Synology Drive' package
3) set up 'Quickconnect' (~DDNS)
4) Remote backup can be though a number of providers (e.g. Backblaze) for cheap
Technical, non-experts in computing stuff can do this. Not quite simple enough for the general public though.
I then use cloudsync to backup to GCP nearline. All family pictures are there. Obviously, we don't store stuff that we don't want our mother to see. I take the risk of hacking, but a fire in my house is probably more likely than a hacking of my family photos on GCP. I figure just about everything else is ephemeral.
Plans and Pricing
The base rate of a Spaces subscription is $5/month and gives you the ability to create multiple Spaces.
The subscription includes 250 GiB of data storage (cumulative across all of your Spaces). Additional storage beyond this allotment is $0.02/GiB. If you cancel your subscription by destroying all your Spaces, your bill will be prorated hourly.
https://blog.sia.tech/introducing-s3-style-file-sharing-for-...
What offering is that? Their "storage" page quotes 26 euros for 5TB.
I had heard good things about DigitalOcean, but I would not use them or recommend them to anyone after that experience. The one primary issue I had was a showstopper but there was a lot of other bugginess.
Whole thing just looks like poor ad for DigitalOcean
But for my needs this is all that I need and it solves my use case.
And regarding DO, its just one of the possible S3 providers out there. One could choose something else.
https://help.backblaze.com/hc/en-us/articles/217666928-Using...
https://help.backblaze.com/hc/en-us/articles/360010017893-Ho...
I want something to store peertube videos and serve them. This sounds like too good to be true
Unless you live inside these environments, an easier option would be rclone[1]. I use it on a regular basis to sync large loads to Digital Ocean Spaces, including its bandwidth limiting schedule to avoid saturating the residential uplink at inconvenient times.
Can you share more about this please?
The biggest (related) issue I run into running rclone from home is hitting the request speed limit on Digital Ocean Spaces. I frequently get the "Slow down!" error when backing up a bunch of small (<1mb) files.
On the opposite end, I'd trust Google or Dropbox with my encrypted backup much more than a rack in someone's garage on a home internet connection.
There are tons of companies offering S3-compatible storage, and there's not much difference between them, other than price. Performance capabilities are similar; they all limit bandwidth. All the ones I've evaluated charge for egress over a certain threshold. Some are more reliable than others. For personal use, there's not a whole lot of difference between them for me.
One really cheap alternative for some use cases is Office 365. For $99/year, you can get a family subscription, which allows six accounts. Each gets 1TB of S3-compatible OneDrive storage. If you can handle it being broken up into six 1TB chunks, it's by far the cheapest storage out there. For comparison, Backblaze is $5/TB/mo, and Office 365 gets you $1.37/TB/mo. I didn't end up using this option because the authentication mechanism was inconvenient for my automated use case and I didn't end up needing so much storage, but it's worth investigating if you're on a budget and don't need automated access via e.g. rclone mounts (or want to take the time to overcome the authentication issues I ran into).
I evaluated about a dozen S3 storage options for both backups and as backing storage for a personal Plex server (of media I own) and eventually settled on Backblaze because it's cheap, easy to use, and I didn't need more than 1-2TB in the end (since not a single one can keep up with streaming video bandwidth needs, I ditched the Plex idea).
Similarly, I took a stab not too long ago at attempting to replace my Dropbox [1] - I opted for an implementation using Git and FSWatch.
Pro: version control for your files
Con: more expensive and takes more storage
dropbox client does so many more things (including compression,delta uploads,filehandle watch, incremental backup and so on)
i mean its fine if thats what you want,but that is not a backup solution by any means
What's sad is that increasingly people are feeling that the price for these easy, convenient services is too high, in terms of privacy loss, data lock-in, and so on. Are we going back to the early 00s, when the tech-savvy people would roll their own solutions and everyone else... just has to lump it? Are hackers just being too idealistic? Or do organizations like Signal show a possible way forward?
But yeah, I could use AWS S3. I agree.
> For customers that use Wasabi’s pay-as-you-go pricing model, Wasabi has a minimum monthly charge associated with 1 TB of storage ($5.99/month). If you store less than 1 TB of active storage in your account, your total charge will still be $5.99/month (plus any applicable taxes). See FAQ#4 for more details.
> With Wasabi minimum storage retention policy, minimum number of days are as follows:
> 90 days (default) for customers using Wasabi’s pay-as-you-go pricing model
Wasabi's not intended for small-scale use cases.
What, exactly is that level of control of $ spent ( as it is going to be lower than pennies ), that people want while maintaining reasonable reliability, stability, durability and scalability of storage.
> All I need to do is write a Bash script that does the reverse and downloads from remote server to local folder.
All I need to do is reverse parameters.
Documentation states:
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
But I need to test it out before I post that solution.
The magic of Dropbox is that it sync files across multiple computers and allows multiple people to share files. This doesn't do any of that.
You can use Dropbox for that, but there are definitely cheaper/ more robust solutions.
It does encrypted, incremental backups and can sync to many protocols including S3, DO Spaces, Dropbox, rsync, Mega.co...the list goes on. Super easy to set up. This is my go to backup solution for the cloud.
Only thing I’ve found which is even fit for purpose so far is rdiff-backup.
For personal stuff I’m just using iCloud and time machine to local disks.
In the end, it took me one evening to update every service that I cared about to use my new address. The rest (mostly e-commerce companies) I’ve been doing as I use them.
Do you just inform your contacts with an auto responder?
How long do you intend to keep the gmail account?
It copies remotely to /{day of the month} + /{latest}
I could easily change my script to run every hour.
It works pretty nicely and i have a backup for 30 days
( Outside of the default backup daily ofc)
Which isn't to say you can't learn a lot of useful practical stuff from it, just like you can from cosplay
A backup strategy without a recovery strategy already in place isn't a backup strategy.
> Show me any other vps provider that silently provides access to customer A's data to customer B after receiving commands from customer A to destroy their instance and then I'll believe you guys aren't at the very bottom of the "takes security seriously" list.
From: https://github.com/fog/fog/issues/2525#issuecomment-31337481
YC News Discussion: https://news.ycombinator.com/item?id=6983097
> You do not need to scrub or write anything to not provide user A’s data to user B in a multi-tenant environment. Sparse allocation can easily return nulls to a reader even while the underlying block storage still contains the old data. ... On top of all of that, when I pointed out that what they were doing was absolute amateur hour clownshoes, they oscillated between telling me it was a design decision working as intended (and that it was fine for me to publicize it), and that I was an irresponsible discloser by sharing a vulnerability.
From: https://news.ycombinator.com/item?id=20091026
> You've got an additional problem though, which is that this tells us you have two support channels: one that doesn't work (i.e. yours, the one you built), and one that does (Twitter-shaming). The first channel represents how you act when no one's watching; the second, how you act when they are. Most people prefer to deal with people for whom those two are the same.
From: https://news.ycombinator.com/item?id=20064169
Speaking of randomly locking accounts, the post-mortem kills me:
> The initial account lock and resource power down resulted from an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). These signals, coupled with a number of account-level signals (including payment history and current run rate compared to total payments) are used to determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers.
From: https://www.digitalocean.com/blog/an-update-on-last-weeks-cu...?
In other other words, DO will kill your account with a curt email staring simply: "We have reviewed your account and have declined to activate it. No further information or action is required from you." for simply using "too much CPU"! https://pbs.twimg.com/media/D76ocofXoAY_xB5.png
edit: Point being, I don't see how DO could get their act together as their business model essentially requires them to act like this. If they don't cut corners in one area then they will somewhere else.