Migrating 1 terabyte of files from OneDrive to Nextcloud | Hacker News Reader

Migrating 1 terabyte of files from OneDrive to Nextcloud

herrherrmann.net

122 pointsherrherrmann141 comments

I once lost some important files stored on OneDrive. These were pretty marge media files. There were downloaded as a zip file, apparently successfully. Then, I deleted those files from OneDrive.

Imagine my surprise some time later when I went to access those files. Upon expanding the zip file, I saw that the media files in question had been replaced by some text files, with their content indicating that the download of those media files failed, and instructing me to try to download them again.

What the heck, so I could download a thousand files, which realistically I can't go through one-by-one to check if any have been replaced by a text file, and OndeDrive does not alert me to the fact that some files could not be downloaded.

I already had issues with the slowness and clunkiness of OneDrive, but this was the last straw. I'm migrating all my data from there.

I can confirm this 100%.

We have multiple customers that send us data via OneDrive, and for all of them multi-file downloads do not work.

Microsoft's ZIP-Streamer to download multiple files simply does not work.

The text files contain exceptions with contents such as:

    PK^C^D^T^@^H^H^@^@<80>L^PS^@^@^@^@^@^@^@^@^@^@^@^@9^@^@^@__myfile.png_Error.txtThis file cannot be downloaded. 
    ExceptionType: ZipMeTAException. 
    CorrelationId: 08132d8f-77b7-4c75-a66b-345e8a15c340, 
    UTC DateTime: 8/16/2021 9:36:53 AMPK^G^H<FB>%Mw<A4>^@^@^@<A4>^@^@^@PK^C^D^T^@^H^H^@^@<80>L^PS^@^@^@^@^@^@^@^@^@^@^@^@-^@^@^@myfile.png<89>PNG

This is nondeterministic per file, so you have to either manually click every single file in a folder to download everything to avoid ZIP streaming (you apparently cannot use any API to download files that were just shared via link and no other auth), or repeatedly "download the ZIP to convergence" -- which is exactly as silly as it sounds.

Since Microsoft has been outsourcing development and QA for some of their consumer products quality has gone downhill.

The picture browser is barely usable. Keeps scrolling back to the top. The iOS file provider sometimes doesn't respond. Downloading 500 photos locally on an iPad results in a unresponding app.

> the media files in question had been replaced by some text files

this is why a download program should verify with a hash (which should be embedded into the name of the file, or some kind of tracking metadata).

My experience with Nextcloud was underwhelming. Regular crashes, very slow sync if you have thousands of files, full reupload on any change and ignored files that do not get synced with no warnings. I had made five accounts for my family and each of them got a separate blocking bug. The worst part was the syncing of shared folders.

I ended up using Seafile, which is mostly open source and has beek rock solid for the past two years. I'm not looking back!

+1 here. Nextcloud is a pile of PHP scripts, while Seafile splits and diffs individual file blocks, and absolutely flies at 'whatever the lower of of your network and drive speed is', handling any file sizes you can throw at it.

On the other hand, Seafile is also a binary distributed by Chinese team, and if I was a Chinese secret service, I'd build in all the backdoors I can, which I guess already happened.

> On the other hand, Seafile is also a binary distributed by Chinese team, and if I was a Chinese secret service, I'd build in all the backdoors I can

This was the reason I tried and subsequently passed on Seafile. It’s one thing if it’s just my stuff, but I was setting it up as a syncing server for friends and family. I just wasn’t comfortable with this prospect and being responsible for others’ files, as well.

Their documentation says they can provide end2end encryption, but the docs also seems to contain a number of caveats that a security professional might find problematic - or not (IANASP): https://manual.seafile.com/security/security_features/#how-d...

It's not weak as such, but there are gotchas in this faq (mostly related to metadata and key file caching) that effectively mean a casual user won't have end to end encryption. That said, I do not know of a cloud provider that does this well. You could just as easily build on top of this service with rclone (and the crypt backend) as any other. Then it is probably fine, but then you are already not a casual user.

I will say that in the west it is probably better to have your personal life invaded (which for a casual user is the reality you must face. It will happen) by China than the US (broadly your choices due to prism), as they are less likely to have an impact on your day-to-day functioning.

> Seafile is also a binary distributed by Chinese team

Web search for Seafile turned up a GitHub repository, so seems like it's open source? That doesn't rule out the possibility of a carefully-hidden back door that no one has found yet, of course. But I think it should increase confidence over a closed-source binary distribution.

Regardless, this level of xenophobia is getting a little tiresome. If you have evidence that this project is run by or sponsored by the Chinese government, then sure, I'd find that a showstopper as well. But a group of people, who just happen to be Chinese, building something shouldn't immediately be grounds for dismissal. China is a very big place, with a truly staggering number of people, and the Chinese government -- contrary to popular belief -- doesn't have its hands in everything its citizens do.

This! He says OneDrive is buggy, but that’s nothing compared to Nextcloud. He’s in for a big disappointment.

Seafile is a much better choice if you’re only looking for FileSync (without the whole App Store thing).

Another new and already better and faster contender: ownCloud OCIS, which is a complete Go rewrite of ownCloud and is already very fast and efficient and works like a charm for file sharing.

One thing that made me shy away from Seafile was the limitations of its Android client. IIRC it could only upload photos, not sync folders. So couldn't use it for notes, voice recordings, downloads, books...

I considered Syncthing as well, but I wanted both the arbitrary file sync and the web-based features. Figured I'd rather deal with Nextcloud's poor performance than cobble together a crappy Nextcloud clone via three separate apps.

I'm definitely keeping an eye out for OCIS though.

Pretty much exactly the same position. I use NC for file-syncing, WebDAV, and the webinterface for those files only, but it’s not exactly great. But for similar reasons as you, it’s still the solution I’m stuck with for now.

Syncthing works well for me (I use it also on Android to sync photos and notes) but yeah, if you want web access that's not the right tool for the job

If you are looking for sync only there is Syncthing

As a sysadmin managing Nextcloud on an on premise VPS, we have none of the problems you mentioned. We have 20+ accounts, tons of shared files, and whatnot.

Just upgraded to Nextcloud 25 (literally 20 minutes ago), and no problems whatsoever.

I deployed it on a virtualized server which is not extremely powerful, and everybody seems to enjoy the productivity boost it brought into the team.

I upgraded to Nexcloud25 and all Gui Customization's are gone...once again. I don't really like the php-nextcloud, using it for well over 4 years there is always something not right. I really hope the go-version is better.

Isn't the go version limited to owncloud? Are they open sourcing the new code base?

Looks like NextCloud is also thinking about migrating to Go.

We have 150+ and had a bunch of problems, users with weirdly long file/dir names having some problems, and sync very rarely being flaky.

The performance is underwhelming (it's PHP after all) but we just threw CPU power at the problem...

snark remark... Has nothing to do with PHP but more with the implementation.

As with javascript a lot of PHP's problems are not inherent to the language but rather to the culture around it. It must be avoided for that reason, but it is hard to state that in full every time. So folks use the shorthand you see.

What is tons in this case? Thousands? Tens of thousands? Hundreds of thousands?

Surprising that your and the OP's experiences are so different. Based on the description, I think it's safe to assume the OP also has thousands of files, some of them very large.

I wonder if the issue is with Nextcloud itself, or with the particular hosting provider.

As polished as the UI looks. You can’t cover up the fact it’s a PHP turd under the hood.

Would it run better if it was a "Python turd" under the hood?

Likely, although that’s not ideal either. Would want to see it in something performant like Go or Rust.

Not relate with the post but a rant against OneDrive a couple of weeks ago came to my mind:

My partner uses Windows 11. She works and also studies in college. In both places they use Office360 so, she has her laptop set up with her Microsoft accounts and switches between college one and work when needs it. The other day, she came for help asking why the college OneDrive was getting full when she only has documents. Turns out that the fucking OneDrive decided to synchronize the whole user personal folder (where she has very personal stuff) into the college's OneDrive. Surprisingly, disabling such a wrong behavior was not straight forward as I expected. Microsoft has managed to make it ambiguous enough for having to spend a fair amount of time on such a simple thing.

I don't know it's just another example of them trying to shovel down your throat their cloud services or just some obscure plan to simply have access to all your very personal data (or both).

This sounds like College IT is using Known folder redirection to have files backed up seamlessly to Onedrive.

Source: Msft Employee

So, you are telling me that my girlfriend's college IT department, has the capability, through Office 360, of manipulating which folders to backup on the clients side? Man, I really hope it's Friday and my brain is exhausted and not getting what you said straight, otherwise, this weekend is gonna be a session of urgent OneDrive removal and Linux indoctrination for her.

When you log into a business Microsoft account on Windows or in Office, it asks you if you want to give your business control over the device. She might have answered “yes” to make the prompt go away (and “yes” is a sensible answer on a corporate laptop, but definitely not when using a personal laptop for work/university)

I would assume that the college IT dept is using tools made for businesses, which businesses will usually set up on machines they own (that is, not employee personal hardware), so in that case it's not unreasonable for the business to be able to set up which folders get backed up. Of course, it's never a great idea to store personal things on your work computer, for this reason and others.

But of course that model isn't a great fit for university students, who will usually not have a laptop issued by the university. Not an excuse for this happening, but I can understand how it would.

Seems like that's true: https://learn.microsoft.com/en-us/sharepoint/redirect-known-...

I remain baffled at how presumptuous Microsoft is about OneDrive and its behavior.

One of the big issues we see with Office365 in the browser are people working with two different Office365 accounts in the same browser, which causes a lot of issues. Known folder redirection, which is the ability for us to turn on automatic syncing of the Desktop, Documents and Pictures folders on Windows machines has been pretty helpful for us. So many people dump everything on their desktop and if the computer dies or files have been accidentally deleted it saves them. Known folder redirect is now rolling out to Mac clients, which is more of a pain, as some of our users sign into iCloud and sync those same folders.

I know way too many people whom this has happened to. High school accounts, college accounts, even work accounts.

Even when it's a personal account, that feature messes up the C:\Users\username folder layout so then some apps save things into the non-OneDrive desktop folder and there's no way for an ordinary user to find them. Not to mention syncing broken program shortcuts (.lnk files) across devices and so on. This whole synchronization feature is a sysadmin nightmare slapped on top of a filesystem that wasn't designed to support it.

Its a hack and ugly. But lots of system admin folks love it and between them pushing it at work and MS pushing it its hard to avoid. Its reaaaaal shitty though, especially when they enable it on someone with a huge profile and you're in a baffling in-between while its trying to sync.

Our business migrated from OneDrive for business to a self-hosted Seafile [1] setup (we run it in a docker).

Seafile has a syncing client, and a drive client (Seadrive) for viewing files as if it was a shared drive.

Since moving to Seadrive I have not had a single support issue, nor heard of a single issue syncing, whereas I'd have them weekly with OneDrive to the point we'd have to delete their local files and re-setup OneDrive from scratch.

OneDrive truly is an abomination built on top of archaic Sharepoint.

[1] https://www.seafile.com/en/home/

> OneDrive truly is an abomination built on top of archaic Sharepoint.

OneDrive _for business_ is built on Sharepoint. Onedrive ... erm ... not for business (?) is an entirely separate product, and they don't interoperate.

Which is another baffling decision by Microsoft branding, since for regular users they make no distinction. It only becomes apparent when you discover you can't work with people outside the organisation.

> OneDrive _for business_ is built on Sharepoint. Onedrive ... erm ... not for business (?) is an entirely separate product, and they don't interoperate.

That is oh so weirdly common in MS ecosystem... like Skype for business being Lync and being significantly worse than "just" Skype (no idea what MS teams voice chat is based tho)

Teams is literally Skype for Business with a different skin, there's loads of traces of Skype left over where they haven't bothered to change the name.

Such as the process name in Linux for instance.

And now they will start to roll out Mesh which may or may not overlap with Teams (and Skype/Lynx behind that)

Mesh?

Teams for AR/VR environments

https://www.microsoft.com/en-us/mesh

And teams for business...

>It only becomes apparent when you discover you can't work with people outside the organisation.

You can share externally with Onedrive for business...

It's a setting in Sharepoint admin...

Once again, people here have no idea that most of their complaints with O365 is how management sets the security policy... not the platform itself...

https://learn.microsoft.com/en-us/sharepoint/turn-external-s... :)

You are right in that it used to be bad.

Groove.exe... The old Onedrive for business client was a horror

> You can share externally with Onedrive for business...

Yes, but not (simply) with OneDrive, as the branding would suggest. Lesson learned, anyway.

Sharepoint is complicated enough as a system that it requires its own certification programs. People should really learn to get competent IT and ask them questions instead of trying to figure it out without any relevant skills and then complaining. I get it, most of us have been told and have witnessed that we’re much above the average joe when it comes to computer interfaces but that’s because a lot of low level stuff is intuitive and not as complicated as this. Just swallow your pride and ask questions.

Ive been using seafile for over 5 yrs in my business and it has so far worked flawlessly!

There are only very rare hiccups with binary files such as excel sheets or PSD files, where seafile will create a “SFConflict” file. Havent figured out how to fix that het.

I think the author misrepresented the NAS option a bit. I have the same Sinology DS220j and from my experience, you definitely do not need to constantly install/manage software, configure network access and renew certificates. Yes, these tasks pop up every now and then, but I don't think you can say you're "very busy" with something that takes 1 hour/year.

There are also significant benefits of running your own NAS - no big tech company has your documents, and they can't lock you out on a whim (such as if your Google or Microsoft account gets suspended for sometimes unfair reasons). The access speed is much higher with the data on LAN, and therefore there's little need to keep the data duplicated on your machine. And you have offline access. Security options are generally better than with most cloud providers - you can configure the firewall to deny access to anyone but whitelisted IPs, or do the many other things you can do if you own your firewall.

Finally, it's not true to say that all of your files would be gone on a NAS if something happened to the server. This goes back to the saying about data copies "two is one and one is none". You can set up backups to another NAS overnight, or use one of the cheap cloud storage services not meant for random access like Backblaze B2. Of course, the NAS itself is vulnerable physically - if it gets caught in a fire, stolen, or flooded for example.

Hey, author here!

I agree that objectively, it’s not much work to maintain the Synology NAS once everything is correctly set up. But I personally just got tired after attempting to configure the network access alone for the nth time. My goal was to make the NAS accessible remotely (i.e. from outside my home network) and I felt like that was already a big task (comparing different DynDNS providers, port forwarding, asking my internet provider for a static IP address). And the recurring tasks (mostly renewing certificates for the , as far as I remember) would annoy me every year or so.

And I agree with the advantages you listed about a NAS setup! That’s why I tried one out, after all. I did run into some scary security situations (once the DynDNS was set up, someone was immediately trying to log in via brute force), but nothing too major, fortunately. And I never got the network setup right (e.g. to use the local network speed when I’m home and still access my files the same way when I’m away via the internet). So, these were only theoretical advantages that I could never really get to myself (I’m sure other people figured it out, but I didn’t exactly find good documentation on these things online either).

And regarding your last point, I also generally agree! Although it feels silly to me to have a NAS at home and then still do regular backups via another storage/backup service elsewhere. I wouldn’t want to back up 1+ TB of files every day or even week, although there might be diffing solutions for that.

Anyway, I hope this answer makes it clearer why I decided against a (Synology) NAS in the end. I’m sure other peeps figured it out and are happy with their Synology setup, but for my purposes I’ve never really gotten there.

Thanks for the additional comment.

> I personally just got tired

Its true that a Synology NAS takes some effort to set up initially and a little bit of maintenance work every now and then. But I don't think about my NAS most months. And on the months I need to, it's usually a short task that I do along with other house chores.

> I did run into some scary security situations (once the DynDNS was set up, someone was immediately trying to log in via brute force)

Yes. Unfortunately, that's pretty common on the internet. You can auto-blacklist IPs after a few incorrect login attempts, and firewall-blacklist IPs out of your country in general. These two measures should almost eliminate the brute force spam, and you should be able to verify that they are effective in Log Center. However, you also have other options because you own the firewall if what I suggested doesn't make sense (like if you are traveling a lot).

> Although it feels silly to me to have a NAS at home and then still do regular backups via another storage/backup service elsewhere.

My backups are be automated with a Synology app (like Hyper Backup). It uploads only changed files. The cloud storage on Backblaze is pretty cheap - I think I pay under $2 per month. But this gives me peace of mind for the "NAS destroyed in a tornado or something" scenario. On the other hand, keeping files in a NAS gives me peace of mind that I own what I have - no cloud storage provider can take it away from me.

All in all, I'm happy you found something that works for you. I think you value simplicity and time more, and I value control over my data more. So our choices differ. In any case, I just wanted to offer context from a NAS user.

> Although it feels silly to me to have a NAS at home and then still do regular backups via another storage/backup service elsewhere.

I do exactly this; a nightly backup to S3/Glacier. I use duplicity, so it only uploads diffs from the previous night. My NAS is a 4-drive RAID5 setup (RAID is not a backup!), but if I lost more than one drive -- and that has very nearly happened once -- all that data would be gone, forever. It only costs me a few bucks a month for that added peace of mind.

Thanks for writing this all up! FWIW, I have gone that NAS route in general though with TrueNAS (formerly FreeNAS), I didn't like Synology myself after trying it. I think remote access works better if you just do it via modern VPN. Wireguard has been incredible for me, and there are options like ZeroTier and Nebula for more advanced automated meshing and so on. For me at least I've found it better to have less vs more in a given component, so the NAS is focused pretty purely on storing and making available data and that's it, network services are all done elsewhere (in OPNsense for me), same with VMs. That keeps things much simpler (admittedly for HN-values of "simple"), the NAS and various other services can simply act purely as "LAN". Also makes it easier IMO to keep things better isolated and secured by avoiding any public WAN exposure except for VPN ports (and even that can be further offloaded with a cloud-based bastion or the like). I also just made my own internal CA, which means I'm free to set certificate lifetimes for 10-20 years if I want. Linux/BSDs of course are straight forward to add your own CAs too, and least with Apple devices as well it's super, super easy and free to make up a mobileconfig profile containing all CAs/certs and various other configs, then install is just a few clicks that even non-technical users can follow. Let's Encrypt is certainly cool and easy to setup in OPNsense but I don't think it's necessarily ideal for purely internal services with nothing public facing.

100% agree however that documentation is often lacking, though the OPNsense and FreeNAS docs remain pretty good.

>And regarding your last point, I also generally agree! Although it feels silly to me to have a NAS at home and then still do regular backups via another storage/backup service elsewhere. I wouldn’t want to back up 1+ TB of files every day or even week, although there might be diffing solutions for that.

Every backup solution I know off only syncs diffs, that's pretty table stakes for precisely that reason. For units that support snapshots, there are also plenty of controls for dealing with snapshot lifetimes and pruning. All of that certainly still is more upfront to setup however!

I think the big reasons for the expense and effort of a NAS though are mostly down to if you can/will make use of the speed and scale advantages. The security aspect might be a bit of a wash, one can do E2EE anyway with a typical cloud provider, and while they're more publicly exposed they've got much more serious effort behind them too. At 1TB it's probably not worth the effort. But other options definitely get more expensive as one gets into 10-20+ TB, and the value of more serious data integrity and snapshotting and so on too. 10 gigabit and even more is also getting quite reasonable to do on LAN, it's easy to saturate these days, and particularly for those of us living in rural areas with mediocre uplinks that can be genuinely handy.

I feel fortunate to though to live in a time when we all have so many options to meet our exact needs. Possible for a home or small business to put together options now that would have been top end enterprise stuff not that long ago.

Nextcloud on Hetzner has been great in my experience. Running for a few months now as a Dropbox replacement and very happy. Mobile app, mac app, it all seems to work exactly as expected.

My only gripe is that they do not allow Nextcloud external storage[0] but that's understandable given what they're selling.

[0]: https://docs.nextcloud.com/server/latest/admin_manual/config...

https://docs.hetzner.com/konsoleh/storage-share/configuratio...

Seems like they do now, you just have to turn it on manually.

Thanks! had not seen this yet.

Good to hear that you haven’t run into any issues for a few months, at least! Some peeps warned me that Nextcloud is also quite buggy, but I didn’t run into any issues yet myself.

I used syncthing[1] for that althought use case is a bit different - syncthing is mostly to sync the data between many devices (p2p) without option to have partial sync on a directory; you can have more than one synced directories but "one big directory" will be "one big directory" on every machine.

So I have my phone pics synced to my NAS and PC, my blog's markdown files between server and my computers, another dir for some music etc. There is also option for simple versioning in case you need to dig out the old files but it's not integrated with OS as it is "just a directory".

Disadvantages is that you need at least one other device running to sync as it is P2P (with some community-ran server to forward if you're NATed out of sight) but other than that, literally zero problems.

[1] https://syncthing.net/

> no partial sync

That's not really true. You can specify which files you don't want synced. So, on my phone i can say "don't send me files that match the following pattern" https://docs.syncthing.net/users/ignoring.html

So in your example you could make a subfolder "big files" and exclude that subfolder from being synchronized to your mobile device.

I also love syncthing. I've got it set up to transfer my phone's pictures to my PC where I regularly sort through them and weed them out. And I've got a shared folder that's replicated across all my devices that contains my Obsidian vault, my KeePass file and other useful things

As a current user of Dropbox, Google Drive & OneDrive I can definitely say that Dropbox beats the other two (and probably any other similar service) in both network transfers (up/down) and indexing time (especially if you consider LAN sync too for when mirroring data between devices). The level of file/folder movement you can do in Dropbox while it's sync'ing data is unparalleled. If you try doing the same e.g. with Google Drive, you will either end up with duplicate files or even lose data overall.

But we all know that this probably comes at a potential cost, e.g. privacy related concerns (anyone remember the controversial Dropbox board appointment of C. Rice back in the day?), services getting hacked etc.

So knowing the above and knowing that with Nextcloud, Owncloud, Syncthing etc. you fully control your cloud backup, any (perhaps) lower performance takes a step back for sure. But then again, if you're hosting on a server that's geographically close to you & on a solid datacenter with great peering (e.g. Hetzner, Scaleway/Online.net or OVH for Europeans), performance can be on par or perhaps even better compared to "established" backup cloud services.

Several years ago, I had a personal Dropbox account with lots of stuff in it. I joined a company that also had a corporate shared Dropbox. Since I couldn't sync multiple Dropbox accounts with the same client on a single computer, I joined my personal Dropbox account to the company one.

Later, when I left that job, the company removed me from the corporate Dropbox account. Dropbox removed all the corporate files from my local machine, and also all my personal ones. And deleted my personal files from my personal dropbox account, too. Luckily, I had another backup I was able to restore.

Now, I never link personal and corporate accounts of any kind, and stopped using Dropbox once they stopped supporting Linux.

Personally, I really love Syncthing https://syncthing.net/, if you do not need an iOS client.

I have it set up on all my families windows machines, my personal machines and between two NAS. One of the NAS does nightly backups to backblaze.

For using Syncthing on iOS, there's Moebius sync, which encapsulates the syncthing client as a one-time paid application.

Works pretty flawless.

Agreed, though you do have to export your photos and videos to another folder before you can sync them. I do that weekly.

Owncloud/Nextcloud... beware. It's significantly worse than the other services.

I've tried really hard to make it work (including self-hosting), but it's just poor software, for a few reasons.

Worst problem, which I haven't experienced with the other services, is that if there is a connection problem (this may happen routinely, if servers have scheduled downtime), files may go out of sync. Result is that, ridiculously, one may get conflicts even if they're using a single client in total. There was an issue opened, which has been ignored and autoclosed for a while.

Second, lack of centralization causes a bit of a mess. One never knows which version the servers ship (although one may not care); hosts may be less realiable, and finally, hosts may change frequently (I've see a few changes in the list of free servers) so one is never sure how long the service will be offered.

Also, the configuration shipped with the Nginx version was not great. Overall, I think that the software "isn't great", and I don't trust it for personal data.

It wins by amount of features but yes, software craftsmanship is generally shit.

Hell, I managed to kill our corporate server by navigating to settings page. How ?

Browser auto-filled fields

JS auto-submitted that as changes. Didn't verify them working before submission either.

What browser (for some fucking reason, I didn't click any confirmation) remembered in LDAP settings was no longer a correct config so it auto-broke any time I navigated into settings page.

If comparing on costs, hosting it on home HW is by far the best way to go. I run Rsync and keep the backup in my shed so it is off-site.

Nextcloud provides way more than data backups. It is best in class for personal internet services. My family and friends use my server for syncing their contacts, calendars, tasks, kanban, music, RSS, bookmarks, passwords, notes, and finances to all their devices. Ours is also setup with Talk for voice and video conferencing, polls, forms, phone tracking, CMS, mail, maps, activity pub social media, and there are a ton of other possible apps. It is perfect solution to avoid Goopple.

> in my shed so it is off-site.

My neighbour had a lightning strike hit their robot grass mower. It fried every electronic device on their parcel. Including half the non-connected power tools in their non-house-connected shed. It was a wild thing to witness the aftermath of. Basically anything electric that had been semi-close to the robot grass mower "stop here" line circling the grass, got fried. The plastic of on none-electric tool literally melted. Nobody got hurt, thankfully.

Your shed may be on another planet for all I know, but the amount of damage caused sure made me respect the "two separate physical locations" more than I used to.

That is a good story. My shed is in the corner of the yard. I physically walk my backup drive back and forth once a week and keep it in a box unconnected to any electrical. Call me a cheapskate, but I don't mind.

Yeah I bought some cheap server with storage off OVH and keep a syncthing instance there (with archiving enabled) + some backups just for that reason.

I kinda thought about also getting a solar/battery backed small server that was connected via WiFi for backup for galvanic isolation (for case when it hits nearby power pole and not building directly) but eh, would need to be in noticeable distance from everything else to make sense

Why is home HW the best option?

You'd have to:

- Purchase the hardware & replace if it breaks

- Install the software & keep it updated

- Pay for electricity

3.2€, that the Hetzner 1TB storage box costs, would be cheaper than the current EU electricity cost alone.

Presumably you'd be using your regular PC to host Nextcloud. You are correct that you'll spend more in electricity than a Hetzner sub costs, of course. (The 1TB tier costs 5.11€ now btw.)

However, Hetzner storage isn't encrypted (AFAIK) so I would _never_ put my personal files in there. I pay the extra cost gladly.

There's lots of alternatives, of course - right now Icedrive is the one that I find most interesting, as it has similar prices to Hetzner, provides nextcloud-ish features and is E2E encrypted. I still need to look into how trustworthy they are though.

> 3.2€, that the Hetzner 1TB storage box costs, would be cheaper than the current EU electricity cost alone.

4.29€ in Germany. And I’d need a server that constsntly consumes 14.5W to make Hetzner cheaper, that’s 3-5 times what my homeserver averages.

I have an HP Microserver Gen 10, average energy consumption is 35W. (4x HDDs) [0]

And it's considered one of the most energy efficient.

If you have one that's more efficient, it's probably running Raspberry Pi and SSD storage or you're not measuring from the outlet.

Intel J4105 (Futro S740), running off SSDs. Here are a few user measurements [0], technically in German, but it’s mostly names. Power consumption should be below a PI4 with the same SSDs according to the internet ;)

But yeah, with 4 HDDs you will use a lot of power, that is to be expected, HDDs are power hungry. I’d guess they are responsible for most of the consumption.

[0]: https://github.com/R3NE07/Futro-S740/blob/main/power_consump...

edit: Also, your server is running AMD Opteron X3421 from 2015. That’s even older than the already old J4105. I doubt that’s currently considered the most energy efficient anywhere.

I've really wanted one of these, but the price is what stops me in the end. For the benefits I get from using a VPS, the near wash in price in the end pushes me more towards using the cloud.

~$400USD in the used market for the chassis. Add another ~$100 each for four drives, maybe one extra as a spare, so we're at ~$900 for just the hardware assuming your used box had enough RAM and what not for your needs.

I have cheap electricity right now at ~$0.09/kWh. 0.035kW * 24 * 30 = 25.2kWh/month, * 0.09 = ~$2.27/mo in power costs. I won't add internet costs as I'd have the same internet connection with or without the server.

Over 5 years, the power cost adds up to ~$136.20 at my current rate, which will probably only be higher over the next 5 years. So total costs assuming I don't have any hardware failures outside that one spare hdd over that period is then $1,036.20, or $17.26/mo.

Comparatively, I spend $15/mo for a VPS with a large cloud provider doing similar tasks as what I would use that microserver for. For doing things outside my house, it has far better networking than what I could get at home. It has automated backups stored at that cloud provider. It has way more reliable power than just using my mains (I didn't include UPS costs above!) and generally more reliable internet services. Its running a way newer (more features) and faster processor than that Gen 10 microserver is running. It doesn't take up space in my closet.

There's definitely networking tradeoffs to consider. With the VPS, obviously I'm limited to my internet connection performance and increased latency between the two sites. But conversely, I probably get better performance on most networks outside my home to that VPS. It then really depends on what kind of access patterns you should prioritize.

At the end of that five years, I do technically still have that hardware whereas at the end of 5 years on my VPS I still have a monthly bill. I agree that hardware has an excellent chance of living way more than 5 years, but at the same time it might break somewhere along the way and last you less than 5 years. I definitely have hardware that's lasted over a decade though, so I'd lean towards it lasting a bit more than 5 years.

There's pros and cons both ways to think about, and I don't necessarily think either direction is the answer. The answer really depends on what you're wanting to prioritize.

I do constantly think about these Gen 10's though, they seem pretty awesome!

I use a Raspberry PI 4B with some MVNE drives. The power consumption is somewhere between 2 and 10 W. I have an extra RPI and backups. Hetzner does have good pricing and is easier for most people than running own HW.

> 3.2€, that the Hetzner 1TB storage box costs

That looks like a really great deal.

They don’t have an issue with those apps? As I use NC for Syncing, I thought I’d try the other apps. They were all horrible. RSS, Music, bookmarks, notes, recipes, contacts. They are all far below what any of the dedicated replacements offer. The unified interface is nice, but doesn’t make it worth it for me.

I describe it as best in class rather than best of breed. Having one instance of Nextcloud rather than a bunch of different containers for the different functions is easier to maintain and backup. I don't know when you tested these apps, but they have been progressing rapidly. I find most are really good now.

Been a year. But the apps were so far behind standalone services, they had a long way to go

Why didn't you stay with pcloud, again? You mentioned encryption was uncomfortable, but in the end you selected nextcloud which doesn't have reliable encryption.

*I am a pcloud user since 5 years ago

I’ve used pcloud for 7 years, it’s been very stable and i even use it for backups (700GB daily).

It’s the only solution I’ve been able to send at consistent 1Gbps speeds to (except when it’s many tiny files).

About encryption, you unlock it once per restart and it stays open. It’s not like you have to enter password every time you move a file..

(For the backups i use rClone with encryption)

Same here, happy pCloud user (unlimited plan) for 2 years now. I didn't get the encryption, but it goes on sale from time to time for $75 lifetime.

I encrypt all my files locally before sending so the encryption doesn't matter for me.

I use rclone with --transfers 32 (for smaller files) and it seems to work fine without throttling.

You’re right, pCloud is not a bad alternative! I remember being annoyed by the password prompt for the encryption feature, but another user here wrote that they only need it once after every restart, which seems fine to me now. Back when I tried it for the first time (2 years ago) I decided against it because the OneDrive migration was not fully working (many files were skipped and it wasn’t transparent which ones or what was the reason), and the support couldn’t help me figure out any viable solution to migrate all my files. Nowadays with the NAS the migration would have been easier, I suppose.

Aside from Dropbox do any of the mentioned providers offer auto save functionality for docs and spreadsheets? For me that’s really the killer feature (that and the fact that I paid $100 for 7 years of service using MS’s employee discount offering for everyone during the pandemic).

Every time people talk about self hosting or using a different cloud provider my biggest concern is auto save. And I haven’t heard anyone address it.

I am running Nextcloud v23 with the Draw.io package installed using the autosave feature. Works great. I am looking to install the Only Office (or LibreOffice) to see if they also have the auto save feature.

When you save a document (ie ctrl+s) Nextcloud detects it and uploads the new version. It can even do some versioning.

Clicking ctrl+s is not autosave :)

That is saving manually, and then the file gets synced.

Autosave you don't need to click save/ctrl+s

https://support.microsoft.com/en-us/office/what-is-autosave-...

>AutoSave is a new feature available in Excel, Word, and PowerPoint for Microsoft 365 subscribers that saves your file automatically, every few seconds, as you work.

That's a feature of your editor, not the cloud sync tool. If you're editing a file in Vim, for example, called work/foo.txt, then it will create a hidden file called work/.foo.txt.swp and periodically autosave to that file. And if you have Nextcloud set to sync the work folder, then it will automatically back up .foo.txt.swp as well.

https://vimdoc.sourceforge.net/htmldoc/recover.html#swap-fil...

If you use app with auto-save it will auto-sync. Many apps have that feature.

It's been around since Office 95.

Next cloud plus only office. Gets you realtime collaboration and auto saving. Web based and desktop based editors.

I was looking into how I could set up Hetzner Storage Box as a storage device for Nextcloud. Somehow I didn’t even realize that Hetzner offers their own hosted Nextcloud instances (Storage Share).

In case you still want to build it yourself: you can use sshfs to mount a Storage box on your Hetzner server instance.

Yeah, it seems to be not well advertised. Have been running one for a few months now and so far it's excellent as a Dropbox replacement, mobile app and all.

How is one company replacement for another company?

Both have access to data.

It doesn't exactly save your privacy and it won't matter to everyone, but Hetzner is European.

iCloud+ 2TB: 9.99 €

does the macOS and iOS backup. Click buy and pretty much done. Also have my parents data backed up with family sharing, I don’t have to be family IT support!

If you have other devices… Tim Cook recommends replacing those with Apple devices /sarcasm

Only requirement not met is not having a plan that supports more than 2T.

Yes, Apple owns my life. But happy about it for now, good value.

I’m happy with it in general, but it feels like iCloud has a higher sync latency than some other services like OneDrive. And because it is a black box, there is no way to force sync it. I would drop a file for example to sync up to my old MacBook that acts as a CNC controller, walk over and OneDrive would reliably be there. iCloud however…

Yes I sometimes have to airdrop a file because sync isn’t happening - yet - for no explained reason.

OneDrive is just too slow and clunky for me on MacOS

Dropbox just kept failing to start a sync from iPhone (likely entirely due to the OS not giving it a chance) but was otherwise very reliable and fast.

Google … I just use for a second copy of my photos and occasionally check if it’s syncing okay. It usually seems to be.

For photo backup I use my Amazon Prime subscription. The client isn't amazing (what is?) but it seems to do the trick.

iCloud is good stuff, at least for ease of use. I use it to store my photos and passwords. Photos are then automatically synced to my Mac, which is then backed up to Time Machine. Same goes for the iCloud Keychain.

You could say I’m 100% on Apple working decently, but with 3 copies I feel relatively safe.

I wish there was a third party tool to occasionally verify the whole library to ensure photos don’t rot or just disappear.

Ah, you reminded me that I did actually try out iCloud as well! I was unhappy with how the sync worked on macOS – basically lots of magic in the background without any idea what is happening and how long it might take. And you couldn’t even turn off the laptop in between sync attempts, as far as I remember? And their online services are quite far behind the competition as well (no proper folder sharing without requiring the receiver to have an Apple account, for example).

But yeah, I’m still keeping the lowest-tier iCloud subscription to sync my photos via Apple Photos, which is quite nice to have.

If only iCloud had E2E encryption ...

This!!! I always laugh when Apple advertises its supposedly superior privacy features...

For sensitive stuff I placed a encrypted sparsebundle on my iCloud drive.

>good value

one terabyte of HDD space ≈ $12

But an external HDD is not going to sync to a central server or other devices. icloud/onedrive etc is not just the space but also redundancy and backup and sync bultiin.

sure, but still, the profit margin is such that you're getting literally milked

Value here is not just the amount of storage:

1. Multiple people and devices use this one account.

2. Said multiple people are not technical - they don’t always know/remember to plug in a drive to their device for backup.

3. The mental load reduction is amazing. The iCloud keychain feature + faceID + multidevices is so convenient, I don’t understand why I ever spent effort in crazy tool and file plumbing schemes. I use the $2.99/month plan, and my time is worth a lot more than that (at least 3000 times more using paycheck…). I’d rather spend time with friends and family doing other things than debugging our collective backup schemes. That’s value.

you're not paying for the hard drive, otherwise they'd ship you the hard drive - you're paying for the management.

What about backblaze b2 with rclone, I'm sure there's a front end somewhere and with the clousflare cdn it's no egress cost. Or clousflares storage amounts

I've made one such frontend for not only backblaze but also Nextcloud (via webdav), and pretty much every protocol like FTP, SFTP, S3, ... It's all open source: https://github.com/mickael-kerjean/filestash

The trouble is that backblaze is actually quite expensive for home offerings. When I looked a few months ago, the cheapest provider (per TB) by far was actually mega.nz, and they offer a platform with no upper storage cap (if you pay for it). Works with rclone, e2ee, and some other "side benefits" too. No idea what the native clients are like, however.

the big issue with Mega is always how sustainable they are.

Its a business with a questionable legacy, and I have no idea what the longevity is of the service. Your data is probably fine, but it attracts a certain clientele that might put the service at risk long term of another seizure or whatever.

I also use Mega for big files, but not as my primary backup solution or anything important.

So this article is about looking for a perfect solution, not finding it, and finally compromising on some of them, like end-to-end encryption, ease of migration...?

> if anything happened to my trusty server, all my files would potentially be gone

This is what a good backup solution is for.

While Hetzner's product and other hosted NextCloud instances come with some protection baked in I would still strongly suggest an off-service backup rather than keeping all your eggs in one basket.

(I'm assuming here that NextCloud is being used as the primary storage, rather than a backup location for other storage, if everything else is elsewhere already then the need for an extra full backup solution is reduced)

Interesting! So, you’d still back up your full cloud storage elsewhere (Backblaze has been mentioned by other users), even though your cloud storage provider does backups themselves (e.g. Hetzner claims to do several backups every day)?

Not full, but anything that is not easily replaceable. My documents, code, photos & videos, ete. This sort of stuff is on at least two providers as well as any original copies or copies synced/cached on other devices. And one of the backups is "soft offline" to ward against malicious tampering taking out everything (originals, backups, synced copies) in one swoop. A small selection of things (keepass db, a few legal documents, …) get the full offline backup treatment.

Anything I'm only keeping for convenience or sending cloudwards for transfer/sharing purposes, doesn't need the extra protection. Though it sometimes gets it as errors in that direction are better than risking failing not-safe with other data.

I see, thanks for the details! I’ve never considered differentiating between different levels of “importance” and thus different backup strategies, but it makes a lot of sense!

I backup (encrypted) to multiple places, including to Google Drive using a Turkish billing account on another google account. 2tb works out to about $30USD per year.

In terms of actual computer backups, thanks to btrfs+btrbk I can incremental backup every hour over LAN to my server and also to my attached USB drive. With compression/snapshots this is pretty small.

Thanks for the tip about the Turkish 2tb for $30/yr. Very good deal. Just did this. :)

Just be careful, I haven't heard of account locks, but I'm sure it can happen. I just use it for cheap storage. You could add your other account onto it as a family plan, so Account A (main) can share Account B's (Turkish account) storage. But given Googles track record for support... ¯\_(ツ)_/¯

Re the article and PCloud, the way the crypto folder works is that you get a mapped PCloud drive similar to OneDrive, DropBox, etc., and within there you get a "Crypto Folder". When you try and access that, it's empty. You then unlock it. However the article is wrong in that you are not prompted every time. You are prompted until it locks again, and you are in control of whether it locks automatically or not.

I've used PCloud for years now. I only unlock my Crypto when I need files from inside it, keep it unlocked and accessible for as long as I need, then lock it again. It's actually a really good way of working. Less valuable files are always available in the main mapped drive so working with them is easy, but the more secure option is available too.

It also supports Linux properly, and whilst not as fast as DropBox is certainly faster than Google Drive and OneDrive. And it allows multiple synced folders in addition to the mapped drive.

I moved on from Google Drive when I found that some of my documents, which I'd specifically requested be mirrored locally, were found to contain http links to the online version rather than an actual local copy. Meaning some of my local backups contained useless stub files. And the total inability of the web UI to give me simple folder size details was annoying - almost like they were encouraging me to keep filling the space I had.

I moved on from OneDrive when it wouldn't let me store files it thought might be dangerous (including `.js`, though I suspect they fixed that years ago). I also stored some older PHP code files in there (old enough to predate my git usage). When I looked at them they had an extra line of text added to the top of each file. I can't remember if it was a hashbang, comment, or text, as it was too long ago. I just remember my shock that they would change my uploaded files automatically.

Conclusion: after trying all those the article did (except, oddly enough given the article, NextCloud) I found PCloud to be ideal. For me.

I pay 4eur for 1tb managed NextCloud from Hetzner - it's a good service overall but it's unusable for photos. Apart from iCloud at 10eur per month are there any alternatives with a good photos (viewing, sharing, tagging, ai etc) integration?

>...but it's unusable for photos...

Hi @athrow, would you kindly provide a little more info here? What exactly are the issue(s) around photos? I ask only because i've been managing my own instance of nextcloud for years, and had seriously been considering moving to the managed Hetzner approach. (I love nextcloud, but just trying to reduce maintenance tasks that i do every month, etc.) So, any details or experience you would share around nextcloud and photos is greatly appreciated! :-)

Couple of issues:

1) It seems that some movies are placed first in the Photos screen, no correct thumbnails are generated for these. I'm unable to play the vide of these movies (.mov files). This means that when opening the Photos app I'm met with a wall of thumbnail placeholders.

2) It seems that for the photos the thumbnails aren't being pre-generated. There is a wait time until the thumbnails are generated. https://docs.nextcloud.com/server/latest/admin_manual/config...

3) Random error loading of the photos - some photos just don't open and display the placeholder thumbnail.

4) Linked to 1) - some videos appear between photos so I'm getting a mix of photos thumbnails and placeholder thumbnails.

5) As I scroll down the thumbnails are generated so there is a noticeable lag between scrolling and photos appearing. iCloud can generate thumbnails almost as quickly as I'm scrolling so the difference is noticeable.

6) Lack of any AI integrated into the product (face matching etc.)

Thanks so much for this! I guess because for my usage, most often I'm viewing photos and videos either via mobile or via my local directory (which is what gets synched to my nextcloud instance)...so i have not encountered some of these issues. But, now that you mention these, i tried a few, and see exactly what you mean. For me, it does not spell doom - though your list is spot on - because as noted i'm not getting hit too hard by most of these. But, this does help me because i am going to try nextcloud via Hetzner. Again, my goal is simply to lessen a few maintenance chores every month...so if i get a few cuts, maybe not so bad; especially since the Hetzner cost is quite fair. Thanks for your help; very informative and I'm very appreciative of your time and attention!!!

No problem, honestly I think for the price it's really hard to beat considering that you get unlimited (subject to fair use I'm guessing) traffic and backups included in the price. Also, supposedly v25 of NextCloud does offer big improvements (including for Photos) - Hetzner are yet to implement this.

Photoprism is good. Have you upgraded to NC 25 and photos 2.0? It is a massive improvement, and you can use Recognize app for auto tagging your photos using facial and other object recognition.

Hetzner offers a managed product -I'm still on version 24.0.5. The users aren't given the possibility to upgrade or configure the product. For example, I'm not able to turn on the option to create the photo thumbnails.

I see. I manage my own instance. Hopefully they will get you to 25 soon to see this: https://nextcloud.com/blog/press_releases/nextcloud-hub-3-re...

> On top of that, I became very busy with managing software, installing updates, configuring the network access

Ironically using a Windows as a NAS is easier because everything is point and click, reducing cognitive load.

You can also afford to be on a slower update schedule if you stick the application behind a NAT and proxy, which blocks 99% of drive-by attacks (because the attacker don’t even know where to look).

Interesting to hear that Windows would simplify some things! Upon reflecting about my time with the Synology NAS, I remembered that there was mostly the network setup I struggled with (DynDNS, port forwarding, static IP address, remote vs. local network access, etc.) – I guess this wouldn’t have been much easier on Windows either, I suppose?

It might be worth trying the Insync client instead of the official OneDrive one as I've found that quite slow to sync large libraries. It's far more customisable and runs on Linux too.

https://www.insynchq.com/

I'm thinking of moving off Dropbox. The 2TB plan pricing is quite fair, their sync client is far superior to competitors in terms of speed, efficiency and correctness.. but their plans above 2TB are just not worth the money to me (it gets more expensive per GB)

I have used a couple of OneDrive family accounts to host some media files that I wanted to preserve (yt video downloads, academic courses and papers collected from questionable soruces). What are the chances MS blocks access to my data in the next 1 year?

Is that a lot? That seems like very little data.

I assume this is the test run and they will move the rest over later or something?

Reply on news.ycombinator.com