Open Source Continuous File Synchronization
github.com
github.com
Instead, I have Syncthing set to sync a half-dozen folders (Pictures, desktop, documents, audiobooks, etc.) between my phone, laptop, and desktop over the local network. In total, I sync more than 150 GB of files and it works almost instantly and seamlessly. I use the SyncTrayzor GUI for Windows and the Syncthing app for Android. The Android app occasionally needs to be restarted manually, but that's the only problem I've had with it.
Importantly, I keep local copies of everything on all of my devices, including my phone (256 GB storage plus a microSD card slot which I have yet to need). Lack of support for Syncthing on iPhones is one major reason I stick with Android, even though there's a lot to like about the platform. Once it gets too big I'll move a bunch of old stuff to backup/archive drives, but I don't expect that to happen for a while.
Now, I use Google Photos to selectively share photos with others, and that's it.
Even so, Dropbox/OneDrive have other problems, such as only syncing a single folder and not syncing over the local network. Plus it's expensive.
Seriously though, this can't possibly be THAT much more difficult than fragile attributes on a filesystem node. If someone at Dropbox is reading: please, I beg of you.
[0] https://f-droid.org/packages/com.github.catfriend1.syncthing...
Tresorit has that feature and it works great. Aside from data encryption, it was the main reason why I switched from Dropbox.
https://support.tresorit.com/hc/en-us/articles/217103697-Exc...
https://github.com/syncthing/syncthing-lite
Unfortunately, it is not maintained, even if it works and I use it daily.
so drop off an old machine at your parents / buddies / exs house, give it internet connection and cron some updates + reboots + power on after an outage and boom, if your house and all your devices burn down, you can get your files via decrypting with that password you saved to your email / password manager.
I love Syncthing and it's replaced Dropbox and Microsoft 365 for me. ok i keep the latter for online dictation when I'm not near a pc with dragon installed on it, but still, I don't need to trust MS with my data in their cloud.
wtf does this even mean?
So schedule auto updating, automatic restart and login so that it always get back to syncing so that you don't have to actively maintain anything.
Edit: actually it’s the .stversions that I was thinking of. Deleted versions can go there.
>It’s amazing how great computer products can be when they don’t need to deal with corporate bullshit, don’t have to promote a brand or to sell its users.
I wanted to sync my personal files between three computers, two desktops and a notebook, which don't always see each other. At first, this worked great. Obviously, there were conflicts when I accidentally edited something in two places without syncing up. What I didn't figure out was how to permanently resolve those. I ran into endless cycles were I had a conflict between two systems, resolved that, but the third one hadn't seen the conflict resolution, which produced another conflict, which I resolved, which the first system hadn't seen yet, ad infinitum. At least, that's what it looked like. Among other errors, files long since deleted on all systems would still show up as conflicts.
To be fair, I didn't actually lose any data, and it's very possible I did something wrong that would be obvious to other people. But to me, it wasn't at all clear what and why, which scared me enough to move away from this setup. I don't want to take a lot of chances with my data.
It seems like synchronization is the next logic step and I could really use feedback and testing from interested users. The application is available at https://github.com/prettydiff/share-file-systems
I would love for syncing to occur upon changes in real time as opposed to timed intervals like a cron job.
So if you have 3 machines that you need to sync, designate one as a hub which syncs with the other two, and the other two do not see each other. If you absolutely need to transfer data between two others as well, set up another completely independent share between them (a different directory on both machines).
I am not sure but fully automated replication setup with cycles might be impossible due to https://en.wikipedia.org/wiki/CAP_theorem.
(Addendum: I think actually what is needed is even stronger - you should make sure before editing files that the designated "central hub" node is available and has been synced.)
SyncThing is very handy, but not something I’d use as a backup solution.
Can you explain further?
I imagine what happened is that this was somehow related to the fact that the PC and Mac have case-insensitive file systems, and the Linux machine does not. Somehow one of these machines renamed the folder (I have no idea how this could have happened). I would hazard to guess that SyncThing on one of the nodes (probably Linux) decided it needed to delete the "backgrounds" folder because it no longer existed, and then sync the new "Backgrounds" folder. If deleting the "backgrounds" folder was synchronized back to the PC or Mac though, they would have also deleted the "Backgrounds" folder, since in case-insensitive-land, they're the same folder.
I went digging through their github at the time to see if there was an issue about case sensitivity and didn't find anything that was relevant and open. I didn't have a lot to go on, and my attempts to recreate it failed at the time, so I didn't bother opening an issue. It's entirely possible my specific issue is fixed by now.
One of the few projects I'm sponsoring on Github.
I do know that Syncthing has a way to keep older file versions around for awhile but I find that a bit cumbersome to use, especially during restore.
So, I am curious to understand more on how you use Syncthing as a replacement to Dropbox. (I'd love to do the same but don't for reasons I wrote above.)
If you need to go back to an older version, pick it in .stversions .
If anything, this seems simpler than dropbox for me. Does Dropbox still make you use the web UI to restore old versions?
For simple undo operations (recover a recently deleted file), Syncthing's version history is good enough.
User for about 3 years, after I left Dropbox (they incorrectly tagged a shared file as copyright-infringing - it was a bit too invasive for comfort).
I sync about 8TiB between 10+ devices, ranging from tablets and TV to desktops and laptops.
Since relatively recently, they also solved case (in)sensitive file names, which used to be a pita when changing file name casing only, but that’s all history now. Big fan!
OneDrive also opens all uploaded pictures and automatically tags them based on the visible content. As long as an online service doesn't offer end-to-end encryption it's a safe bet that they'll read all your files (famously leading to the Google Drive-based DDOS attack scheme[0]).
[0] http://chr13.com/2014/03/10/using-google-to-ddos-any-website...
I tried it with multiple different devices and know someone who has the same problem. At the same time I know many people it works flawlessly for. Has anyone else had a similar experience?
I've switched to KDE Connect for manually transferring files from the phone (it can't be scheduled on Windows as far as I'm aware, but at least it works every time).
[0] https://docs.syncthing.net/users/faq.html#what-things-are-sy...
The thing that really annoyed me was that SyncThing synced .DS_Store files, leading to tons of unnecessary transfers and conflicts. They could be excluded via ignore patterns, but those patterns needed to be added to every folder—there was no global option. To fix this, I had to edit the code and create my own custom SyncThing build.
Resilio Sync says it does partially [0] but IMHO not to an acceptable level (it keeps Finder tags for instance but loses Finder icons and completely cripples legacy Mac Fonts).
Note that .DS_Store files are another different thing altogether. They are standard files storing the view state of their parent folder (and, incidentally, Dropbox does not sync them [1] and Resilio Sync is set to not sync them by default).
[0] https://help.resilio.com/hc/en-us/articles/204754729-Alt-Str...
There is also a setLowPriority setting which I believe should prevent it from taking all CPU, but not sure since never had such problem on Linux.
My secondary computer is a Windows 2-in-1. It's frequently running on battery power, and during that time, I often wake it up, check something, and put it back to sleep - all in a minute or less, repeated a dozen times an hour. If each time I turn the screen on Syncthing is going to run at full throttle for a minute or more, it'll drastically cut the effective work-on-battery time.
(This article popping up on HN reminded me that I wanted to check Syncthing out. But what 'gnufied described is a total deal breaker for me, so I'm not going to bother for now.)
I am a fullstack web dev so creating web app or electron app is right up my alley, but not in distributed databases or peer to peer stuffs.
I was looking at OrbitDB the other day, but I'm curious can it be programmed to just connect peer to peer to my family computers ip addresses?
Also, as far as I know, I can't upload blob data to OrbitDB, and has to use IPFS instead. Can this syncthing solves this problem?
Or more generally, is there any OSS software that already solves this problem? I'd love to learn how to build it.
The whole system doesn't need to be high performance, it is just for family uses.
there is a function to ignore deletes, but it is hidden in the advanced settings which are difficult to access (some of those advanced settings should be hidden, but ignore delete is not one of them i think)
another thing i miss is an easy ability to see if all files have been sent. currently the status only shows if all files from the remote device are received, but i care more if all local files are sent (and thus backed up)
the point is that it's complicated and the 'ignore delete' is easily missed. also because it is not shown in the status. so every time i have to wonder, did i set this up correctly?
and i have to check the status on both the phone and the backup-device to make sure all is right.
when i want to delete something on the phone i have to check on the backup-device to make sure that the file is backed up. because the phones status does not tell me that. it only tells me whether it downloaded everything from other devices which is less usefull, i can see which files are on the phone anyways. i can't see what's been uploaded to the backup-device unless i go look.
just the other day i realized that for a new folder i had not set up ignore-delete. luckily, because i am paranoid, i made a hard-link duplicate of the backup before starting to delete files on the phone.
if i sync deletes from the backup device to the phone then a cron job to move files would potentially cause files to be removed from the phone before i am done with them.
rsync would cause the files to be duplicated (although i could still hardlink using rsync)
ah for rsync I meant with --remove-source-files flag -- it's just sometimes rsync is easier to use than move, depending on the hierarchy of your files
unles you mean to sync first, then move files i want to keep on the backup only out of the way and rsync --remove-source-files those files again.
the problem with that is that when i do remove files on the phone, then it's because i need the space. at which point i don't want to wait.
syncthing does what i want. that is not the problem. them problem is only that it is easy to forget to set up ignore-delete, and that it's hard to monitor the sync status of the backup from the phone.
i appreciate the suggestions, but they don't really do much to help my case. what is really needed is an improvement of the syncthing UI and in particular a redesign of the way the status is handled.
for my backup maybe i'll set up an rsync with hard-links to create a second version of all files which protects them from accidental deletion.
It has all the features you desire, but the UX/UI is streamlined for synchronization, not archiving.
It's transport agnostic, so WebRTC or websockets or whatever is fine.
Works in browser or Node.
Switched to Syncthing something like 2 years ago and it has been solid for me across Mac and Linux so far.
And I do moderately crazy things like using it to sync git repos to create replicas across multiple machines instead of manually using git across 5 machines and a hundred or so repos.
Only complaint is that initial synchronization is really slow.
You could set up your own git remote for this, Gitea is all the rage I've heard. But git remotes can be implemented many ways, no need for a dedicated server running.
I was syncing files on 3 laptops on the same WIFI and the speed was just abysmal.
That was about 2 or 3 years ago so maybe it's better now.
The only slow bit I know of is when it's initially syncing too many files (e.g. tens of thousands). Even if they are tiny, it takes a while. I suspect it has something to do with the index serialization, but also there is a maxConcurrentWrites setting which is 2 by default -- perhaps on an SSD this is an unreasonably low default? Not sure.
Is it possible that you didn't have inotify enabled (fsWatcherEnabled)? There is also fsWatcherDelayS [0] setting (10 by default), which groups changes to avoid excessive sync operations, so it might be percieved as latency.
[0] https://github.com/Catfriend1/syncthing-android#readme -- I recommend the 'fork' over the original app, it's got some nice extras. Although seems that Google doesn't want it on play store https://github.com/Catfriend1/syncthing-android/issues/704
[1] https://docs.syncthing.net/users/config.html#fswatcherdelays
When I was syncing over mobile hotspot (LAN) on 2.4GHz, speeds weren't impressive. But then I got a 5GHz router with 3 devices and there is sufficient bandwidth. I share ~120GB of total data, spread across ~20 directories, largest being WhatsApp media backup with thousands of crappy low-quality photos. If I wait over a week, I easily accumulate more than a GB of data and it syncs in few minutes at most. Absolutely robust, and zero friction.
But notably missing: iOS.
If you are looking to fully replace Dropbox, you need a server to serve files off of that you can access from the Internet. You could do this using a NAS running Syncthing for example. (There’s at least a couple different ways to get it working on a Synology NAS.) That way, you could have a CIFS or WebDAV share to actually browse from mobile devices.
The developer made it with a little help from the syncthing team. There are comments there: https://forum.syncthing.net/t/mobius-sync-ios-client-now-in-...
I've been using it for some months, and it works fine, when it works. I sometimes have to start it manually because iOS killed it in the background.
My workaround for this has been to install iCloud on a Windows machine which then downloads all the photos from my phone, and Syncthing on Windows monitors the iCloud folder and shares it with a Linux machine that does my backups. A bit convoluted but it works.
[1] https://github.com/icloud-photos-downloader/icloud_photos_do...
And apparently these days they do offer a shared storage space, anyway.
My wishlist for syncthing though, in prioritized order:
1. support for symlinks, to be able to sync files/folders that live outside of syncthing dirs. Some files I’d like to sync are in fixed locations that I cannot or don’t want to change, and Seafile allowed me to simply symlink them into an already synced dir. The counter-argument in GitHub discussions is having to distinguish between syncing symlink destinations vs. the symlink „as symlink“. For my situation that would be overthinking (as I’d never have the second use case) but I understand the hesitation, sadly.
2. provide a smart versioning/rollback solution leveraging ZFS snapshots, as my file server as participating node already creates rolling snapshots
3. (nice to have) mark certain directories as „sparse“, to sync data only when really accessed. Similar to what OneDrive provides
It doesn't matter which one I push/pull/commit/whatever from, the others mirror the operation.
I started trying it last year as part of an effort to reduce my third-party & cloud services & effectively got rid of Dropbox.
The one trickier piece of the puzzle for me was mobile device support. I use Documents.app on iPad & iPhone combined with SFTP to my machines. Two versions per machine — LAN IP access or port-forwarded WAN access, depending on where I am. All setup with SSH keys so it’s really just one tap to get into whichever home machine I choose.
Syncthing is great and has improved a lot over the years. It only hiccups when it detects changes in VeraCrypt containers which are still mounted and thus inaccessible to read.
I've heard several mentions of it being used in place of services like Dropbox, OneDrive, Drive, Box. For the features above, I agree.
What Syncthing is missing is usability for folks that use the proprietary services because they are easy to set up and maintain and share files from. For that I recommend Nextcloud. It is centralized, so perhaps a different use case. But certainly a more complete replacement if you (or people you support) are used to Dropbox.
Also, just curious, but why is this on hacker news today? Was there a new announcement or release for Syncthing?
[0] https://github.com/sireliah/dragit
* Note *: author here
More tightly integrated, much faster, lighter and polished app overall, has good iOS client.
I have also tried Unison, which is closer to my needs, but it has its own issues like having to be compiled with the same version of Ocaml on all servers.
- Frequently I get sync conflicts, although I always edit file only on one host.
- Sync is not triggered automatically, but based on the schedule interval. If I need synced files immediately, I need to open the control UI and force rescan of files.
Pretty sure there's a "Watch for changes" toggle in the UI.
I was pointed to syncthing and never have had any serious complaints. Good job
It was surprisingly able to punch through firewalls in some cases where Tailscale couldn’t (even though both use UPnP and similar stuff).
When two devices can't find each other on the local network they look for each other on public discovery servers and then pick a community provided relay to connect. The data is end-to-end encrypted between both devices, so the relay doesn't see the file contents. This works out of the box on almost all networks. Unfortunately IPv6 doesn't seem to work with their public relay network though...
It uses external, community-run relays for that.
Other advantages are opensource, privacy, lightweight, fast. Happy with it for months now.