Syncthing: Open Source Dropbox and BitTorrent Sync Replacement | Hacker News Reader

Syncthing: Open Source Dropbox and BitTorrent Sync Replacement

syncthing.net

623 pointsushi184 comments

Can we please please please make it a standard that synchronization tools spell out how they handle conflicts on the front page.

Bittorrent Sync just overwrites files based on last mod time (terrible option). What does this do? Does it support backups? Versioning?

Their protocol is published https://github.com/calmh/syncthing/blob/master/protocol/PROT...

I think the point is also "this is actually critical information that is important for end users of the system to understand, not esoteric knowledge only required by developers".

I learned about Syncthing here[0]. The article mentions that there is no versioning at the moment.

[0] http://jasonwryan.com/blog/2014/05/10/syncthing/

Newest change wins. More detail: https://github.com/calmh/syncthing/issues/61#issue-27202590

Specifically, this is not dependent on last mod time, but instead a "version" field in each file's metadata that is incremented every time a file changes. I'm not really sure how that's going to help with concurrent (or conflicting) edits though.

What does Dropbox do?

Overwrites unless the file is locked (ie. a .docx open in Word that two people are editing concurrently). If a file is locked you get multiple files with "(X's conflicted copy)" appended to their names

They also usually keep the old copy around for a little while, or forever if you pay for the packrat option.

True, though I've found that very difficult to use in practice. I retrieved something through it once, but it wasn't pleasant. Especially with periodic autosaves, finding the last good version before an accidental overwrite requires trawling through pages and pages of old versions, with poor/nonexistent diff tools.

We use it all the time, with great success. We have one Dropbox (free account) for kids to keep their programming, drawing and other stuff in (one folder per kid). They lose their stuff all the time, in the most creative ways possible. Dropbox's 30-day version history can be a real lifesaver in those cases.

Although sometimes stuff is just gone and they're out of luck. I'm still not entirely sure what happens then, possibly a weird combination of everyone saving their project near closing time, the Internet connection maybe choking up a bit, and people shutting down their computer before everything is properly synced.

Conflicts are handled more elegantly than a simple overwrite. There are many ways to spin off a conflicted copy. This article covers the most common way: https://www.dropbox.com/help/36/en

Dropbox is pretty good about never losing data. It can be suggested that this is one of the most interesting or valuable parts of its technology. Its practical use of bandwidth is another.

Also what happens if one of the sync volumes becomes read only? Does it start deleting new things off the other volume? That happened to me with bit torrent sync and I lost new image files as a result.

Kind of a shame no one has implemented a source code control style interface or how OpenVMS did it.

What he's talking about is versioned myfile.cfg~

  $ DIR
  myfile.cfg;2
  myfile.cfg;7
  myfile.cfg;9
  $

Since we're listing alternatives here: I setup SeaFile (http://seafile.com/) a couple of months ago and I'm loving it so far. Mainly because it has client-side encryption and allows a single client to sync with different servers and selectively choose which 'libraries' (basically directories that are under sync) to use. Typical usecase is having a personal server for personal files and another one at the office for work-related stuff.

I couldn't decipher what SeaFile was or how all the components fitted together, can you recommend a review or write-up?

Components overview:

https://github.com/haiwen/seafile/wiki/Seafile-server-compon...

(Installed it yesterday on a debian LXC)

I use seafile as well - I host it on my digital ocean instance and it works very well and things sync very quickly.

best part is that it does automatic, granular snapshots of my data - I've stopped using git for things like papers because I know it's backed up in seafile.

I've been using Seafile for a few weeks as well, and have been pretty impressed. It's been a better experience then ownCloud, for me.

Having an open source syncing service with client side encryption is great.

Very much agreed. Seafile absolutely blows ownCloud out of the water.

I tried Owncloud for a while and was disappointed. Ended up using Syncdocs (http://syncdocs.com) to do end-to-end encryption to Google Drive. The main advantage of this was the versioning is done by Google and fast access from anywhere.

I'll be taking a look at Seafile now, looks promising.

I'm curious as to why people have had such bad ownCloud experiences? I've had it set up on my servers for 12+ months now and am really enjoying the experience. Could someone please explain further?

Also happy owncloud user for over a year; though i mainly use it for calendar+contacts (both synced with both phone and emacs) and auto-backups of pictures I take with my phone.

Documents and such I tend to keep in various public or private git repos; habits die hard.

My issue with ownCloud is that it lacks partial (differential) updates. Change a bit in a file and have a whole one re-uploaded.

Also, security track combined with the implementation language scare me to the extent of not really willing to expose the whole thing to the Internet. Just being superstitious here.

By client side encryption you mean you can use a friend you don't have to trust as a cross-backup system?

Also, anyone know how they compare? :)

EDIT: Login Log <- none for the free version?

you mean you can use a friend you don't have to trust as a cross-backup system

I mainly mean it has client side encryption, the consequences of that I'd like to leave open since nobody really knows or can predict all possible vulnerabilities of all possible systems. But yes basically: your friend would have a very, very hard time doing anything useful with the data or the incoming packets.

none for the free version?

not sure what you mean, I host the thing myself (serisouly, 5$ a month for a DigitalOcean box: best way ever to spend money:) but the site seems to claim there is 1GB free storage if you try their service?

Seafile sounds promising but I'd like to see someone review the code (call me sceptical).

I did, and it did not inspire confidence. https://github.com/haiwen/seafile/issues/350

I like seafile too, but after doing some of my own review of the source (https://github.com/haiwen/seafile/issues/350) I'm not willing to trust it on the open internet.

For now I run seafile inside a private network behind a vpn. But I'd rather find an alternative with a better security story. Something from a team that can justify their cryptographic constructs in detail.

I dont get it, why is deduping at a chunk level important, can't it just take it as a tradeoff and dedupe at the file level, or does that have the same problems? And why exactly is deduping important for a personal use case, bandwidth is relatively cheap.

In any case, while I want security in these systems to avoid mass survey style attacks, I assume anything I put over the wire to be compromised at least potentially. I keep the kind of files that need that level of encryption on encrypted flash drives.

If you are encrypting each file individually, you are still giving away the file names. Hence you use encrypted file containers. They benefit from block level deduplication. They can be large in size.

If you are encrypting each file individually, you are still giving away the file names. Hence you use encrypted file containers.

That doesn't follow - you can solve the problem without containers, e.g. by encrypting or hashing the filenames.

Containers protect the file sizes, but they lose the ability to selectively sync certain files to certain nodes.

Hashing doesn't solve the problem completely. You are still giving away information about file count, size & hierarchy.

Regarding selective sync : you'll have to organize the files logically into containers. I think even Dropbox doesn't let you sync files selectively, you have to have content separated into folders.

I'm not saying that one is better than other, just giving an example of where de-duping at block level is needed as @lugg asked.

Hashing doesn't solve the problem completely. You are still giving away information about file count, size & hierarchy.

Sure; I was just talking about the file names. You can avoid the hierarchy part by transmitting it separately from the files in a secure way, though.

Regarding selective sync : you'll have to organize the files logically into containers. I think even Dropbox doesn't let you sync files selectively, you have to have content separated into folders.

git-annex does. You can implement rules to decide what files should be synced, which should be dropped locally, etc.

For example, an MP3 player running it might have a rule such as

  filename(*.mp3) or filename(*.ogg)

and it'll only sync those files, regardless of their directories.

git-annex avoid all except (2), by simply encrypting each file using GnuPG, and applying an HMACSHA1 to the filenames.

http://git-annex.branchable.com/encryption/

SHA1?

It's configurable.

The default MAC algorithm to be applied on the filenames is HMACSHA1. A stronger one, for instance HMACSHA512, one can be chosen upon creation of the special remote with the option mac=HMACSHA512. The available MAC algorithms are HMACSHA1, HMACSHA224, HMACSHA256, HMACSHA384, and HMACSHA512.

Don't worry, git-annex uses SHA1 just as a checksum, not for cryptographic purposes.

no doubt it would be my first choice if it had proper windows support, but last time I checked that was still lacking

I tried to use git annex assistant: I can get files to be backed-up on a server, but when doing sync between 2 different machines it has never worked (I've been doing the xmpp-setup dance multiple times, and it's especially painful due to having 2factor auth enabled on google)

I'll have to try using plain git annex from the commandline to try to understand what is going wrong (I already used it to clean some repositories that the assistant was refusing to delete)

PS: I tried it with 2 machines in the same LAN (both linux... though one of the 2 distros had a slightly older version of the git-annex package)

Are there any issues you suspect Syncthing could suffer from?

Has anyone else taken a look at Ori (http://ori.scs.stanford.edu):

> Ori is a distributed file system built for offline operation and empowers the user with control over synchronization operations and conflict resolution. We provide history through light weight snapshots and allow users to verify the history has not been tampered with. Through the use of replication instances can be resilient and recover damaged data from other nodes.

It seems well thought out, and competitive with many of the other approaches mentioned here. It uses Merkle trees (as does Git) that encompasses the file system structure and full history.

Academic software usually for publication. After paper accepted, unless they could publish another paper using same software/concept/idea, this software will dis-continued.

"Replacement" is too strong a word here. P2P sync requires at least 2 peers to be online at once. For the simple case of syncing your work and home computers or sharing with coworkers that is not always a reliable assumption.

It's only a replacement for a centralized service like Dropbox if you have an always-connected peer (a de facto central server).

Not to be condescending, but what did you expect from a "roll your own" solution? Of course you have to host it yourself. This has been the case forever, OwnCloud being the biggest example right now.

That's fine, but call it what it is. I only objected to calling it a "replacement". There's a big functional difference between P2P and centralized architectures.

It seems as if you explained that it was useful as an always on central server to replace Dropbox, though.

Yes, absolutely. But that is a significantly greater requirement, each end user or workgroup running their own server. I'm not sure why I'm getting downvoted for pointing that out. It's not an apples-to-apples replacement.

This + a hosting provider would be more like an actual Dropbox replacement (e.g. owncloud.org links to a bunch of hosts).

That's not what I'm looking for. I do have a computer at home that is always on.

edit: Do people generally still turn off their home computers?

edit2: After thinking about it - what you seem to be describing is an alternative and not just a replacement. I want to replace the service that Dropbox offers with one of my own, I don't want to switch to an alternative Dropbox clone service.

Huge numbers of people only have a laptop at home which sleeps when they're at work. So the only sync service that's going to work for them is something with online storage as an intermediate step.

>>edit: Do people generally still turn off their home computers?

Yeah. My desktop computer eats up a lot of power even when idle. I do want the convenience of never turning it off, but I'd rather be environmentally friendly.

Syncthing is less purely peer-to-peer. It's easy enough set up a central server and have your devices only sync to that server because of the way connections are made.

I don't understand why you're being downvoted because you make a valid point. It has no iOS or Android client, no cloud storage (unless you buy your own), no versioning and sync conflict behaviour is very different. Ergo, it's not a replacement.

> no cloud storage (unless you buy your own)

That's kind of the whole point, isn't it? It's clearly meant to be a Dropbox replacement not for everybody, but for those who consider the lack of mandatory cloud storage a feature. So yeah, technically more a replacement for BitTorrent Sync than Dropbox, but not everybody knows about that.

Missing features can be added. Mobile clients by third party devs will surely crop up if/when this thing gets traction. The protocol is open, after all.

One of my favorite open source tools of this kind is unison. It works great. I set it up to go and I don't even notice it is there. It is quick, seems to have been around for a while and is packaged for most Linux distros. It has a GUI but you don't need it.

http://www.cis.upenn.edu/~bcpierce/unison/

Unison is a under-appreciated gem of file synchronization. The author also published several papers on the subject: http://www.cis.upenn.edu/~bcpierce/papers/index.shtml#File%2...

I can't really seem to find this information in the documentation.

Does this support delta/block-level sync for large files (e.g.: does mounting a 100 GB truecrypt container, modifying a file inside the container and unmounting it cause the entire 100 GB container to be uploaded)?

Does it utilize the native OS platform APIs for detecting file modification (e.g. inotify on linux) as opposed to scanning/polling large directories looking for modified date changes?

Protocol information is linked from the homepage: https://github.com/calmh/syncthing/blob/master/protocol/PROT...

> Does this support delta/block-level sync for large files

Yes.

> Does it utilize the native OS platform APIs for detecting file modification

It uses polling.

Edit: inotify issue is here: https://github.com/calmh/syncthing/issues/9 (the author seems to want to implement this feature once go has a decent cross-platform file watching library)

Thanks for the quick response. I just got through reading PROTOCOL.md and was about to update my post.

Aside: I was wondering this about Bittorrent Sync as well - would you happen to know?

EDIT: i realized i answered about btsync while question was about syncthing, sorry.

Regarding detecting file modifications: my friend had issues with his synology diskstation eating 100% cpu with btsync with around 50k files. With less files it was down to 20%. We never debugged it properly but it may be polling issues, but it could also have been out of ram-swapping issues (512MB ram was not enough).

Others have reported issues with many files:

https://forum.bittorrent.com/topic/23802-syncing-600gb-10-mi...

Is Synology ARM? syncthing seems to have a couple of different versions if he wants to try that: https://github.com/calmh/syncthing/releases

Fancy being interviewed for http://redecentralize.org/ ?

If so, email me francis@redecentralize.org! (I couldn't see an email or contact form for you on the syncthing site)

You can find an email at the authors GitHub profile[0].

https://github.com/calmh

Thanks!

I use http://owncloud.org and am quite happy. But also happy for more competition :-)

No matter how much I tried, I always had crappy experience using OwnCloud (-> very slow to recognize new files and sync them across other computers). I tried it maybe 4-5 times in total, last time just couple of weeks ago.

Does it work fast for you? Do you use desktop client or primarily web GUI?

I tried it too, but ended up replacing it with something else. It was slow as a pig, was unable to handle datasync with failures and timeouts on any reasonable amount of files (a 2000 file user) and constantly made my CPU fan hit max.

My server was a Core i7 with 12GBs of RAM, SSD backing and a more or less dedicated NAS. I tried with both SQLite and MySQL backing. MySQL was better, but still unacceptable. If that's not enough to run it for a single user, I can't imagine what it's like trying to deploy it for a full family, even less an organization.

I love the idea behind owncloud, and I admit touting it myself in my own "honeymoon phase", but in hindsight I have no problem admitting that the execution is just too poor.

Same experience. I tried it on DO droplets (1, 2, 4 and then 16GB of RAM to see if I'm nuts), computer I'm working on (16GB RAM) and from a RPi (that was for fun though). Every single time it was painfully slow.

It took it about 1h to sync a folder with about 100 files, under 30MB total across 4 computers. Not to mention that minor modifications appeared only after 15 minutes (small text files).

BTSync worked great, but I wouldn't trust it with something sensitive.

What did you end up using?

> What did you end up using?

Long story short, I went from an "all cloud" (Google + Dropbox) setup to a "all mine, no NSA" setup. Now I've decided that counter zealotry doesn't have that much for it, and I'll try to balance privacy with best of breed services, without trying to end up with all my eggs in one basket. Not doing that ensures I keep interop and portability in mind.

My end-result is calendar via Google (only solution I can get collaboration and invites working properly), files via Dropbox, mail with Fastmail (much better than Gmail) and contacts via Baikal on my own hosted server.

Fastmail is adding support for calendering and contacts via CalDAV and CardDAV, but I've yet to fully investigate that.

It's a moving target for me, but currently it's been stable for a few months, and I'm mostly happy.

Thanks for sharing!

Yeah, it was completely unusable for me (a few months ago). It ran a separate process for each file upload, taking a few seconds minimum and completely serial. I estimated something like a month and a half to do the initial sync of 100GB, with a mix of large and small files.

Git-Annex is a great existing option (made by joeyhess). https://git-annex.branchable.com/

This is a year or so old, but it still works for new customers:

http://www.rsync.net/products/git-annex-pricing.html

Can you explain to me why and how I would use git-annex for real time sync? From what I understand of it, it works well for backups and for managing media files, but I don't understand how it handles the use case of making two directories on two computers pretend to be one.

git-annex assistant, by the same author, builds automatic synchronization on top of git-annex: https://git-annex.branchable.com/assistant/

I once gave git-annex a serious try (a few weekends of all-day efforts), as a way of sharing my music library between personal machines. It seemed to fail pretty hard on the cross-platform part. I really wanted it to work, I just couldn't get it to do so.

It was great at first on a single Linux machine, but once I had to support Windows, things became much more complicated. Most of this I attribute to Windows - different filename character restrictions, no usable symlinks, git only kinda-sorta supported through cygwin. Setting up an SSH server under Windows was something I didn't want to do, and HTTP sync was pretty buggy (different hashing schemes between the two computers, for some reason).

If things have gotten better, or someone's written a good tutorial on having an actual decentralized repo shared between Windows and Linux machines, I'd be interested in giving it another go. Also might need to hack up a mobile app.

I haven't dug into Syncthing's protocol yet, but I also like the idea of being able to easily share a single sync'd folder or file with an outside person, but not having to point them at a full repository.

One thing I've never gotten about these "syncing" apps...

Lets say I install this software on my phone, my desktop, and my work computer. I have 100+ GB free on my work computer and my home desktop, but I only have 16GB on my phone. If I add 20GB worth of movies to my sync folder, its going to fill up my phone.

I believe Bazil[0] is aiming to solve this problem, among others.

[0] http://bazil.org/

Dropbox, at least, lets you select which directories get synced per device, i.e. so your desktop and HTPC can share movies but your phone never sees any of them.

I thought you could only sync individual files (mark them as favorites). Can you download a folder to your iPhone with dropbox?

Bitcasa (https://www.bitcasa.com/) also solves that problems by streaming your files on demand.

"Bitcasa’s proprietary block-level encryption..."

This phrase does not inspire confidence.

Dropbox has "selective sync", so individual PC's can choose to only sync a subset of folders from your account. Also, phones running Dropbox don't actually "sync" anything to the phone. You can choose to download individual files, or stream them straight from the Dropbox app, but it doesn't offer you any way to keep files on the phone up to date with whats in the "cloud".

I use dropsync on my phone. I have set up rules to sync screenshots to a different folder from photos, and periodically it will check a folder on dropbox and sync files from it to phone.

The two clients I have used (Dropbox and Bittorrent Sync) work on a selective basis - you can download a file on an one-off basis, or keep selected files synced, rather than the whole repository.

git-annex allows you to see all the files, without having the contents; then you can do "git-annex get <file>" to pull it, and when you're finished, you can do "git-annex drop <file>" to free up the space.

(It also allows you to do full automatic sync, or just automatic upload, etc. And different machines can have different configurations, e.g., RaspPi syncs everything with my VPS, but my Nexus tablet doesn't)

I just setup Sparkleshare (http://sparkleshare.org/) this weekend. I wanted something which wasn't Dropbox, and preferably open-source, and while Sparkleshare has a slightly clunky pair mechanism, it works beautifully.

Syncthing looks similar, and LAN sync'ing is a killer feature for those of us in offices with poor bandwidth.

Dropbox actually have a "Sync over LAN" checkbox. It just never seem to been implemented/enabled.

I actually just dropped Dropbox for btsync the other week and are very happy, it's iOS app even serves other clients with files over LAN.

Btsync however uses quite crappy crypto and is closed source etc etc.

Syncthing mentions they use TLS encryption, hopefully i can roll my own 8k keys.

I tried Sparkleshare last year aswell, but my experience was mixed. Also it relies on quite a different architecture (git) than a "versioning bittorrent protocol".

Looking forward to evaluating syncthing when time permits.

How come?

I have seen it working.

Thats curious. I never got it working between Windows, Linux and Mac clients. Sorry for drawing conclusions.

Isn't it always enabled by default?!

The last few times I've set up a new machine locally I've used Dropbox to pull over stuff. It usually moves at 40-50 MB/s (megabytes) which is certainly using my gigabit LAN.

According to [1], "LAN sync" only works when file has already been uploaded to the dropbox servers.

If this is accurate, then I guess this is the issue i have noticed.

1: https://superuser.com/questions/663600/set-dropbox-to-only-u...

UPDATE:

Confirmed by [2], that Dropbox LAN sync indeed only works once the file reached their servers.

"Well, when you add a file to your computer's Dropbox, the file is then synced with Dropbox servers. Dropbox will then initiate the syncing process as soon as it determines a change has been made to the file. All linked computers and shared folders will then download any new version of the file. With LAN syncing, Dropbox will look for the new file on your Local Area Network first, bypassing the need to download the file from Dropbox servers, thus speeding up the syncing process considerably."

2: https://www.dropbox.com/help/137/en

For anyone wondering, Dropbox changed this a while back. The requirement to sync to Dropbox first was added at that point.

That's good to know. Basically LAN sync is a misnomer, because it only works in the download direction.

Dropbox actually have a "Sync over LAN" checkbox. It just never seem to been implemented/enabled.

It is enabled by default and works beautifully.

I can confirm it's probably enabled by default, every network I see these days is spammed by Dropbox broadcast messages.

I never had much luck with Sparkleshare - in particular, I got burned by it a couple times when files I placed in the shared folder never got synchronized and I was not notified of this anywhere. Get to office, and oh crap my files weren't there..

I've been using Syncthing for some months now and it's working well for my use case of keeping laptop/desktop and home server files in sync. I had one occasion when I lost everything as I'd brought up syncthing on my server without the "sync" directory mounted. It happily deleted all files from my synced laptop as a result. That's now fixed, but it was a buttock clenching moment. Yay backups, and a third machine (desktop) which was suspended and thus out of sync, so still had my data.

Upstream developer is very friendly and attentive & seems happy to discuss new features and use cases.

It appears HN readers are terrible at self-organizing. Threads for articles like this should include by default a top level node for:

  "Here's the alternative I use"

  "Important question about the technology"

  "Pertinent question about the article"

How can we organize after threads have already been started?

Do I smell a niche for time travel startups? Disrupt the Big Time monopoly!

Well, I can't be sure, but I think colloborative technology for rearranging threads on-the-fly might be slightly less complicated than monopoly-busting or time-travel.

Good idea.

That looks like a promising project in a space which definitely needs improvement. Owncloud and Sparkleshare are okay, but they are far from perfect, and there is large room for improvement.

A better title would be "Syncthing, an open source Dropbox replacement written in Go".

But seriously, it seems promising.

How good is this at traversing firewalls? AFAIK, Dropbox will do some manner of HTTP trickery to allow syncing when behind overly-restrictive firewalls (so it just goes out the usually-provided web proxy), but the documentation here references forwarding ports + UPNP, so I'm guessing that doesn't apply here?

"Each node scans for changes every sixty seconds". There isn't any portable way to get notfications about file changes instead of polling? I know about jnotify in Java but well it's in java.

I don't think there is. Joey Hess ended up having to support inotify, kqueue, hfsevents and ReadDirectoryChangesW for his git-annex assistant.

You may have picked this up elsewhere in the conversation, but they are waiting on Go to have good cross platform support in the standard library for file change notification. This was originally going to be part of 1.3, but it's been pushed off to the next version of Go.

https://github.com/calmh/syncthing/issues/9

https://code.google.com/p/go/issues/detail?id=4068

https://docs.google.com/document/d/1-GQrFdDVrA57-ce0kbzSth4l...!

I have large media files, multiple TBs.

I deal with a constant stream of these and want to have a distributed network - connected via the inet - that allows me to sync the drives in all locations.

i.e. I would like to setup a server in my home office, one in my co-founder's home office and another in my editor's home office.

Whenever my editor runs off a few hundred GB of data to a specific folder or to their drive, I would love for that to be auto-synced to both my server and that of my co-founder.

Will Syncthing allow me to do this easily and will it be appropriate for an application like that?

It will do what you describe, but since you are handling large files you should know that Syncthing currently doesn't do dedupe, so if you rename or move a file, or have multiple copies of it, it will sync the whole file again. Changes within a file that don't move are handled efficiently.

If I am not mistaken, you can't do dedupes on media files, right?

It sounds like you could be using Tahoe-LAFS. There are also a number of other distributed file systems.

This looks very promising. But the documentation is not good. I have not managed to find a "1 minute" friendly overview of how it works. I mean, what data gets sent how where and why.

It is a very young project (pre v1.0) - so give it some time. You can read about the protocol here[0].

[0] https://github.com/calmh/syncthing/blob/master/protocol/PROT...

> But the documentation is not good.

The linked website is aimed at end-users.

Try the protocol docs: https://github.com/calmh/syncthing/blob/master/protocol/PROT...

On, I meant end-user friendly documentation. Like ushi said, it is probably too early for that. Thanks!

You could try the discourse site: http://discourse.syncthing.net/category/documentation

I did, it drowned me in other things. ;-)

someone above posted this: https://github.com/calmh/syncthing/blob/master/protocol/PROT...

Indeed, the documentation leaves much to be desired. I wanted to give it a try but I couldn't figure out what IP it listens on for the sync protocol. It logs a message to console about the web interface listen addr:port but nothing about sync. Finally found it under 'Settings', hidden up in the corner.

Here is the original from a year or so ago:

https://raymii.org/s/articles/Set_up_your_own_truly_secure_e...

"Then all current commercial services drop off, including SpiderOak, Bittorrent Sync and git-annex. This resulted in a clever combination of EncFS and dvcs-autosync. Because, in this day and age, you cannot trust any "cloud" provider with your unencrypted data."

The author was uninformed, though. git-annex has supported encryption for special remotes (including your own rsync.net) since the early beginning (the wiki page[1] has been there since 2011). And it's certainly open source.

[1] http://git-annex.branchable.com/encryption/

A few months ago I looked into using Syncthing for my decentralized browser, Syncnet[0]. At that time it did not seem ready for primetime. Does anyone have a good feel for its maturity as of late? For example, is there an API? Syncthing looks very promising and I would love to integrate Syncnet with it.

[0]: http://jack.minardi.org/software/syncnet-a-decentralized-web...

Syncnet looks great!

Alternative would be AeroFS (https://aerofs.com/) which also does P2P file sync.

Kind of a pig for resources though

I am very happy with the timemachine backup on my mac, but I have been looking for a good offsite backup solution so that I can trade storage with my family in case something happens to my house. This might finally be the right option. BT Sync seemed ok, but was more than I wanted my parents to try and setup.

Try CrashPlan:

http://www.code42.com/crashplan/

It's free for peer-to-peer backup and is designed to do exactly what you want.

Their cloud storage for backups is really cheap if you have a lot to back up and want them to handle your off-site too.

Thanks. That looks like a good fit, and the paid plans are probably less than I would spend on a set of external drives anyway. I will give it a try.

Let's hope this becomes what it is promising and reliefs me of dropbox and the likes... ;)

I'm curious, what are your complaints about Dropbox? Security, UI, UX, features?

Does this need to use the web gui? I tried setting this up on a digital ocean server and it spawns off a webserver on 8080 that I can't reach from my machine. Is there a "headless" mode for client-less servers? Thanks

   ssh -L8080:localhost:8080 mywebserver.com

You can edit ~/.config/syncthing/config.xml and restart.

This helped me ( though I'm sure the ssh tunnel would've worked, I couldn't get it running right) -- I changed the web address to my ip:port, and could then log in, add my host, and then change it back to headless or localhost. Thanks!

It's headless by default (but it starts a web browser going to localhost:8080 by default too).

If you don't have a web browser, it won't open it. If you don't want it to open a web browser, you can configure that.

I would also like to point out https://www.syncany.org/

Syncany can work with any backend (like AWS S3) and is encrypted.

It is more of a dropbox replacement while sycnthing is a btsync replacement.

That's great news, have been looking for a OSS sync app for quite some time.

What's wrong with BitTorrent Sync? The fact that it's a company?

It's closed-source; we can't fix things like the crappy crypto (shared key), and can't spot backdoors or see what it's actually doing without a lot of effort.

@damncabbage is right. I wanted to use it to integrate it into photo sharing application for mobile devices but I couldn't because it's closed source. This has a lot more promise.

This project's aims seem very similar to the earlier camilstore project, also written in go:

https://camlistore.org

Anyone know how it compares?

Does somebody fund projects like this? Or is it just that the people in charge of them understand something about UI and marketing? Seems like a nice trend, if so.

I've wondered this too. Perhaps they charge extra for more features or get money consulting?

Since I installed Bittorrent Sync, my need for such software stopped as it works really well and provides all I need from it.

I couldn't understand quite advantages and why would I replace BTSync, which BTW, works really well already and does all this nice things. Plus works on my Phone and Ipad and Nexus.

To clarify one thing, I have home server which obviously hosts BTSync repos with ample space. Ability to fine-grained share parts of it is invaluable.

So this may be as good a thread as any to ask for assistance:

My small business is looking for a combined file collaboration / file backup service that doesn't cost an excessive amount of money (we're a non-profit on a budget). Is there a good service for this? For example, Dropbox is mainly for sharing files, whereas Carbonite is mostly for backing up your computer. Is there a solution for both?

SpiderOak does both. Their interface is, IMO, not nearly as good as Dropbox (I've never used carbonite), but it does check both boxes (backup and sharing/sync).

Yeah but SpiderOak is $5/user/MONTH, so with 30 employees we're looking at $1800/year which is just crazy.

Basedrive might do it; it's an actual local server running their platform which you backup to/collaborate on, that you buy, rather than a service.

http://basedrive.me

We love to through around the words "alternative" and " replacement" it is neither until it is as easy to use/deploy as X for the average Joe. You know 90% of people out there who would't be able to out the words 'git', 'deploy' and 'heroku' in the right order as their eyes glaze over in confusion.

I'm looking for a replacement for Dropbox, but it seems that none support direct links, like in Dropbox's public folder (example: https://dl.dropboxusercontent.com/u/38901452/fox2.jpg ) Is there one?

The reason is because most of the options; owncloud, btsync, seafile and syncthing dosn't upload to a unknown company server. It only shares the files between your devices. Its a huge pluss when you think about security. The downside is that a device have to be online to share or update the files.

I use BTSync and to avoid the problem i just upload the files to my server and display them under a subdomain. http://pub.volpe.pw/pub/fox_gl.jpg

BtSync also offers a phone client, so to mimc the way Dropbox uploads pictures to dropbox was rather simple. I backed up a gallery on my phone and synced it to my server and symlinked it under the pub dir, as BTSync dosn't follow symlinks.

I don't mind uploading them under a subdomain or such, but I really like how easily I can get a public link from Dropbox. How hard was it to set up?

Once i had setup the subdomain it wasnt hard at all really. I only had a problem where nginx couldnt serve the files from my home dir (i preffer to have the syncing folder there), but it was solved quickly with google and SO.

I just set up a BTSync server with Pyd.io on top, which I can use to generate URL for my file. No manual copying.

I love that public directory feature! I ended up using seafile and just maintaining a copy of a public directory there using the command line client. Works like a charm, very easy to setup.

ownCloud supports sharing links, also direct links to files: https://owncloud.org

This looks very promising. And it's written in Go! The only major feature I think it's missing is file versioning.

I am curious, though: what do people use to get their files remotely? And what's the cheapest solution for hosting your own central server? Would a simple AWS instance work fine?

I server them to myself using _h5ai (stylized HTTP index) and nginx, displaying a portion of my files.

I'd also take a look at digitalocean for a VPS.

I really like the idea but one thing is stopping me: portability to iOS and Android devices, and mobile apps that work with Dropbox. Dropbox has a first-mover advantage.

This is mostly a problem for people like me who use both Android and iOS devices so alternatives need to support both platforms.

Yet another sync app is Pyd.io The Web UI is super neat, and you can choose between various backends for storage. Pyd.io offers its own sync app which I found to be horribly slow. I'd suggest to use Pyd.io as a frontend and BtSync/Seafile/Syncthing as a backend.

Unrelated, is one of the best open source project landing pages I've ever seen.

Have you checked Hive2Hive? Something similar but not yet there. https://github.com/Hive2Hive/Hive2Hive

In contrast to Syncthing, Hive2Hive is more a library, providing an API for any kind of program that requires file synchronization. Another major difference is that Hive2Hive is written in Java and not in Go.

It would be quiet interesting to point out the differences between Hive2Hive and Syncthing... H2H uses for example a DHT as underlying peer-to-peer technology.

Is there any mobile support?

I use Dropbox pretty frequently to share stuff between mobile devices and desktops. If Syncthing can't do that it isn't as useful.

As soon as someone makes a Dropbox alternative that has no servers, has selective-sync to my iPhone, and is compatible with Keepass to store the database, then I'm there. I'd happily pay for it, too.

I don't see a Solaris build on the download page but it's listed next to the download button on the home page, am I just missing something?

This looks incredibly interesting, and I would very much like to move from Dropbox to something open source. Thanks, will definitely play with it.

another alternative: http://www.filement.com/ I don't use it but friends do and are pretty happy with it. From their home page:

- Combine devices and cloud services into a single interface. - Transfer data between computers, smartphones, tablets and clouds. - Manage and use data directly on the device or cloud it is stored.

++ this effort. I'm looking forward to replacing my current BitTorrent Sync (btsync) setup with Syncthing.

I would like to see something like this that does not place trust on the server hosting the files.

seafile lets you have encrypted directories.

https://github.com/haiwen/seafile/issues/350

Is there a paper / doc explaining, how the synchronisation works in detail?

Can we get a "Bitcoin meets Dropbox meets Airbnb" already? Thks :)

I would of liked the name "synctank" better. :)

Cool, I always like it when things are open source!

So its basically pirating, but its saves across all platforms?

Not really

Reply on news.ycombinator.com