Building your own secure storage space that mirrors Dropbox's functionality | Hacker News Reader

Building your own secure storage space that mirrors Dropbox's functionality

defcon.org

137 pointsbuckwild55 comments

TLDR: The code is at https://github.com/philcryer/lipsync

What I'd like to see is an easy-to-set up way to run my own version of Dropbox on a small server in my own home. It needs to have a web interface, not just rsync, because a lot of people (like me) don't have access to anything remotely like rsync on our work machines.

> don't have access to anything remotely like rsync

What...? Why not?

I am an employee in a non-technical position working in a locked-down environment with (appropriately) paranoid IT staff. The only applications on my work computer that can talk to the Internet are a mail client and a web browser. There are millions of me.

So you're saying that you expect an application in your browser to reach down and monitor, upload, and download gigabytes of data to your filesystem?

Your appropriately paranoid IT staff would collapse in convulsions of terror if this were possible. Fortunately, it's not. The reason Dropbox (and rsync, and lipsync) are native apps isn't because the developers are unaware that there are people in locked down environment and need a browser-based tool, it's because the apps need to be native.

I think he's saying that it'd be nice to have a web interface for getting/putting one or two files on machines that can't have the client installed. Dropbox has this.

At which point, his (appropriately) paranoid IT staff should give him full-time access to his files by way of a visit to HR and a final paycheck.

IT has the machines locked down for a reason.

Installing rsync and uploading a file to Dropbox's web interface are significantly different actions. The install restrictions might not be to prevent offsite transfer of files - it might just be to prevent people from installing AIM and trojan horses.

http://www.pogoplug.com/mobile/ It doesn't have folder sync. But, a web interface.

Is this just a way to trigger rsync every time a change happens to a given folder?

Looks like this command powers it:

  rsync -rav --stats --log-file=/home/$USER_NAME/.lipsyncd/lipsyncd.log -e "ssh -l $USER_NAME -p $SSH_PORT" --delete $REMOTE_HOST:$LOCAL_DIR $REMOTE_DIR

Couldn't you just toss that in a Guardfile and get the same effect?

  https://github.com/guard/guard/

Or am I oversimplifying it?

  while inotifywait -e modify -e create -e delete -e move -r $1; do
          rsync -vrae 'ssh -p 2299' --delete $1 noentrar@noentrar.net:files/$2
  done

Here you have → https://forrst.com/posts/Two_10_seconds_scripts_to_keep_your...

I made this long time ago, the same concept just with inotify, to don't abuse the my small bandwith

Every time I read about the X-th "clone with Dropbox's functionality" I wonder where it will store its files. How many of the potential users have access to online storage they really control themselves? Instead of having the files on Dropbox they end up on AWS or the like. Thats because they try to copy Dropbox down to the flaw, that it doesn't encrypt files on the serverside.

For me, any solution would have to include the capability to upload only encrypted files to the server. I know of the duplicity project, which does that for simple, manually triggered backups, but which once caused me headaches to get the data back from partially corrupted files.

> Every time I read about the X-th "clone with Dropbox's functionality" I wonder where it will store its files.

I have a different reaction: I keep wondering how long Dropbox can lead in a market with such low barriers to entry.

I presume that the de-duplication saves them enough bandwidth and storage to allow a much lower price point (as well as the free plans)

A secure system can't do either of those things, as far as I know. Assuming the data is encrypted per-user, on their local devices, you can't easily compress it (because the files are now very high entropy), and you can't scan for preexisting files, because you'd be giving an attacker an oracle about which files (or rather, which file hashes) exist already. Even then, you could probably use some traffic analysis to deduce something about the data.

Having data securely stored also removes some of the dropbox unique(ish?) features, like being able to share content with people via the web.

The pretty much diametrical opposite of dropbox is tarsnap. Everything else sits somewhere in the middle.

The real selling point of Dropbox, for me, is the ability to just drag and drop files to a folder in Windows and have them be shared.

I can also create a subfolder--then I can hit the Dropbox website and type in an email address, and voila, my files in that folder are shared with that person.

I agree with Dropbox having security issues and would like to see a replacement. I hope this project implements the drag-and-drop functionality cross-platform. Then I could really recommend it to non-technical users (or some enterprising soul on HN could use it to build a more secure Dropbox clone for non-technical end users.)

Correct me if I'm wrong, but doesn't FUSE allow this as well?

While it's cool to see people replacing Dropbox with OS alternatives, this only seems to capture a fragment of Dropbox's functionality.

How, for instance, does it deal with conflicts where files have been changed on two machines independently prior to sync? Dropbox is (somewhat) clever about it, and renames the conflicted versions - and IIRC you can resolve the conflicts manually. At least both files are available on both machines - not sure how this situation is dealt with by lipsync.

Dropbox push-updates are immediate. Lipsync relies on a cron job to kick off a receiving file sync.

Dropbox will sync directly between clients on a LAN - great for when teams are sometimes working in an office, and sometimes at home.

Dropbox maintains revision history - does this?

Dropbox has a web interface, mobile apps, etc...

I realize that Rome wasn't built in a day, but until at least the first two points are addressed, this isn't much of a replacement.

THANK THE GODS

Now let me tell you what is REALLY needed :)

Teamwork.

I work on a virtual team - we all work independently from our homes on client projects.

It would be great to create the following for real enterprise version of this:

Create an master account. /lipsync/

Then have project/team folders under /lipsync

/lipsync/1

/lipsync/2

/lipsync/3

/lipsync/4

Then have users A, B, C, D

Each user can have subscriptions to the project/team folders.

But they also have their root /lipsync/ account

Thus I can have my lipsync account - and in that I can be on team 1 and 4 with user D

Users B and C have subscriptions to 2 and 3 and the master server is all four.

This is how I have been wanting dropbox to be able to accomodate a virtual company.

You can sort of accomplish this with "shared folders" in drop-box - but because I cant have a server of my own I have to pay ....

EDIT: Wow - they jsut came out with Teams (or I have not seen this before... but it is still $13/month per user - which isnt that bad - but you still dont have a local server.)

Anyway, good work - I will use lipsync.

I dont really know what the big deal is. If you have a file that is top secret dont have it on the internet whether it be your email, in dropbox or even on your own aws/hosted server.

Keep it on a flash drive and have it stapled to your arm if you want to transport it. For images that you want to share, or files that you kinda dont give a rats about if they were to get compromised or disappear, place them in places on the internet that match their confidentiality requirements (email - for secure, dropbox - for kinda secure or your own private server - for very secure)

I think arguing whether a new solution is required because dropbox can't do the job or whether dropbox can keep your files safe is a moot point. If your files are on the internet they are never 100% safe. Just keep your most private files on an external hard disk and have that disk detached from your computer and the only risk you run is if someone robs your house.

I personally think dropbox is great for what i use it for, i put images that i want to share, i transfer files to friends overseas and we all happily share our stuff easily and seamlessly and if i wanna access it i login through the browser or install a new client. WIN. No other service at the moment does it this easily for me. If any of my files where compromised, well whatever, a few holiday pictures or some itineraries.

The legal responsibility is put on Dropbox to keep your stuff safe, but I think the "common sense" responsibility is put on the user to decide whether s/he thinks dropbox is the place for a specific file.

I don't get your argument. It seems that you're saying that if something can't be made 100% secure, then there is no point trying to make it more secure than it already is.

Its not an argument, its a state of mind in keeping your information that you value safe.

Im saying, as things get more secure you can start to move your information there (as i said with email, dropbox or your own servers). Use the online medium that matches your documents safety requirements.

If your a security guru then out of all those your own servers that you harden yourself would probably be the most secure. For non security guru's maybe email or dropbox do a better job.

No one ever said dont make it more secure, but as the sophistication of the security methods goes up over time so will the sophistication of the crackers/hackers. This means your judgement not matter how much better security gets will always play a apart.

Im stating that you shouldnt rely on others for security. Whether dropbox, or a clone. Stating that dropbox as a vendor cant be trusted isnt a valid as it will apply to all the clones. Bottom line you should take responsibility of your own files. Any new system that tries to be more secure will "most probably" still have security issues with it anyway.

All security is compromise–Bruce Schneier.

It's not even close to dropbox.

It's supposed to have more features like detecting when a file was also changed locally and remote and create a new file from the remote file.

It seems very fragile and I wouldn't trust it to backup my data somewhere.

I used rsync to sync files I was working on to a server >10 years ago. Only difference is that it can detect changes to file locally then update the server. To retrieve updates from the server, it relies on cron to poll the server, rather than implement some kind of push notifications. That said, it's always nice to ensure people are aware of what alternatives exist.

This approach seems to overlook the bigger picture which is that Dropbox makes sure it doesn't lose your data. Aside from that the web interface and dealing with conflicts is an important issue. Just imitating the syncing is not going to cut it, and if you're not outsourcing the storage it rather defeats the purpose of not having to worry about the data anymore.

People should check out Strongspace (https://www.strongspace.com/)

Combined with their OS X app it makes a reasonably compelling alternative for mac users.

http://blog.strongspace.com/announcing-strongspace-app-for-t...

The site looks exctly like http://jekyllrb.com/ . Are the project related ?

Or, use AeroFS and get something that actually works like Dropbox, and not what somebody who has only read a description of what Dropbox does thinks it does.

(also, I lol'ed at "vetted by the community as being 'a good idea'" and backing that put with a screenshot of a Reddit post (of all places) where a couple of people say 'yeah that's the ticket')

Sure, use yet another proprietary thing to share your files.

What could possibly go wrong?

Sure, rely on a hacked together contraption of scripts that have no idea about conflict resolution and requires manual editing and configuration for each machine you want to install it on, and doesn't offer near the same amount of features than other options.

What could possibly go wrong?

You would've known I don't like this scripted thingy either if you had read my other comment.

Also, personal work, confidential data and other stuff should never go anywhere near something like dropbox, aerofs and anything like this.

Any spare invites to AeroFS, anyone? I would like to give it a try.

co-incidentally, I wrote a dropbox replacement this week using AppEngine. It is just a webdav server with a simple web interface.

I will release the code at some point this weekend

I'd like to see compression, anonymity and encryption built into the storage.

And de-dupe. This is not a contrary aim from the compression, anonymity and encryption bit.

>And de-dupe. This is not a contrary aim from the compression, anonymity and encryption bit.

From a technical point, they largely are contrary aims.

Tarsnap gives deduplication, compression, and encryption. Some technical background on how it does this here: http://www.tarsnap.com/efficiency.html

There is a detailed breakdown provided as well. Sample output:

                                         Total size  Compressed size
  All archives                               697 MB           323 MB
    (unique data)                            215 MB           100 MB
  This archive                               148 MB            67 MB
  New data                                    17 MB           5.2 MB

Edit: Don't want to misrepresent Tarsnap's granularity. The print-stats option gives size in bytes. Above output was generated with the --humanize-numbers option.

More sample output, from a heftier machine, in case anyone is curious:

                                         Total size  Compressed size
  All archives                        3321059166180    1113881916696
    (unique data)                      116996664943      27182261608
  This archive                          13831036740       4855451330
  New data                                336950399         62343192

(Yes, that's 3.3 TB deduplicated and compressed down to 27 GB.)

Just been looking at Tarsnap. It looks good but I'm curious what happens if the author/creator/owner/maintainer (singular) gets run over by a bus?

I do my best to avoid busses. (I've always been more of a fan of switched point-to-point connections than busses, to tell the truth.)

In all seriousness, if I get hit by a bus Tarsnap probably won't live on without me -- but it runs itself smoothly enough that there should be plenty of time for people to download their data.

Thanks for your reply. I'd been looking at the FAQ and it seemed like there wasn't much redundancy built in - not wishing ill of you at all.

You're quite right, and it's a reasonable thing to ask. I don't mind.

Can I automatically sync files using it? I work in multiple places and if I have to manually run a sync operation before I leave, I'm certain to forget or be unable to at some occasion, and then I'm screwed.

Tarsnap doesn't do sync. Doing sync while maintaining security is a much harder problem: Tarsnap's design requires each system to know what's on the server before uploading or deleting blocks, and that breaks if you have multiple systems accessing the same storage space.

I assume this is only deduplication within a single account? Because I don't see anything there on how one would provide dedupe across crypto domains (which, to the best of my knowledge, should be theoretically impossible if the crypto is done right).

Correct.

The server need not be party to the key that is used to exchange the filenames themselves between the clients.

The content can be compressed in blocks before upload; dynamic, anchored block sizes would increase de-dup.

The compressed blocks can be individually encrypted with a key derived from the filename and block.

There's still plenty of opportunity for whole-file de-dupe there.

You can deduplicate within accounts; cross-account deduplication is indeed a problem.

Try one in Ruby? https://github.com/evantahler/synchzor

Why would I want a login page that lets anyone login to any account without a password?

Thanks.

I will propose do something like this on my work.

Even if it's not practical, it will be fun to implement it.

the site is at http://lipsync.it/

sparkleshare is much closer to dropbox. Mac and Linux only for now, git back-end.

Good luck getting it to work for mobile apps built to Dropbox's API, which is my main use of dropbox.

I do this myself on my OpenBSD server by simply using an SFTP-only account that has its home directory set to a virtual filesystem that uses encryption through the svnd(4) node. If I want to move the entire encrypted filesystem to another server or so I just copy the file its contained in, and mount it there. Setting this up took about 5 minutes.

If I'm understanding you correctly (and my apologies if I'm not), this isn't really quite the same thing - it sounds like a remote, encrypted filesystem.

Dropbox and lipsync are file-syncing mechanisms, not filesystems. So, if you're offline, you can continue to work - your changes will be pushed to the server next time you're online.

Reply on news.ycombinator.com