Hacker News Reader

Comment by kijin | Hacker News Reader

Seriously? Wuala is a service run by LaCie. LaCie is owned by Seagate, an American corporation. It doesn't matter where the servers are, because all the important decisions will be made in Cupertino, California.

http://www.lacie.com/us/company/news/news.htm?id=10722

Now, client-side encryption is a much more interesting aspect of their service, but is it worth the trouble if Wuala's clunky client takes 100 times longer than Dropbox to sync a file between two devices? Ditto for SpiderOak, JungleDisk, and every other backup/sync solution that I've used so far that boasts client-side encryption. And it wasn't due to Dropbox's deduplication, either. Some of them just talked lazily with the server for several minutes before they even started to upload/download any files.

View on HN

> is it worth the trouble if Wuala's clunky client takes 100 times longer than Dropbox to sync a file between two devices

Is this some general FUD about encryption being slow, or an actual measurement?

Maybe all client-side encrypted solutions happen to be crappily implemented, but I don't think that client-side encryption would necessarily lead to something noticeably slower.

I have absolutely no intention to spread FUD about client-side encryption, and in fact I don't think Dropbox's speed advantage has anything to do with (lack of) encryption. Encryption shouldn't have any noticeable impact on sync speed, since modern CPUs can encrypt a file a hundred times over in the time it takes to upload/download it. I agree with you and suspect that it has more to do with crappy architecture.

Anecdote: SpiderOak takes 5 minutes synchronizing metadata with the server if I start it on a laptop that I only use occasionally. That's how long it takes for the client to find out what files I changed in my other machines in the meantime, including all the files that have nothing to do with this particular machine. Nothing gets synched during this time, and the client sometimes consumes a whole CPU core for extended periods. It's very annoying when I just want to see a few files that I know I changed in another machine. I understand that the protocol needs to be different if all the encryption is done by the client, but this is just ridiculous. Wuala is no better in my experience, although YMMV depending on your use case and filesystem structure. (I have over 50GB of data in my SpiderOak account, made up of a quarter million files. At one point I had over 100GB stored in there, and it was a PITA to sync any files at all.)

As an interesting comparison, Microsoft SkyDrive performs just as terribly as SpiderOak does when you give it a few hundred folders with ~100K files to sync, despite the fact that it does no client-side encryption. It just sits idle for half an hour or more, scarcely utilizing any network resources. I have no idea what it's doing. Maybe it does sleep(random()); between each upload. In any case, encryption isn't making it slow. It's just bad app design, bad protocol design, and perhaps overloaded servers.

Dropbox, on the other hand, immediately begins to upload/download my files, and once it's done scanning any large folders, its speed is only limited by the bandwidth of my Internet connection. When I need to do a presentation, I can just work on the .pptx file in my Dropbox on my desktop PC, take my laptop to the conference room, and expect the .pptx to be there as soon as I wake my laptop. That's the meaning of convenience. By the same metric, none of Dropbox's commercial competitors (perhaps with the exception of Google Drive, which I don't use) are even close to convenient.

SpiderOak cofounder here.

It's not really about the encryption directly; it's about the way the database design has to change to support zero knowledge (the server can no longer do the database work.) Things like garbage collection have to happen client side. Described in detail here: https://spideroak.com/blog/20091026143000-why-and-how-spider...

FYI, when SpiderOak first starts running, like any backup software, it has to scan your filesystem to see what may have changed while it was shutdown, and that can take a lot of IO and CPU if you have many folders and/or many small files (hundreds of thousands). (SpiderOak works with any folders you like, including externa/network drives, not just one folder like Dropbox.)

In a profile, basically none of this CPU time is spent in the crypto module, just mostly for unserializing the per-folder journal to compare state. SpiderOak does some optimizations with directory hashing to avoid having to open the journal to scan for specific changes in unchanged folders, but there are some situations where that optimization is defeated (such as if you remount a file system with changed gids, inode IDs, timestamps, etc.)

Anyway, once it's running, it uses the operating system facilities on Linux/Mac/Windows to notice changes automatically so rescanning the whole backup selection after startup is not usually necessary.

In any case, thanks for your interest in SpiderOak. :)

Hello Alan,

I have a quick question for you which might be of interest to others as well:

Why isn't SpiderOak open source yet?

I've read your FAQ answer[1] on this, however it doesn't really give a concrete explanation, besides "soon" and "licencing concerns" which is definitely disappointing.

I currently use my own, EncFS based sync solution, however I'd love to be able to use SpiderOak, or a third party open source application which supported syncing with SpiderOak (if you'd ever consider exposing an API/Protocol which allowed such.)

This is the only reason I'm not using your service currently, however I love the concept and hope you'll take it into consideration. It's likely I'm not the only one with similar concerns.

[1]: https://spideroak.com/faq/questions/35/why_isnt_spideroak_op...

My hard drives are reasonably fast, so even with a quarter million files it doesn't take long to scan the filesystem. I have other backup software that scan the exact same folders and finish in less than 30 seconds. So I suspect that it's your journaling/unserializing system that takes up the bulk of the startup time.

Fortunately, I only need to start up SpiderOak once in a while, because nowadays I put my computers into hibernation instead of shutting them down and starting them up again. I also noticed that the more often I use SpiderOak, the less time it takes to start up, presumably because there's less delta to process. Also, as you said, once SpiderOak is up and running, it's relatively fast. Rest assured that the performance issues, although annoying, have not dissuaded me from renewing my SpiderOak Plus (100GB) account once again last month.

I also like the fact that SpiderOak's "Queue" and "Log" screens tell me exactly what it's doing at any given moment. Waiting becomes a lot more tolerable when I know what I'm waiting for. I hate backup tools that assume the user is too dumb to understand what's going on behind the scenes. Even Wuala's Upload/Download queue never seems to work properly.

Client-side encryption can be very fast when done right. For your use case a filesystem stored in the cloud might be a better solution, that way only the data you are using is downloaded to each machine and the writes are done immediately. Give ObjectiveFS[1] a try, the free preview is available for a few more days, and see how fast client-side encryption can be.

[1] https://objectivefs.com

Interesting, but an all-cloud solution would seem to be a better fit for servers with fat pipes (especially EC2) than for individuals with crappy residential connections. I have fiber at home, though, so I might try it out sometime.

I wouldn't go into technicalities because I can't - mostly because it's not my area of expertise and I've never looked into it deeply or maybe not even from the surface other than a user's perspective. The end user.

Wuala is seriously slow and clunky piece of software. Period.

"...is it worth the trouble if Wuala's clunky client takes 100 times longer than Dropbox to sync a file between two devices?"

I have no idea what the performance difference is (and clearly , neither do you), but the answer to your question is "yes". For some people, at least. It's a must-have feature for some people, not a nice-to-have-as-long-as-it-doesn't-inconvenience-me-too-much feature. Some people have data that they'd like to keep synced across systems that they simply can't keep in unencrypted form on a third-party server, no matter how much trust they have in the company that owns the service.

Of course, there are a bunch of different ways to keep your data secure and still have it be available to you on different systems, and non-open-sourced cloud storage with client-side encryption is still an imperfect solution, but it is a solution that is perfectly reasonable for many people.

BoxCryptor[1] has been my choice. I've been now using it on daily basis together with Dropbox and I'm satisfied. Boxcryptor works on the client side and stores the encrypted files on Dropbox folder as ordinary files. It is also possible to encrypt filenames. Boxcryptor comes from Germany.

The only drawback I have encountered is that Dropbox limits the path names to 255 characters. With encrypted file/folder names this can become an issue especially with some Java projects.

I think this really gives me the best of both worlds. Security for the files I think need it and the Dropbox infrastructure for syncing files.

[1] https://www.boxcryptor.com/

Yeah, tried that too. Also tried CloudFogger [1], which has a similar feature.

When it comes to performance, a layer of encryption on top of Dropbox is much better than any competing solution that has client-side encryption built into it, because you're still using Dropbox's awesome infrastructure and lightening-fast sync protocol.

But I ran into the same problem as you did. The 255-char limit sucks, and it's completely unnecessary because even Windows/NTFS allows you to use longer paths (up to 32KB) if you know which APIs to use. I suppose fixing this is not a priority for Dropbox because you normally don't run into it unless you use encryption, and Dropbox has no reason to encourage the use of encryption among their users.

[1] http://www.cloudfogger.com/

wuala isn't just doing client-side encryption, it is also doing reed-solomon coding.

Why not just store a TrueCrypt volume(s) in Dropbox?

Because TrueCrypt is not exactly "free and open source" software either.

http://en.wikipedia.org/wiki/Truecrypt#Reasonable_paranoia

"""It is challenging to create binaries from source code that match the official binaries bit-by-bit for purposes of verifying their integrity due to compiler options

...

There has been no known comprehensive review of the source code by a qualified cryptographer.[46][44] Thorough security code review and testing is hard, tedious, and painstaking work, and very few people have the skills to do it. There was, however, a functional evaluation of the deniability of hidden volumes in an earlier version of TrueCrypt by Schneier et al. that found security leaks.[47]"""

This is assuming the Government doesn't force Dropbox to install a keylogger on their client to get users TrueCrypt passphrases and possibly keyfiles.

If Dropbox bundled a keylogger with the Dropbox client, I'm pretty sure it would eventually attract some attention...

Who's to say they can't be convinced to push a one-off update to your account/machine. Perhaps a transitory one, to help cover their tracks?

The underlying problem is that you're trusting closed client software from a third-party. Once that is running on your system, arguably it's game over as far as what Dropbox or its like can and can't do to you.

(If you trust them, ok. But when documents state that "they're next", many of us start to wonder. And again, this isn't just access to a server. Through their client software, this is access to your machine, that you've already granted.

Admittedly, I'm describing the typical Dropbox user scenario and not where the client is manually applying -- or not -- each update, and somehow supposedly gaining insight into the capabilities of each update/version before doing so. And I'm also ignoring various forms of restricting the access of the Dropbox client, although if they really want you, they and parties using them would have extensive resources to apply to breaking out of such constraints.)

https://grepular.com/Protecting_Your_GNU_Linux_System_from_D...

Very interesting and useful, not just for securing DropBox either.

FYI this article suggests to use encfs for an encrypted folder in your dropbox, this comment https://news.ycombinator.com/item?id=5855465 suggests that this might not be a wise choice, so you might want to (re)consider that part.

Thank you. I will read through this.

Let me take the opportunity to also thank you for writing up your email best practices. I hope to implement them or something similar for myself.

One of the challenges for me in all this is keeping up including with the implementation details. It feels akin to swimming upstream, with the current outpacing me.

Yeah, 4 years later when a whistle blower said something.

Storing an EncFS on Dropbox is another alternative, that doesn't depend on Dropbox' ability to recognize small changes in a huge single file.

It'd be better to use EncFS in this case, as it does per-file encryption. With TruCrypt, the whole volume file would need updating once you unmount it, if I understand correctly.

I like Boxcryptor since it encrypts each file individually rather than the whole volume. However, the file names are exposed.

encfs. Encrypts each file individually and file name are encrypted.

encfs sometimes generates very long path names, which runs against the 255-char limit that Dropbox imposes if you have any Windows computers linked to your account. Last time I tried it, the filenames were silently corrupted and much data was lost.

That's what I do, I have a small (100 MB) small volume sync in my DB, works like a charm :)

Because, as far as I know (I haven't used TrueCrypt too much), it will see the volume as a file and then you'll be syncing a huge, x GB file every time you make the smallest change within it.

That's not true. Dropbox uses librsync so small changes of big files yields small diffs, thus TrueCrypt volumes work just fine with Dropbox.

Speaking out of ignorance here: But aren't TrueCrypt volumes completely changed when you add/remove something? I'm thinking similarly to a hash: The slightest modification changes it completely.

But aren't TrueCrypt volumes completely changed when you add/remove something?

No, the crypto in TrueCrypt doesn't amplify the number of changed blocks. But it does disperse the changed blocks throughout the encrypted image, and this does result in higher bandwidth used during the sync (at least with DropBox). Here's an analysis I did a few years ago:

https://news.ycombinator.com/item?id=2546626

If that were the case, then terabyte-scale volumes would be prohibitively useless - and I know plenty of people with media volumes in TC.

That's not how it works - it would be inefficient to the point of uselessness to rewrite the entire file constantly, and my 100GB TrueCrypt volume is fast to save to.

Reply on news.ycombinator.com