Hacker News Reader

Comment by PrimalDual | Hacker News Reader

I love dropbox but one of their biggest problems is their refusal to support more than 300,000 files in their sync clients. I have considered moving to google but I need local copies since I often don’t have internet connection but I still need my files. Either way, they have a useful product even if they won’t dominate the enterprise or consumer market.

View on HN

Oh wow I didn't know about that limitation. Doing `find . -type f | wc -l` I have ~270000 so I am about to hit it. It would be great if they can implement a .dropboxignore so I can get it to stop syncing node_modules folders.

I really wish I could agree with the others saying "It just works" for dropbox sync. I have ran into many other issues around filename casing, file permissions, symlinks, etc in the past.

This is a great idea. The only current ability I've found to hide files is by entire folders and per machine. Its also less of an ignore as a don't download here. Its really limiting and creates weird folder abstractions I don't need just because I want something to be able to be excluded on a specific computer.

shouldn't you be putting your code repositories in something like c:/www folder, and using purely git to track it?

There's lots of 3rd party solutions for .dropboxignore personally I've used many of them and am not a fan of it (perl scripts, 3rd party apps, etc). I wish dropbox would natively implement .dropboxignore, but its probably never going to happen since only a small % of users voice their concerns for it. Average dropbox user probably doesn't care or doesn't know what a .dropboxignore is.

I do use git too. But it's nice to be able to jump between multiple computers without needing to push an incomplete commit to remote (when I get tired from sitting in front of desktop I switch to laptop to code in bed).

Also dropbox's 30 day version history did save me a couple times when I accidentally deleted something before committing. And not having to worry about losing the day's work even if your computer fails is good for peace of mind.

That said this hits a lot of edge cases with dropbox syncing since it's not built for this kind of workflow. Every time you do npm install it will cause thousands of files to sync. I have had the .git folder get corrupted when I start using it before it has fully synced. Or git would handle syncing internal symlinks/permissions/casing differently and I would need to rely on git to restore.

The only problem with using dropbox in this case, is if you accidentally dump node_modules inside of there, it really makes your file history revisions in dropbox messy.

That and if you have to sync with another PC at home makes your computer sluggish for like half an hour while its syncing

My solution to you is to have your code repos in another storage solution (not dropbox). Unfortunately you would have to pay for that too

Or, you could use a built in IDE that has a autosave local revision history. I know most of the brainstorm products have this feature, such as PHPstorm. This would prevent accidentally deleting something before committing

For incomplete commits / switching computers, I can't really think of any workarounds for this unfortunately. Dropbox is very convenient here. The only solution is to use something like teamViewer and remote into your desktop PC from your laptop

node_modules will muck up jall the big filesync applocations. It's not cloud based, but Resilio Sync supports an IgnoreList file that I use for all my dev folders (whcih are also typically backed up with Git).

Syncthing might be an option for you. Someone is rocking 17 million files in a sync'd folder (https://data.syncthing.net/).

Where do you see that?

In the usage metrics section

I really want for this to work, but it is so not comparable to Dropbox; Syncthing takes forever to synchronize between machines (all on 1 Gbps LAN) unlike Dropbox, which can saturate my link. It is the best of the Dropbox alternatives I've tried though (having tried almost all) and if it performed better I'd be able to forgive the lack of an iOS client. I'm still evaluating it though (3 hosts, 329GB in 1.3M files).

I have about 750,000 files and 38,000 folders (I compile a lot of stuff). Are you putting everything in just one folder or splitting the folders up? I'm on really old hardware with mechanical drives and 100Mbps LAN so my observations are going to be different than yours.

I started out with just one folder syncing everything in '/home/kindone', that takes 30 minutes just to scan every time I started syncthing. Syncing over the LAN would not saturate my network unless it was a large file. That took about five hours to get everything synced when you have 700,000+ 4 - 400KiB c/c++ files. (It seems to send about 4 - 5 files at a time, someone correct me if I'm wrong.)

I removed '/home/kindone' and creating separate folders in syncthing for things like '/home/kindone/build/linux-4.14', /home/kindone/build/wine', and '/home/kindone/Dropbox'. Created about 65 different folder for various things, added them to my other machine on the LAN (that was annoying). Once they started syncing my 100Mbps LAN was saturated. Starting syncthing now takes 5 minutes to scan everything.

With the risk of getting down-voted for being too off-topic, here is my setup and observations:

- the three host range from pretty fast (Coffeelake @ 5 GHz, 32 GiB, Samsung 950 Pro) to just ok fast (some i5 @ 3 GHz, 8 GiB, files from a fast NAS on 10 Gibps)

- I two syncthing folders, with a 90/10 split. Some 155k directories between them. The smaller one is scanned every 6 seconds, the other every hour.

- I have disabled relaying etc. so traffic is only p2p and LAN-only, static IPs.

Observations: Scanning is slow, which is expected. However, even when the scanning is done and Out of Date files are found, nothing happens. Zero bytes are transfered. The syncthing process is running, but at a fairly low rate. Sometimes pausing and resuming seems to start things going, but not always. Even when data is traveling, it's pretty slow, at most 70 MB/s, and usually way under this rate.

Another frustration with Syncthing is that there are entirely too many knobs to frob. I like how setting up Dropbox is stupidly simple and to set up a new host is 1) install Dropbox and 2) add my userid/password, done.

As I have honestly looked for a Dropbox alternative for what must be a decade (primarily because Dropbox, Inc. can read my files) it's truly astonishing to me that nothing else works as well. Of course I have considered writing my own clone, but there are other projects to work on and I don't discount how difficult the problem really is.

EDIT: rewording

EDIT2: FWIW, another "solution" I'm evaluating is to endorse Dropbox but run gocryptfs on top for sensitive files.

Never heard of gocryptfs, tried it for an hour or two and its rather interesting.

I guess you could ask on the forums, https://forum.syncthing.net about your issues and see what they say.

Is that info up-to-date? Their new client features should have made that limit meaninglessness a year or so ago!?

Reply on news.ycombinator.com