Ask HN: How do you manage large files across multiple laptops/desktops? | Hacker News Reader

Ask HN: How do you manage large files across multiple laptops/desktops?

19 pointskaiku19 comments

My wife is a graphic designer and works on both a Mac desktop and laptop. The assets she generates for a single project can reach into gigabyte territory – multiple versions of a set of print layouts, RAW photos, etc. One problem she has is how to reliably sync large files between computers. She uses Dropbox currently, but sync over network can takes a long time or sometimes just doesn't finish depending on the file. And when she runs into space limits on one computer, she'll need to adjust Dropbox settings to selectively sync items.

I'm curious what people do in this case. I think she needs:

1) A large amount of storage, expandable as time goes on 2) Files accessible from multiple computers 3) Ability to access files securely when working from a coffee shop, traveling, etc.

View on HN

Have you looked into Nextcloud?

If you primarily do it at home, the speeds of Nextxloud are typically limited by the network bandwidth, and the storage is limited by what server you have. I use it and manage 5 TB of data on it without problems.

I am using Nextcloud for precisely thw usecase OP outlined and I am quite pleased with it I must say.

NextCloud is decent but not technically polished enough. Web UI is not performant, not too user friendly and mobile apps aren't really that good. I do have 4 instances running but I'll welcome alternatives if I find one.

Heh, I would say the Web UI works extremely well and is very friendly, and I really like the Android App. What issues do you have with it?

Here are my gripes.

- Pages load pretty slow over mobile networks. Every app page requires like 5 seconds to load, as it loads the page in 2 steps, first the page and then then the content. Not talking about navigating folders in Files but apps like Calendar and News are especially slow.

- Bookmarks app silently stopped working few weeks ago when I thought I kept adding Bookmarks through Android mobile app, so I lost 2 weeks worth of them. Not sure if the server or the client app went wrong but I switched to Pocket even though I run everything self hosted where I can.

- Contacts app previously had a problem where you couldn't save more than one information per page reload until recently when used through CloudFlare which was very confusing to see some saved data disappear.

- News app's UI is a bit of a joke in terms of speed and usability, so you need a mobile app to consume it than from the web.

- Frankly all of the apps need mobile apps because pages load slow and it's reluctant to go through them from the web but the mobile app quality varies so much as they're all dependant on third party developers.

- NextCloud talk, I had a high expectation of it but no matter how I tried, it never worked reliably and client apps were nowhere near the quality to recommend to non tech work people.

- Using an external storage for Files makes the performance and reliability quite low and when you think about using it for work, you need maximum reliability but I ended up using local storage as that seems the only way that's tested enough.

- You can't tell how much of a file is uploaded reliably at all. The "seconds remaining" display jumps all over every second as it's probably calculating from the speed of the moment instead of averages of the last minute or something and you can't cancel the individual files queue.

- I have basic auth on the NextCloud instance but the official NextCloud app refuses to work through it, even though the web version allows you to have yourself logged in via basic auth.

- Email app acted like an alpha version, so I instead use integrated RainLoop which is fine but I couldn't get people to login automatically using the NextCloud logged in user even though I have that option turned on, instead I had to edit some source code to make it happen.

- Logging out takes forever.

I am impressed that the look and feel is good and the caldav being so easy to use but I'd say it's close when it comes to overall quality to have work people use it than just for yourself.

There are already a lot of suggestions to use an externally attached storage (or "sneakernet"). I think it's the simplest solution with best performance. I dual-boot off an EVO SSD in an external thunderport/USB-C enclosure, and performance is similar to the built-in HD. The enclosure is the size of a pack of gum. Make sure the USB-C to USB-C cable is actually USB 3.0 certified.

She should also verify her Time Machine configurations on both Macs are set up to back up her sneakernet drive fairly frequently. ... You only seem to need backups when you don't have them...

Something that isn’t uncommon in some shops is using a big system/VM where the person does their work and just remotes into it from their mobile system as needed, so the file only actually is in one place.

Of course, it only works if you’re online, and VDI-ish solutions aren’t really a thing on OS X, so YMMV, but I remote into my desktop while I’m at school because my laptop doesn’t have much ram.

One thing Dropbox does well is transparently resuming a stalled or failed sync. The other solutions I can think of are more manual or the UX is very basic or obtuse. You could set up your own storage space on AWS and rsync (which has GUI tools available), but that's essentially roll-your-own Dropbox.

I work on large video files that I transport between locations and had an external drive that I used for a while but would end up forgetting or not wanting to unplug everything. I have settled on a high speed 64gb SD card that fits in any pocket or even my wallet. Out of all the fancy network tools I set up, this option has proved to be the most enduring.

Doesn't help with synchronizing which files are most current though. You might look at Transmit 5 from Panic. It has lots of useful syncing features now.

I use an external USB drive. You can get anywhere from the standard thumb drives, up to multi-terabyte larger drives. So when travelling I just take whichever drive I need and whichever laptop I need, and plug them together.

It isn't exactly syncing, but it solves the problem, at least for me.

If they’re reliably on the same network, then I’d probably use syncthing. Or, a NAS device and mount the volume across the LAN. Or rsync or syncthing to the NAS.

If you have to share files off the LAN, that gets more involved.

We currently use a mix of syncthing and NAS for our internal purposes, and iCloud for external syncing.

Use rsync to make one folder structure match another by copying only the changed parts of files. You can sync many gigabytes VERY quickly this way.

You'll want to have an external USB3.0 drive to get from one machine to another IMO, because you would have wanted it backed up ANYWAY right? ...so there's no need to try to copy directly from one machine to another machine, without using the USB drive.

Here's an example command:

sudo rsync -aAXv --delete --force "/home/clay/ferguson/" "/media/clay/BAK/ferguson"

I'd hate to recommend a command line to a designer.

It really seems like there should be some kind of "SyncGUI" app built into Linux that does this sync with a nice GUI, and maybe there is one, I've never looked that up...

because all it needs to do is prompt for source and destination folders, with a stern warning about what it DOES. ha.

Buy a large SSD like 2-8TB (just avoid the Samsung 860 QVO 4TB, it sucks more power during write than USB A standard allows) and buy a nice case like https://www.amazon.com/dp/B07D2BHVBD and mount it there. Work from the external SSD, it's fast enough in my opinion. Use the cloud only for backup purposes whenever the drive is attached (I use Insync for that but this is Google Drive centric).

A Mac Mini expanded with two SSDs with syncthing is superior to sneakernet because it provides replication. And, if you really want to, add a 14 TiB HC520 placed in an external USB 3 case.

Add an rsync server to get backup. Forget the cloud, most solutions are overpriced and insecure. Use TarSnap if you have absolute have to backup reasonable quantities of critical data and call it a day.

Second the use of high speed SSD or SD Card, at night just make sure it backs up to the cloud just in case you lose it.

If just a 2 terabyte cloud service faster than Dropbox would be enough, pCloud is nice.

Try them all and then pick one (or two):

- SpiderOak One

- Mega (very cheap)

- Syncthing (open source)

icloud, otherwise external storage

Reply on news.ycombinator.com