Hacker News Reader

Comment by vocatan | Hacker News Reader

Just to add context - I do have backups, but they're spread across a myriad of different hard drives. I had used Amazon Photos to consolidate, and benefit from the facial recognition to be able to quickly locate pictures of family members quickly.

I think that I'll give DigiKam another go, against the NAS loaded up.

Thanks for the comments!

View on HN

Not that it helps you now, but i also keep all our family photos in the cloud (iCloud in my case), but at the same time i have a small ARM machine at home that keeps a mirror of the iCloud data.

That ARM machine also has the responsibility of making backups, local to a USB drive, as well as to another cloud. Not mirrors, but proper versioned backups (as in Restic, Borg, Arq, Duplicacy, Kopia, etc).

I also maintain a couple of USB drives with yearly updated mirrors of the entire photo library. The drives are stored at geographically different locations, and surface scanned, updated and rotated yearly.

And finally, as a "last ditch recovery", i maintain an archive of M-disc Blu-Ray discs that contain a complete copy of our family photo library. Every year i make an identical set of discs containing the past years photos, and these sets are stored alongside the USB drives.

I don't bother archiving documents as everything that is important is stored on government servers anyway, or exists in hardcopy. Also, if every step in my normal 3-2-1 backup scheme has failed and i need to recover from the archive, i probably have bigger issues than retrieving my budget for this years finances.

> I don't bother archiving documents

Isn't that 2 or 3 orders of magnitude less space though?

I do understand where you're coming from for sure.

You have to fuck around with the storage format because by default scanning sofware, even when scanning documents, will basically just take poorly compressed high resolution images. Maybe there's some service which sorts that out for you but so far I have a little webapp which stores an index of paper copies. Once I figure out how to automate scanning, compressing and OCRing documents then I think it will be worth storing digital copies. But for now it works.

And don't forget adding metadata.

I do scan some paper but there's increasingly less of it and I'm likely to need future access to so little of it that it's mostly not worth the trouble.

https://docs.paperless-ngx.com/ might be of interest here

I do occasionally want to retrieve older documents. As you say, they're smaller and the effort to put individual docs into a "I might want this someday" folder seems more trouble than it's worth. Of course, having a vast sea of basically write-only docs does make finding things harder especially as I don't put as much effort into organizing things any longer.

I still do a 3-2-1 backup of documents with 2 versioned backups, one at home and one in another cloud provider, just like with photos.

The archive however is the recovery if I’m not able to retrieve my normal cloud copy (hacked, ransomware, loss of credentials, etc), I cannot access my local mirror copy (ransomware, dead disk, etc), I cannot access my local backup (dead disk, separate from the mirror disk), and I cannot access my cloud backup either.

For all of those things to go wrong a the same time, something major has to happen. Besides, where I live, most required documents (drivers license, passport, birth certificate, tax records, etc) exists in government databases, so all I have at home will be various documents that maybe have sentimental value, but not exactly needed.

Furthermore, documents change “frequently” where photos tend to be somewhat more static, so I can archive photos, and maybe get <10% “duplicates” due to later edits, archiving documents will pretty much be a lot of duplicates each year.

That being said, I think we have like 1GB documents in total, so it would be easy to fit in the archive.

As a fellow small ARM machine owner, what's your strategy for getting the photos from iCloud? Is there a tool one shouldn't feel weird to give their iCloud credentials to?

I’d like to hear this too

i have been using icloudpd for years. there is a docker image that makes it simple to install on my synology. because of 2fa, i have to re-authenticate it from the terminal every couple of months, but that takes one minute.

https://github.com/icloud-photos-downloader/icloud_photos_do...

I tried this but I think my issue is the library is too large. It can pull a few dozen or a hundred or so photos just fine, but then it will time out but not quit out, so I have to babysit the process. Maybe there's a flag I missed in the documentation to retry downloads, but it basically was a nonstarter to me in the state it was in. I think its apples fault though; I can't get a big zip file of icloud data to succesfully download with their website either, it will also time out. Likewise when I try and update the bootcamp drivers on my intel bootcamp machine, I will get 30% of the way there and then it times out. I'd blame my home connection, but from cursory glances at various forums this is apparently a widespread issue with these sort of downloads form Apple.

Old cheap Mac Mini with USB drive attached. Put Photos library on it. Set it to keep all photos locally. Back up that drive elsewhere.

I was doing just this for a while but stopped because my Mac Mini was too old for the last few OS releases. I then switched to using the Windows iCloud client but a bug from ~2 years ago that consumes tremendous CPU cycles made that less than ideal. (The best you can do is lock it to a single thread, which will then use 100% 24/7)

Now I just don't backup my iCloud, though I do remove everything older than one year every new years to my home server which follows a good 3-2-1 backup strategy.

TL;DR: If you go this route, try to get a Mac Mini that can run a supported macOS for some time to come.

You can do it in a VM just fine too.

There is usually someone who’ll point out that this probably violates licensing, and it probably does if you do it on non-Apple hardware.

I have an older Mac model I use as a linux box: you just reminded me I install a Mac OS VM without violating the license - thanks!

I was actually running ESXi on my Mini for a while and successfully installed macOS in a VM on it. The performance was horrendous though, so much of macOS depends on GPU acceleration which I didn't get. I think I've read newer macOS builds don't even have a software video fallback, though that might just be the Apple Silicon builds which wouldn't apply to me.

It was definitely a fun project even if not terribly useful.

I've been using Windows iCloud client to backup to a VPS. It works okay for files, but for photos it pegged disk usage even when there weren't any new photos to download and my VPS provider wasn't happy. So far my solution is Windows iCloud client for files, and then OneDrive on my iPhone for photo backup, with OneDrive again on the VPS.

I like how the backup is outside my house, but I'm about to add Yubikey to my iCloud account and I'm not sure the Windows iCloud client is going to like that.

I pull mine from Google Photos, with a docker image running a tool: https://github.com/gilesknap/gphotos-sync

Its not perfect, but as a backup, it works well.

That looks pretty good, thanks for linking to it.

It doesn’t talk about not being able to access person tags. Not being able to programmatically access the data about who google thinks is in each of my photos has been an annoying pain point for me for years. Last I checked, the data is also not included in Google Takeout dumps.

I have recently switched to a M1 Mac Mini, and just have each family member sign in to that using Remote Desktop. It brings the added bonus of working as a content cache for anything iCloud.

My only gripe is that it downloads the shared photo album (new in iOS 16) once for each account, and when your photo library is 1.8TB, that suddenly becomes a lot of wasted space. When it comes to backing it up the backup software deduplicates the data, but not for the initial storage.

I really wish Apple would implement some kind of method for backing up photos stored in the cloud without the need for mirroring them.

Before the M1 I was using iCloud photo downloader ( https://github.com/icloud-photos-downloader/icloud_photos_do... ) on a Raspberry Pi 4 which also worked well, but in the end I got tired of iCloud credentials expiring every ~90 days, requiring each family member to login again through a console.

Considering the M1 idles at roughly 20% more than a RPi4 (M1 at 4.5W) it was an easy sell. I just got the cheapest model and added a large USB drive. Using a Mac also gives you the possibility of using something like Backblaze Personal with unlimited backup storage, if that’s your thing :-)

I use Healthchecks.IO ( https://healthchecks.io/ ) to keep an “eye” on the backup status (and other more mundane tasks like monitoring the power state of my summerhouse)

I’ve been using this tool to backup Apple Photos library to an external drive: https://github.com/RhetTbull/osxphotos

> I think that I'll give DigiKam another go, against the NAS loaded up.

I'm also looking for something FOSS that can do basic face recognition + maybe even more, but last time I checked DigiKam's detection didn't work so well, or maybe I got spoiled by the detection in Google Photos. If you do give it a go, would be nice if you reported back on your experience :)

FOSS photo software with face recognition (note that some of this is off-premise face recognition, sometimes requiring an API key you get on your own)

https://damselfly.info

https://github.com/LibrePhotos/librephotos

https://photoview.github.io/

I may have missed some: https://github.com/awesome-selfhosted/awesome-selfhosted#pho...

What I found in my previous search is that either they're really good at managing large collections of photos, or they're really good at doing various AI/recognition, but none of what I've found have been good at both.

Hence my interest in DigiKam as their management features for large collections is second to none.

Isn't that OK? You should be able to find some reasonable workflow around using the recognizer to tag the photo in its EXIF/IPTC metadata, and then load that into the organizer.

It's not not OK :) Just not what I'm looking for, I spend enough time professionally with writing glue code, that I'm looking for a good out-of-the-box experience for something like this.

I think immich will get there eventually

How does the face recognition compare to https://www.photoprism.app/

Not sure if it has been mentioned already but photoprism added face recognition a few releases ago and it is working well for me: https://github.com/photoprism/photoprism

The quality of this software is overall extremely good. It is a solo-developer as well so you might want to consider sponsoring them if you end up using it.

I am not affiliated, just a happy user :)

Seconding photoprism. The face recognition was pretty meh when it was first introduced, more of a gimmick than anything. But (I assume) a recent update pushed it into "wow, this is actually really good" territory.

Thirding photoprism, got it hosted on a raspberry pi and made it accessible outside of my home via Tailscale. Seems to work fairly well... though I could probably make it faster with an SSD vs an HDD over USB.

Photoprism is absolutely amazing. It was the quickest donation I’ve ever made to an OSS project.

I have a Photoprism instance running on my home server backed by a raidz1 ZFS pool (1st backup). Photos are periodically synced to a Backblaze bucket with versioning enabled (2nd backup). The source of most photos is an iOS device with iCloud enabled (main copy). I rely on PhotoSync to periodically sync from the iOS device to Photoprism.

To photoprism users: do you know how well it handles identifying photos of kids over the span of years?

Google photos absolutely blows me away with its ~1% false positive rate (that is, given a photo, correctly identifying with kid(s) are in it) identifying each of my kids, nieces, and nephews. I don’t really know what its false negative rate (a false negative would be an uploaded photo of John but that doesn’t get tagged as having John in it) is, though.

I can search for each of my kids and see photos of them going back to their birth.

Don't have any data on that unfortunately.

There's a difference between face detection and face recognition. Last I checked (less than a year ago) it could find faces, but was abysmal at recognizing them. I spend a few days on it, tweaking this and that, taking to the devs. It was crap.

It was pretty bad until just recently (a couple of months ago?) when I noticed it was incredibly improved, might want to try it again.

I would be careful with hard drives. As I learned couple of days ago when I got my 10 year old hard drive from storage, I was unable to read most of the data as the drive would start to intermittently disconnect from the computer. Most likely the drive is okay, but the board has developed an issue - maybe capacitors gone bad or something. Now, I am yet to check if it is the one that has hardware encryption (I supposed to get rid of all of them and copy data to alternative drives) - there are drives that encrypt the data by default and if you don't set up the key, the encryption still takes place with some default key. If the board dies for some reason, then you won't ever be able to decrypt the data - even if technically was unencrypted, even by swapping the board from a working hard drive. This happened to me once and I lost 3TB of important data couple of years ago.

That being said - I think the best additional backup is to store important things on BluRay - producers estimate they should last 80-100 years and so far never had a bad disk, even those burned years ago.

I recommend ddrescue for data recovery, worked really well when I needed it.

It will resume when you hook the drive back up, and work around dead sectors.

Digikam is the bees knees in digital asset management. I manage my personal photo archive with that, and it punches way above its weight.

Except for it's horrible face recognition right?.. ugh.

I'm away from my desktop computer, but it wasn't half bad when I last updated its DB. I don't take much portraits, so I don't use that feature much, but give it a look when I can and update you.

Latest release failed the tests (with some memory corruption yikes) even. Doesn't inspire a lot of confidence that it won't eat your photos.

But the v7 uses DNN from OpenCV for face detection that supposedly has 91% accuracy on CPLFW which is pretty good! So how come people here say it sucks?

Because many people test it once, and assume that since it's FOSS, the feature is not evolved over time.

"knees bees" ... maybe I'm missing a joke, but I always said "bees knees" in this context as a play on "business"

Fixed it, thanks. Sometimes my mind to finger interface changes packets on the way. I guess my nervous system uses UDP.

You're alone not :-)

"Bee's Knees" == absolutely the best

but is it better than the dog's bollocks?

Nothing beats the duck's nuts

Or the cat's whiskers

All from 1920s slang. Apart from dog's bollocks I imagine.

I'm almost 65 and I never put "bee's knees" together with "business." And they say you can't teach an old dog....

I tried it as a replacement for Picasa, but found it underwhelming.

It was slow processing the images, so took very long after adding new pictures to them turning up in searches.

One of the main attractions, searching geotagged images by location, wasn't smooth at all and quite clunky to use.

Tried it a few times last fall but gave up.

I can’t find an arm image for macOS. That’s a bummer.

I'm glad you haven't really 'lost' the photos since you have copies elsewhere. It is surprising how many people these days rely on some cloud service to store and preserve their only copy of some important and unreproducible data.

You are right about file systems doing a lousy job of helping you organize and locate photos, especially when they are spread across many different HDD and SDD drives attached to various computers. They are equally bad at organizing other forms of content (documents, videos, logs, music, software, etc.)

File systems were invented decades ago when the biggest hard drives could only hold a few thousand files at most, and drives were also so expensive that most people only had one of them. Traditional file systems are antiquated and need to be replaced with something better.

Self-host and use Photoview [1], it has face recognition. I use this since 2 years with 5TB of photos (on ZFS) and it works flawlessly. It is worth it - everything under your control and you decide when to migrate. My phone's photos are also uploaded automatically to a Nextcloud folder, which is read-mounted to Photoview and scanned automatically.

[1]:https://github.com/photoview/photoview

This is my backup method:

* A nextcloud+recognize (for face recog) instance running on Hetzner. $20/month (I co-host other services on the VM). This is attached to a Hetzner storage box.

* Syncthing pushes all of nextcloud to a zfs.rent (great service BTW) machine, I purchased my disks up front, so it's $cheap/month. ZFS snapshots are taken.

* Local RPi NAS. Syncthing up to the other two.

About $40/month in total. How valuable are 240k photos to you?

Uploading photos is a matter of sending them to Nextcloud. Syncthing does the rest.

I'm curious if you've asked your family members whether they're ok with having photos of themselves being used to train Amazon's facial recognition model? I know it's a common practice and Amazon isn't alone, yet I feel people should be more aware of what's happening.

Digikam is less crashy than it used to be. Its slowly going through my 650,000+ photos and finding faces. I started this past Sunday. I expect to be done in a week or two.

Finding faces or identifying faces? Because I tried the latter and it was horrendously bad.

Right now, it appears to be doing both.

I really miss Picassa... if it wasn't for a bug in the last version, where it sometimes swapped face tags, I'd still be using it.

Can second using DigiKam. It's a great app.

Reply on news.ycombinator.com