Lychee – Self-hosted photo-management done right
lycheeorg.github.io
lycheeorg.github.io
Please someone make a photo/video storage for families. Our photo stream is not artisan award-winning landscapes, they're messy bunch of travel photos, pictures of receipts and random exif-less pngs sent to us.
- Altar
- Bookcase
- Festival
- Keyboard
- Streetcar
Some are very wrong (A photo of a lake as "gallery"), some are wrong but tricky (red geode with white veins as "meat") and most are spot on.
Edit: In fact, I'm not sure that I have the facial recognition configured... perhaps its in a newer release
I just tried the Photoprism demo (https://demo.photoprism.app/browse), searched for "person", "grass", "wave", "head", "eye", "neon" and a bunch of other terms, nothing seemed to find any pictures although I took the concepts from existing photos I found.
Back button is also broken on the demo, which makes it seem like the most basic UX is not there yet in the application.
The quest continues...
I think their actual multi-user story is somewhat lacking, but if you want a shared picture dump with some features that help sorting through a bunch of random images it should work well.
Right now I'm working on a desktop tool (Linux-first because that's what I run) to take a bunch of folders as input, find duplicates and let you clean up the duplicates or "merge to destination" because this has been one thing I've struggled to find something nice to use for.
It's early days and I'm only so far at the point of hashing images and detecting and counting duplicates, but as an experienced backend software engineer the UI tools are a real learning curve.
I'm also trying to make the UX clean and simple with next to no UX nouse.
I'm using Wails + Svelte; I've not worked with Node, Svelte or Wails before so I may or may not end up switching techs but the backend is in Go.
It's not one of these photo gallery/management tools but I've been finding that really, all I honestly need for my family photos that are reaching terabytes in size is: that I'm not waating space and I have a good archive of everything in one place.
I'm curious if this is a tool others might be interested in.
Multi-user is lacking to missing. I'm not 100% sure which state photoprism is in, as I was shopping for a self host service. OTOH I don't care about that for family. I'm not hosting anything for untrusted people, or to default backup everything, but as common library. So I'm fine if everyone sees everything and expect people to play nice.
But I do agree that multi user (and maybe integrated auto-sync for phones) are major missing features. I'm just in the situation that it doesn't really matter to me currently.
The way I had Photoprism set up is that mine and my wife's phones would sync images nightly to the server, the photos would be rsyncd from the public facing sync server to an internal one (not in the DMZ) then I'd have the server send me a link to the import page on Photoprism because it lacks the feature to autosync when new files are added; I think having to open the page while on LAN (link was internal only) and click a button was the straw that broke the camels back here for me.
I think I also had something like 10-12 cores dedicated to it, which alongside the RAM use that constantly crept up, and the manual intervention, was just too much for a photo management tool that wasn't really even managing the photos in a way that was useful to us.
I had considered selling it for a nominal fee too but that comes with additional commitments from me, that I'm not sure I want to take on.
Maybe I'll just add Github Sponsors, KoFi or such and see if there's interest, when it's ready and released.
I have photoprism running on docker, which is running as a VM on a proxmox server that is a 13 year old Dell Optiplex 9020. Only 6Gb ram dedicated to the VM (2GB initial, ballon to 6GB), and photoprism uses 2 workers. I have a large 50k set of photo's from starting from when my daughter was born till now (17). I have never sorted through these photo's other than to dump them from my phone every 6 months (camera before smartphones become common). Initial import took less than 45 minutes and that was before tweaking settings. Full manual index with all options, still takes less than 45. Initial face recognition was about 33 people, once I tweaked settings, it's recognized 97 people with little input from me other than tagging the name correctly for a few photo's per person.
The only difference in my docker file vs the official are optimized settings, and I'm using a separate VM for the database server as I find multi tendency and backups easier with a VM. (VM only has 2GB ram allocated to it)
I'm extremely happy with it, as I always put off organizing my photo collect due to how massive it is. It's made the task so much easier.
I am however always looking for other tools. I'm interested in give yours a try, just wanted see what could be the massive difference in experiences.
Edit: as for my tool, it won't replace Photoprism, its primary goal is to deduplicate your photo library.
I'm currently using HomeGallery[0] behind Authelia[1] for authentication to view so many images effectively. For uploading, I'd been using Nextcloud, but it began to lag noticeably after a few thousand photos. I switched to FileRun[2] with symlink'd photo directories and a user for each family.
With HomeGallery, I get the desired performance on mobile devices with de-duplication and tagging. My instance detects objects fine, but I owe it troubleshooting time to figure out face recognition. The "similar images" feature can be fun with so many photos. A nice tagging modal on keybind per image would be a nice-to-have.
Using FileRun for uploads works fine, but I also needed a continuous cron job for docker exec to generate any missing thumbnails.
[0] https://github.com/xemle/home-gallery (or https://home-gallery.org/)
[1] https://github.com/authelia/authelia (or https://www.authelia.com/)
It does it in few steps, like reading metadata, creating thumbnails, transcoding for browsers, face detection, nudity detection (lol they really do have it) etc etc. The thing is _all_ the steps are performed for each picture/video in your library. Picture 5 won't be even visible in your library until pictures 1,2,3 and 4 don't have all the thumbnails, optimized versions, labels from TF model, detected nudity and many other things.
My 10yo two-core celeron nuc begged for help, I couldn't see it suffer.
I tried to contact devs on github, the response was "go import all your 40K pictures manually, subfolder by subfolder". I checked, it's now marked as "accepted answer", haha.
https://photostructure.com/faq/why-photostructure/
The novel random "taste" UI makes navigating even very large libraries fun and serendipitous.
There are a ton of configurable image and video filters that can prevent the exif-less screenshots and other random nose from getting imported in the first place. Hop into the discord if you need any help with setup, I'm online.
(Also, know that I've open sourced a bunch of the core functionality, but this is commercial software, albeit with a very generous "free" tier).
That's why all of PhotoStructure's metadata changes are stored using EXIF or XMP standards either within the file, or alongside in a sidecar (if you prefer). File organization is also completely customizable, and designed to work in concert with other DAMs, if desired. Details are here: https://photostructure.com/faq/system-of-record/
Also: if the business ends, the code becomes open source. Details here: https://photostructure.com/faq/why-photostructure/#if-photos...
Parallelism is provided by https://github.com/photostructure/batch-cluster.js/
Metadata reads and writes are via https://github.com/photostructure/exiftool-vendored.js/
Persistence is via SQLite: https://github.com/WiseLibs/better-sqlite3
Image transforms are via Sharp: https://sharp.pixelplumbing.com/
My more nerdier blog posts are tagged here: https://photostructure.com/tags/coding/
Create a trip, upload a 1000 of pictures, delete all but 100-200, see them on the map or on the timeline. Add comments to individual pics or to location or to dates.
And yes, the whole family can upload pics and they can be batch assigned a date or location.
Too early to show anything yet.
It's not a shared album with lower res dupes, it's a real camera roll implemented as iCloud photo stream that multiple people can be permissioned to, with photos from your regular camera going in either manually, or automatically based on geofencing or proximity to other roll members. It can also retroactively suggest merging photos from shared events etc.
Your main camera roll then can show only private photos, shared photos, or both.
Apple One for Family includes 2TB storage and account owner can now optionally merge in their own 2TB for a total of 4TB for family photos.
In other news, iOS 16.0 now makes it easier for photo notes to end up in Notes not photos, also easier to snap send and delete, snap images of text to actual text, etc, many easier ways to avoid gunking up the photo roll.
I found not so popular web app - FileRun - and have been very happy with it so far.
Even though you use filerun to host files, you can still point Lychee/Plex etc to the photos and videos, which might be inside a subdirectory under the root directory of your filerun instance.
---
Exif tag management is a nightmare. Dublin core on steroids. Date and time handling for approximate time knowledge kills many systems.
It has to make choices about photo import implications for file path, and for file atime and mtime and multiple exif times, and private tags.
Google honours a ridiculous small set of tags, and never reread. Google does sidecar files to avoid file change breaking hash values.
All decisions have consequences. Photoprism and exiftool forums abound with special cases. A million of them.
I'm on the fifth major iteration of image hashing at this point, using a L*a*b mean hash, along with a kmeans-gathered set of dominant colors, along with dynamic thresholds that take into account differing mimetypes, fuzzy captured at times, and monochromatic images.
This explains a bunch of the issues and tradeoffs I made while assembling the heuristics in PhotoStructure : https://photostructure.com/faq/what-do-you-mean-by-deduplica...
It's AGPL
They have the community version/premium model. Differentiators of premium (from https://photoprism.app/features):
* support
* higher max res (900MP vs. 150MP)
* non-rate limited reverse geocoding
* higher quality maps
* premium themes
* visual configuration options
* hardware video transcoding
What makes AGPL a deal breaker for self-hosting one’s photos?
However, I will say that combined with the freemium model it does give me pause. The reason is I will likely want to add some features that will conflict with the premium version. That means that my changes have near zero chance of getting incorporated into the community version and I'll have to maintain a public fork.
Like the kind of developer that tries to write the Rust/Go/Erlang/Haskell version ends up stuck on finding the perfect way to handle errors, trying to include a complex ML recommendation system, or creating a custom embedded database with fast lookup times for the possibility of albums with 100 million photos.
Meanwhile, PHP and Ruby keep pumping out these fun little systems that have security holes every few months and need constant babysitting.
Sorry, I don't mean to be negative. I'm really glad to see this, it just prompted some reflection.
Surely there are critical web applications that need compiled code and scalability, but hobby projects benefit from the accessibility interpreted languages provide. Maybe if it becomes popular the effort of porting it to a safer and more performant platform would be worth it.
I think that such tools are mostly successful as products, not as code bases. So the successful ones are likely designed and implemented by competent photograpers with some programming chops, who naturally pick languages with very low barrier of entry.
Especially considering performance is often not crucial, as long as it's fast enough: worst case scenario, you just throw more servers at it.
The cost of hosting is often a small part of your budget compared to developers time (unless you're using AWS or developers from third world countries).
One nitpick, the .git directory is rather large and probably needs some attention as it is 129MB. The tests directory also has a bit of photo content. Perhaps this could be stored outside of git somehow? Excluding those directories the repository would be less than 11MB.
[0] https://ente.io/
* Automatic album generation based on detection of an "event", e.g. a cluster of photos taken together around the same time, organizing them in a folder, even better using an optional album title, e.g. 2022-06-24 - Picnic at the Park
* Modern UX
* Mobile support
* If no sync support, at least the detection of new files in a folder, say from syncthing
These are the core features IMHO, and everything else is icing. I'm basically looking for something that can replace Google Photos in a reasonable way.
I'd also be interested in such a service. Sounds hard tough if you don't have lots of pictures to correlate to.
- loading thumbnails should be near instantaneous for the whole album. If it is taking longer, then let me downscale further.
- loading full-scale, several megabytes, photos must continue in the background. otherwise I’ll be waiting on a black screen for several seconds during a slideshow. It’s something dumb to do, but somehow Plex can’t seem to handle that properly…
Oh, sure, it's easier but going that route means end up in the mess we all know of modern "devops and alike" no one can really know nor manage.
If we rediscover the classic Plan 9 idea of anything is a stream on a network we can create our OWN web on our desktop, choosing what to cache locally, what to sync etc, in a far more powerful, simple, resilient and flexible way.
A small example: we have since decades OFX feeds for financial operations. Why instead of spending gazillion of resources to build and keep up web-banking porcals (portals are another kind of beasts) when we can just give customers credentials and relevant links and the customer operate on a simple XML stream with his/shes favorite tool, with anything in a single place, it's style, automation, ... Perhaps because of surveillance capitalism?
My point here is why the hell tie our external masculine genitals to someone else computer and layer crap o crap investing gazillion of resources to make the monster work well enough to be used instead of rediscovering the classic networked desktops model like the one Xerox propose many decades ago?
Oh, sure surveillance capitalism... But are we really so dumb in mean to keep this crap stand?
How simple is just offer a webdav/sshfs (with isolated and scponly accounts) on a homeserver that have an ipv6 global with a proper dns nice domain and those who want just mount the share and read it with their own preferred software like the old classic Plan 9 model?
- extremely fast. I'm using it with my 70 000+ photos. Scanning is ~10x faster, then with PhotoPrism, and it works even without scanning.
- It just works off file system. DB used only for cache.
- it uses file folders structure - no timeline
Nextcloud is nowhere close to what we need and librecloud seems to lack performance (maybe because of the frontend architecture?)
I'm currently on librecloud (after a migration from google photos) but I'm half thinking of building my own with an eye to performance
> https://github.com/LycheeOrg/Lychee#build
> Lychee is ready to use, right out of the box. If you want to contribute and edit CSS or JS files, you need to rebuild https://github.com/LycheeOrg/Lychee-front.
So their JavaScript source code is in a separate "Lychee-front" repository and they commit the build artifacts into the main "Lychee" repo. Not sure why they would do this.
The best way for other people to look at them is via a static webserver serving static html files generated by some photo gallery tool like jigl. It has no active components to fail or get exploited in the future.
For some use cases only. Most of my photos are not public and a large chunk of them is being shared with small groups of people only. I'm not going to risk public-but-hidden URLs just in case they'll get indexed by accident. A static site is not usable at all for me.
A dynamic system PHP like Lychee is very hard to infeasible for an average non-technical user to keep maintained and secure. Even technical web developers have trouble keeping php projects secure over years. A static webserver with http basic auth is more secure now and incomparably more secure over 5-10 years. If you are worried about security then definitely do not give a third party PHP system access to your files.
Personally I have so many pictures that I really appreciate the "memories" that Apple Photos extracts and shows me from time to time. That would be very hard to implement yourself. It always picks good pictures and not random screenshots you have lying around or blurry pictures.
A directy listning is definitly not a scalable solution if you are not looking to just go through all pictures of a specific date.
Searching photos using text, handling live images, displaying on a map by gps coords, categorizing people/places/albums automatically.
None of this is handled by a filesystem or jigl.
to augment that i use a plain file browser. nemo in this case with the cover-thumbnailer extension works great. i could use another image manager, but then i'd be switching back and forth between different image managers to have the features i need. so why should i, when a plain file browser just works
For previewing, it would find faces and locations, I would be able to group by date, location, etc. There would be a way to see the photos in a browser and share, but only see the photos, so kids can't accidentally delete theme (unlike Synology's crap software only made to tick boxes, boy was that a waste of money). And it needs a timeline view, like google photos, or what smartphones do.
But this comment is so inaccurate I had to comment.
iPhoto/Photos doesn’t store your photos in a blob. They are exportable either via the stupid app or by the file system. The library is literally just a folder, and the images are stored by year month and day. It’s not magic.
It doesn't "import", it uses the originals as is and indexes/stores metadata in sidecar files.
You're welcome.