I recently wanted to try to get an LLM to help me organize my photos. (I'm someone who takes a lot of photos when I travel and then back them up to a hard drive -- assuming someday I'll organize them :)

I created a prompt to try to get an LLM to do high-level organization:

> Help categorize this photo for organization. Please output in JSON.

> First, add a field called "type" that is one of: Wildlife, Architecture, Landscape, People or other. Pick the one that most appropriately reflects the subject of the photo. Use Other if you don't feel confident in your answer.

> Next, if it is Wildlife, please add another field called "animal" that gives a general type of animal that is the focus of the photo. Use large, common types like Elephant, Bird, Lion, Fish, Antelope, etc. Do not add this field if your confidence is low.

> If the type of animal is Bird, add a field called "bird" that gives the common type of bird, if you can clearly determine it. For example: Eagle, Hummingbird, Vulture, Crow, etc.

> If it is an Architecture photo, and you can determine with good confidence what specific building (or city) it is a photo of, please add a field called "place" with that name. (Name only, please -- no description).

I've tried with llama-vision using Ollama and it worked reasonably well for the top-level categories. A little less-well for identifying specific birds or places. And it didn't always generate proper JSON (and sometimes added new fields to JSON.)

I also tried with Claude's API -- and it seemed to work perfectly (for a small sample size).

It will be interesting to try with PaliGemma and see what I get.

I have like 50k photos, so I don't want to pay $$$ for the Claude API to categorize them all. It will be cool someday (soon?) for an open-source DAM to have something like one of these models available to call locally.