Show HN: Self-hosted pet diary built on Frigate and a local vision model

I have a cat and a dog, and I worry about them whenever I am away. My cameras ping me with a clip of the cat crossing the kitchen, which tells me almost nothing. I could sit and watch hours of recordings to work out how their day went and whether anything looks off, but that's too time consuming. What I actually wanted to know were concrete things: did they eat, did they drink, did the cat use the litter box, are they sleeping more than usual.

So pet-report sits on top of Frigate, which I already run for the cameras. Twice a day it sends the new clips to a vision model (I run Qwen3.6-35B-A3B on an RTX 5090) on my own server and writes up the day, with every line linking back to the footage behind it so you can check it yourself. You can also ask it things like "did the dog drink today?" and it answers from what the cameras actually saw.

I've found myself needing an app like this for a long time, and pulling contextual metadata out of an image or a video is exactly what these models are good at. Before LLMs this would have taken money, time and R&D I don't have, and I couldn't have done it alone, since I have no background in ML or image analysis. Most of the code here was written by an AI coding assistant, with me directing and reviewing it. The design and the review are mine.

The part I'm happiest about is that a local vision model keeps everything private. Nothing leaves my network.

It does get things wrong, but the whole thing is designed around that: uncertain moments are flagged, and anything can be corrected by hand. Every moment keeps the frame/clip it came from, so you can see what the model saw instead of taking its word for it. The results are only as good as the model you run, the descriptions you give it of your pets, and where your cameras are pointing. If you have two orange cats, it won't tell them apart: it labels the sighting "a cat" and leaves you to pick which one. Fortunately I have one cat and one dog, so every sighting gets named on its own, but I did try my best to prompt the model in a way that smooths out those identification edges.

Anyway, just wanted to share something that has been working for me and my two animals, which is the honest extent of the testing. I'd love to know how it does on yours.