I was similarly frustrated with Photos and worked on a similar idea. My first prototype used CLIP-as-service running on MacOS which is quite fast but meant for building web services. I eventually got the concept running on iOS using CLIP via CoreML but still have some problems to solve. I've faced many of the challenges described in this post. Feel free to reach out if you want to compare notes. Best of luck!
By the way, I encountered an issue on my first load. sorry, can't send a screenshot at the moment. In the auth flow, I granted permission only for a subset of photos (about 200) and now I see "Total -1 Photos need to be indexed". So maybe there's something wrong with that case. Also, the user should be able to update permissions from settings after the first time, in case they want to allow more or less access in the future.