Run CLIP on iPhone to search photos
mazzzystar.github.io
mazzzystar.github.io
- The layout on my 2020 iPhone SE is a bit messed up, the first picture overlaps the search bar which is a bit frustrating
- I am not sure the app uses geolocation. Searching for Paris, it itendified some pictures with typical parisian architecture, but not all. Is it actually possible to include geolocation data? (EDIT: ok, read your other comment about it. Regarding servers, the app is already 300 MB, i think you could hardcode a list of like 10k places (e.g. a circle of e.g. 5km radius around the lat/long center of paris, a circle of e.g. 100km around the lat/long center of a country (or multiple circles).)
- I very much like the privacy conscious no-network-connection aspect.
- PLEASE (if possible) include a link to the original picture in my Photos app, so this way the workflow would be: I search a picture -> I open it in Photos app where I can see the other ones taken around the same time. If that is not possible (which I could imagine is the case), Your workaround of displaying the date and time of the picture is probably the next best thing.
Thanks for making this app!
Regarding geolocation, the photo information is available in latitude and longitude, but converting them to specific cities requires access to a geolocation conversion service, which means an internet connection is required, which is not what I would like to see.
Jumping to albums from the app is not allowed, but you can find the 'i' icon at the bottom of the photo detail page, click it and you can see the date information.
> but converting them to specific cities requires access to a geolocation conversion service
I made an edit to my original comment, you could store certain areas (circle with radius / polygon) and check if given coordindates in that area, but its not something that is crucial for this app to be useful.
> but you can find the 'i' icon at the bottom of the photo detail page, click it and you can see the date information
Yes I saw that, its quite useful, a shame that Apple restricts jumping directly into Photos.
2.Geolocation Yes, I have this plan too: store a large geographic mapping table of major cities and then find the closest known city by some calculations.
3.Date Information haha I think so too.
App works beautifully and layout is beautiful on latest iPad Pro 12.9", including in the new iPadOS windowing where you can change the window to arbitrary sizes.
Stage Manager: https://support.apple.com/guide/ipad/move-resize-and-organiz...
The app also indexed 160,000 photos in a tiny window while running other things (Safari, Teams, Outlook, etc.)
On the overall use case, I found results are on point, far better for descriptions of photos. It only takes about 9 seconds to find good results from the 160K photos.
https://twitter.com/getrememberwhen
I recently added faces to it and have it setup for geo searches but too little time to work on it.
You can't link to photos in the photo app sadly.
At that time last year, I had just had an idea, but was quickly overwhelmed by work. Until I quit my job recently, I had full time to do development. If possible, we may work on something together in the future :)
[1] https://paulw.tokyo/post/real-time-semantic-search-demo/
I have a problem however: the app only sees a fraction of my images. I do not use iCloud, but backup and sync locally with macOS.
Photos.app/sync is configured to copy & delete photos from iOS to Mac, then sync them back through a "all photos of the last two years" smart album.
Those photos appear in iOS Photos, but seem to be invisible to Queryable - it only sees those photos taken after my last backup & sync. Directly after backup and sync, it sees no photos at all.
I think I've seen that once before in another app, so that is probably a bug in iOS/macOS and nothing you can fix, but I thought I'd let you know anyway.
I like Apple’s photo search, I use it daily and can't complain much about it (except it thinks "chicken" only refers to the fried variety).
Queryable can find live chickens I'm happy already.
One thing I'd really need though is the ability to open the picture in the Photos app after it find one.
I use Google Photos on my iPhone just that I can do searches like this on my library. (And that I can auto-share photos with my family, half of them are on Androids.)
This is on some queries much better than that. Which is amazing, given that it runs locally.
It still doesn’t have the face recognition and the geographical information that Google Photos have.
Including a dump of just lat/lon/rad->city from openstreetmap probably kilobytes. Including towns and features might be low megabytes.
But your workflow is such a hassle that I doubt many people would do it. A more realistic flow (without the list of places above) would be to show an image of the world map, allow the user to zoom in, and let them drop a pin on the map. They can drop a pin near some pixels labeled Paris, but the app doesn't need to know that it's Paris, it just needs to translate the pixel coordinates of the pin (relative to the image) to geographical coordinates, and look for images close to these coordinates.
Then on selecting a location you present a button that then jumps you back into the app via a universal link (passing in lon/lat)
But up to you
It's just more hoops to jump through, so simply avoiding it is best if you can.
The page starts "I built an app called Queryable, which integrates the CLIP model on iOS to search the Photos album OFFLINE" - just to clarify, the Apple Photos default search runs offline too (and does the indexing offline). Google Photos search needs a connection, Apple Photos doesn't.
I wonder if it's possible to speed up the search with something like https://github.com/google-research/google-research/tree/mast...
Also kind of surprising that something like this is not officially supported already! In my books that means this is a Good Idea
ScaNN doesn't use a second ML model, it's just an efficient way to store a bag of vectors. But if the software is too heavyweight for you, you don't necessarily need all of the vector quantized multi-level trees and bit twiddling tricks, you can implement 20% of the work for 80% of the speed gain.
Here are a few super simple approaches:
- Random projection: instead of doing the search in 1024-dimensional space, randomly project your vectors down to 16 dimensions and do the search in that space instead. Objects that are far away in this smaller subspace are at least that far in the original space, so you can use this heuristic to prune most of the dataset away; then, you can rank the closest items using the full nearest-neighbor search to get exact results.
Dead simple, lossless heuristic, speedup factor is (new dimensionality) / (old dimensionality).
- Locality-sensitive hashing: in addition to storing the vector representation, store its hash. This can be quite simple, e.g. random projection LSH converts the vector into a series of bits, according to which side of a random hyperplane that vector falls on; see https://www.pinecone.io/learn/locality-sensitive-hashing-ran... Unfortunately, this will be "lossy" - some images will be missed if they fall into different hash bins, and far images may be hashed to the same value if the region w/ equivalent hash is thin/narrow/oddly shaped.
More complicated, lossy results, can search unlimited images in constant time.
- Multi-tree lookup: break the space down into a KD-tree and search that instead. Complicated, and it stops working in fairly high dimensions (certainly don't use this on 64-dimensional vectors or higher)
ScaNN integrates most of these techniques, plus a few more arcane approaches that depend on processor-tuned heuristics. The gory details are available in the team's ICML2020 paper, see http://proceedings.mlr.press/v119/guo20h/guo20h.pdf
I was similarly frustrated with Photos and worked on a similar idea. My first prototype used CLIP-as-service running on MacOS which is quite fast but meant for building web services. I eventually got the concept running on iOS using CLIP via CoreML but still have some problems to solve. I've faced many of the challenges described in this post. Feel free to reach out if you want to compare notes. Best of luck!
By the way, I encountered an issue on my first load. sorry, can't send a screenshot at the moment. In the auth flow, I granted permission only for a subset of photos (about 200) and now I see "Total -1 Photos need to be indexed". So maybe there's something wrong with that case. Also, the user should be able to update permissions from settings after the first time, in case they want to allow more or less access in the future.
Also, Queryable doesn't seem to support granting permissions to only some albums at the moment, and while I know that sounds silly, I really don't currently deal with this issue. You can uninstall and then reinstall, grant full photo access (if you want), and try again.
https://www.allposters.com/-sp/Carl-No-New-Yorker-Cartoon-Po...
Nice work!
3. Any requirements for the device?
iOS 16.0 or above
iPhone 11 (A13 chip) or later modelsAnyways I want to report a minor bug, where the text displayed on the screen saying that the XS isn't supported runs off the left and right margins of the screen rather than wrapping. The displayed text is "ble does not support iPhone X/Xr/Xs, please req"
Edit: I saw further down that only A13 and up are supported, even though I did not get a warning about running on an A12. Bummer.
The only thing that I think could be improved - allow loading original images in the search results. If you are using iCloud to store the images then only the small cached versions are returned.
Only then will Queryable request network permissions in a pop-up window. After granting it, you kill the app, reopen it, and it will automatically download from iCloud.
Once a photo is found, is it possible to jump to that photo in the Photos app?
Am I missing a way to jump from the photo preview to the photo in the Photos app?
Does this work with a terabyte of offloaded iCloud images? Does it generally work with offloaded images?
It's something I've been playing with myself off and on, but at a much poorer level than your implementation.
Or would anyone like to argue that this tool is a violation of someone’s rights when their copyright protected works were used to train the CLIP models and that they should be compensated?
The people who don't like these models don't like the ones that are posed to replace artists. They are not really worried about stuff like clip groups images together.
I do believe that makes their copyright argument slightly bad faith. They only valid argument in this context would be that they are angry that stable diffusion is distributing their images, because training on their images clearly hasn't been an issue for at least a decade.
I don’t see how it is a straw man at all. CLIP is an integral part of Stable Diffusion and where all of the artist’s names are embedded. It has the exact same issues with fair use as any other model that has trained on data without permission.
The courts have already made a distinction between the tool and the outputs of the tool, starting with Sony v Universal.
The SD defense will absolutely use programs such as this iOS CLIP app to show “commercially significant fair use”.
Someone can’t say that a model is fair use when used for natural language search of their own photos but then that same model is not fair use when used to create a new image. The model is either fair use or not. The outputs are either infringing or not, and this assessment is not based on the tool used whatsoever.
Edit - I saw in "Settings" screen that there is no support for iPhone XS and that I need to request a refund. I've done that now.
Luckily Apple makes it super easy to get refunds!
I'm excited to try the app once it starts working on my device.
https://developer.apple.com/documentation/bundleresources/in...
I would also consider putting this information to the very top of the description to try to avoid bad reviews.
---
One more feedback: I struggled to find the App Store link in your blog article. I would definitely recommend using Apple's well known marketing badge[0], you can still enhance it with your brand if you want to.
Congrats on the app and good luck!
[0] https://developer.apple.com/app-store/marketing/guidelines/
(I'm sorry, I haven't dowloaded the app so I'm guessing on your UI, just know its a fairly common mistake) If you're just putting up a dialog saying "building index", running the code to build the index in the same thread, then dismissing once done, it will lock out UI updates, eventually leading to a crash (I think it's 30 secs for the timeout).
Running code in a high priority background thread with a callback to the main thread is fairly trivial in iOS, there's the fine grained control of GCD, Combine for react like code, and the new async/await pattern.
A UIActivityIndicator can be used to show activity without knowing progress.
- iOS 16.0 or above
- iPhone 11 (A13 chip) or later models
AFAIK you cannot set arbitrary compatible devices. You can only set UIRequiredDeviceCapabilities, which are quite limited[0]. There is a required capability for `iphone-ipad-minimum-performance-a12`, but not for A13.
And also if you already have the app in the App Store, you can only relax these requirements. You cannot add new ones as that could prevent users who already bought the app from launching it.
BTW the minimum requirements are listed in Queryable's App Store description as well, although I would consider putting them on the very top.
[0] - https://developer.apple.com/documentation/bundleresources/in...