Some notes from experimenting:
High level, I would expect that using a LLM to caption the images post SAM would do better than CLIP by itself.
I wish that it put my search query and mode as a URL param so sharing would be a bit easier.
Main use-case I could come up with would be spotting areas lacking accessibility features or maybe homeless encampments? Idk, I definitely couldn't think of any commercialization ideas off the jump.
Another thought I had was why you guys chose aerial satellite instead of street view data? I imagine something like "palace of fine arts" would have worked with that approach.
- "skate park" has some classic dense vector failures where it will find an outdoor playground-style area more similar than an actual skate park. Also, big mode is really cool and works much better for these kinds of queries. "chess board" will rank checker patterns over actual chess boards. Bunch of examples will follow this pattern. I wish there was an additional search mode for LLM generated descriptions of the segmented images, but there are probably cost constraints there.
- "USF" also didn't work well. I guess not surprising given it's CLIP, but still kind of interesting. I wonder what it would take to make the multi-modal models better at OCR without actually doing OCR.
- "beach" didn't work great which surprised me
- "picnic tables" and "lots of people" also didn't work | no idea why