Find images from movies based on what you draw | Hacker News Reader

Find images from movies based on what you draw

huggingface.co

61 pointszuhayeer30 comments

Looks like it's just guessing the object and then finding images tagged with that object. I was hoping for some sort of scene detection to find specific scenes from films.

Ah shucks! Well, you saved me a click.

Was hoping for some AI visual processing thing and was doing the mental gymnastics of how to store all that data.

It doesn't seem to work for me - you draw a shape and then it returns some completely unrelated images. You then pick one of those images and try to draw it as closely as possible and then it returns a completely unrelated image again.

I thought the weird thing was that with that second step, if I drew a broom exactly for example, and it guessed “broom” correctly, it returns images with no brooms in them at all?

From what I gather this does object detection on the picture and then from those detected items compares to movie stills that have been similarly labelled. On paper sounds like it could work but in practice .. results are a bit underwhelming when I tried

This is not ready for prime time. This is not even ready to go up against infomercials at 4am on a Sunday morning.

I’m trying to draw the silence of lamb poster with Jodie foster but it keeps returning pictures of bearded men. Lol. Let me know if anyone is successful with that poster.

I had to come back to comment - I tried to draw the Batman symbol and the first thing that came back was The Silence of the Lambs poster.

Tried to draw sheep, but it sees cat...

Literally anything -> syringe

Current state of A.I. in a few words.

I drew a closed fist, it says moon / alarm clock.

Try sketching one of these: - bench (my favorite) - drums - airplane - basketball

Are there details on how it works? How is CLIP being used here?

The code is available in the repo: https://huggingface.co/spaces/osanseviero/draw_to_search/blo...

It first uses a simple CNN to get one of the 100 classes. It then uses that class to retrieve images using CLIP. The original CLIP Space that helped for inspiration is https://huggingface.co/spaces/vivien/clip.

Oof. That 100 class limit is less practical than just asking the user for arbitrary text input and using CLIP as a zero-shot classifier. You would have gotten better responses if the limitations and motivations were explained on the page.

Thanks for the feedback! This was just a demo I shared in Twitter and didn't expect it to be shared so widely by others. We're training a new CNN for more classes and we might switch to another dataset for the image retrieval part. It's really a very small pet project :D

First of all, most results are anime. I tried a few easy-to-draw, pretty popular symbols like Harry Potter's forehead scar, The Incredibles logo but got nothing close to them. It's a very cool idea but still needs a lot of work.

Here’s a demo of it in action: https://twitter.com/osanseviero/status/1447830148225589251?s...

Doesn’t work when I try to use it, always returns the same thing, and it seems like everyone else is having the same problem. Sounds cool though and I’ll be excited about the day we progress enough where something like this works! GANs are very cool.

I don't experience your problems. It returns new stuff for me.

[It takes a long time to load, so be patient.]

Nice idea, but I was never able to find the movie I was thinking of. Perhaps I'm too bad drawing.

Drew Batman in the Animated Series style. Kept getting a cat and Deadpool in the Flashdance pose. Cool, but still has a ways to go.

Was trying to draw the "Cross of Coronado" from Indy Jones, got 2001 - A Space Odyssey... There goes my artist career :)

I got a Chainsaw....

It somewhat decently identifies the object drawn, but fails badly to actually find the identified object.

Tried to draw the Deathly Hallows symbol with no luck :( Cool idea though

It seems to keep suggesting the same few movies no matter what I draw

I tried to draw a stick figure and it didn’t work for me.

my mona lisa drawing is apparently a tree

Given the tiny drawing area and the insanely thick brush, you're lucky you got something as coherent as 'tree'

Reply on news.ycombinator.com