Deep learning for visual question answering: demo with Keras code | Hacker News Reader

Deep learning for visual question answering: demo with Keras code

iamaaditya.github.io

73 pointsfchollet17 comments

Even though this is "Question Answering", it is trained as a classification model. Thus the model will try to come up with one of the top "1000" answers it has seen during the training. This certainly limits the possibility of answers and sometimes returns very weird answer.

It is not for lack of trying that all the top papers in visual question answering end up doing this as a classification task. Results are really poor when it is used as RNN generation, and also extending it more than top 1000 answers does not yield any better results. 87% of the questions in training + validation is within 1000 unique answers.

Latest models have started using more complex form of memory and more tightly integrating the question vectors. One of the top model called DPPNet trains a separate matrix from the question vector (chain of GRUs) to find correspondence on the image filter weights. Their idea is that some question have more relevant areas in the image features. Yet another model DMN+, by Metamind uses dynamic memory network which they build to do language question answers but the extension to images work pretty good.

Surprisingly the models that use visual attention are not the best and I think it is mostly because this kind of model requires even more data and longer training. Just taking 10 different crop of the question image and doing voting of answer beats attention models (based on numbers reported by these papers).

Right now I am working on converting "End to end network" -http://arxiv.org/abs/1503.08895 to this task. I tried working on Neural Turing machine but I could not make it work for this kind of task, but it was mostly because of lack of indepth understanding of NTM.

Any feedback from you guys are welcome.

P.S Thanks fchollet for writing Keras and for this post. Can't wait to try Keras 1.0

> It is not for lack of trying that all the top papers in visual question answering end up doing this as a classification task. Results are really poor when it is used as RNN generation

I'd be curious to know if you have a reference for this. Given that the answers are one word, a word-level RNN language model output should basically be the same thing as a straight 1000-way softmax.

   Model 	Q+I [1]	 Q+I+C [1] 	ATT 1000 	ATT Full 
   ACC. 	0.2678 	0.2939 		0.4838 		0.4651

Where ATT Full represents using all the words in the vocabulary, as you can see it performs worse than "Most frequent 1000 answers".

Source:

   Chen, K., Wang, J., Chen, L. C., Gao, H., Xu, W., & Nevatia, R. (2015). ABC- CNN: An Attention Based Convolutional Neural Network for Visual Question Answering. arXiv preprint arXiv:1511.05960.

(a) Several early papers about VQA directly adapt the image captioning models to solve the VQA problem [10][11] by generating the answer using a recurrent LSTM network conditioned on the CNN output. But these models’ performance is still limited [10][11]

(b) our own implementation of this model is less accurate on [2] than other baseline models

Above two quotes are from -

   Xu, Huijuan, and Kate Saenko. "Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering." arXiv preprint arXiv:1511.05234(2015).

However, I think my words were sloppy, as I could not find more concrete proof in the literature, but I will revisit them with detail to recollect where I read about RNN generating answers not overachieveing softmax classification over Top K distribution of answers.

Also, I would like to note that, I am not using only "one word answers" as the possible set of answers. It contains few two words, and very few three and four word answers.

Here is the distribution Key == Length of words | Value == Count of answers with those many words

Counter({1: 855, 2: 112, 3: 32, 4: 1})

Does this actually make sense at all? Usually I figure out the meaning of a sentence to form an answer, which requires thinking about more that just the words in the given sentence.

So, I experimented with Skip-thought vectors (as question), to embody the the semantic knowledge about the question but that performed poorly, I mean very poorly. Tbh, I was surprised with that.

Currently the way system works, is highly biased towards the individual words in the question, whenever it sees the word "color" in the question it is going to pick one of the 'colors' as the answer, even if the question is not asking about colors. This is certainly due all the priors from training data.

Thanks for your work on this, I find the VQA task really interesting.

The classification-based approach is definitely the part I find unsatisfying about this task. The problem to me is that it biases the models learned very strongly towards the data that was collected for training and testing.

Has anyone tried outputting a vector from the model, and using cosine to predict the nearest word/phrase/sentence etc? This seems to work for non-visual QA.[1] Training is performed using noise contrastive estimation. I've discussed this idea with the Virginia Tech team, but I haven't had time to try it, and they seemed a little skeptical.[2]

[1] https://cs.umd.edu/~miyyer/qblearn/

[2] https://github.com/VT-vision-lab/VQA_LSTM_CNN/issues/14

hi syllogism

Right now everyone doing this is highly focussed on the competition and trying to beat the numbers. For that purpose certainly they would want to stick to predicting Top K answers.

For e.g see this table

  Model 	Q+I [1]	 Q+I+C [1] 	ATT 1000 	ATT Full

  ACC. 	0.2678 	0.2939 		0.4838 		0.4651

Where ATT full is when using all the words, it performs worse than ATT 1000 (Source)[Chen, K., Wang, J., Chen, L. C., Gao, H., Xu, W., & Nevatia, R. (2015). ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering. arXiv preprint arXiv:1511.05960.]

Once the competition is over (in rough two months), there will be more focus on actual AI part, where generating the answers would be the right thing to do. There are other papers where they use external knowledge base like DBPedia, certainly "answer word" could be picked up from there.

What you have suggested is a very interesting approach, and I am not aware of any paper which has tried that. Certainly quite a few paper have tried to extend NLP QA to Visual QA but with limited success (expect Metamind people). I will certainly keep that in my ideas to try list. I will update you if I get some results.

P.S: Thank you for creating Spacy, I love it and I use it everyday !

Comparing Obama with monkeys and bananas is a bit unfortunate considering it's a common racist stereotype.

There is a very weird behaviour in this image that (to me, at least) speaks a lot about the (low) consistency of this method: if upon the last image a sport-neutral question is asked, then the answer is:

    40.52 %  tennis
    28.45 %  soccer

But then "Are they playing soccer?" is asked, and the answer jumps to the following:

    93.15 % yes

What would it happen with tennis then? How can this make sense?

My only rationalization would be along the lines of, "Well, it guessed it was a sport at least", but no person would answer these question like that...

Humans without experience with either sport could potentially run into the same difficulty distinguishing them. For instance, how would one tell the difference between Aussie-rule football and rugby based on a picture?

But my point is, if the network believes with the most confidence that this is tennis, shouldn't it answer it's not soccer in the next one? Or at least with a hugely less confident answer? The fact that it answers like that makes me think it is not a very stable source of question answering.

If the question has some inittial guess like "soccer" then this guess should be added up, as a Bayessian point of view.

It's not based on a bayesian system, it's a neural network that does question answering. There are no logical constraints in there. Subtly different wordings of the question could produce different answers, even if the meaning is the same.

I know, my point is that there doesn't seem to be a part of the system in which the two sports are weighted against, and one consistently outweights the other one. So the model is so sensible that even with exactly the same image, and the same target (identifying this as soccer), you get wild variations by changing the question - and the question not even providing a clear bias.

Hi Harperlee

I ran the question you asked --

"Are they playing tennis?"

and following is the result

99.93 % yes

00.05 % no

000.0 % right

000.0 % left

000.0 % black and white

Thanks for adding more information! This is very interesting. So what's limited in the response then is being able to discriminate among the separate semas of the answer. The network is sure of a lot of common things among the two sports but can't communicate that perhaps tennis is more likely than soccer...

Ideally there should be a way for the model to say "soccer is a type of sport, let's apply this question to every other sport and only say yes if soccer is the best answer", right? I'm not familiar with Word2Vec, but, presumably, there may be some way to get that information and work it into some logical layer on top of the model.

One thing I see all the time with machine learning is trying to solve everything with pure learning, which is quite similar to human intuition in that it can be hard to rationalize. But the thing about humans is that we have these logical constructs that filter our intuition, so it makes sense to create generic abstractions of these constructs and apply them as a sort of filter to the output of a trained model. Of course this has probably been done before, though most of these project I see don't seem to attempt it.

Reply on news.ycombinator.com