As you can imagine, the problem of getting nutrition information from food is a really difficult problem, so we are breaking it into chunks.

For our first phase, we are using user input to provide a baseline of nutrition information. Some people don't want to use the existing apps which require you to select an item from an auto dropdown, then select the quantity.

For our second phase however, we are doing what you said. With the image and input data, we are training our artificial intelligence system to see if it can fix user entries, or at least provide a 'guess' of ingredients for the user.

Some interesting solutions we have seen from Harvard research papers is to get an image, then generate vector image shapes. With the image shapes, we can access portion size + the caption and fix that.