Show HN: I built an AI that recognizes food | Hacker News Reader

Show HN: I built an AI that recognizes food

infino.me

202 pointssubcosmos127 comments

Something like this would be a fantastic addition to MyFitnessPal.

I find myself sometimes not logging food on that app because I couldn't be bothered to search for it.

On the other hand if the product has a barcode I always scan it in because that is convenient.

Same - its one of the reasons I wanted to make my own. That and data export functionality, which is requisite for research.

I feel like its easy to make a better app if you don't try to please all use cases - just try to make the food logging in less taps. Logging in those apps takes me around 5-10 taps plus typing the food name. Ify ou could do it in 2 or 3, it would be great.

If you're eatting a lot of food with barcodes on it then your doing it wrong.

I wonder if somebody would buy a "highly opinionated calorie tracker" that only has non-barcode foods stored in it and mocks you when you try to add bad food to its database. If you miss your macros for the day, it would then text everyone in your phone that you're a fat weenie with no willpower.

jcoffland would. He would throw money at it. A tracker that doesn't even allow for barcodes to be scanned? Perfect!

I don't understand this sentiment. My grocery store allows us to print barcodes that describe the weight and price of any produce or bulk order we purchase. This ranges from quinoa, oats, nuts, seeds, tea, along with the usual fruits and veggies.

Also, don't bulk packages of vegetables tend to have barcodes on them inherently?

But those barcodes wont show up in a public database. Also bulk foods are not fresh. Fresh food from local farmers does not usually have a barcode on it. This is not just snobbery.

I see you work for the folding@home guys. Excellent work!

All food has bar codes.

Produce is probably the most common type of food that does not typically have a barcode. It has a product code on it that is entered in manually while the weight measurement is taken.

Meat from a butcher does not. Food from your own garden does not. Restaurant food also does not typically have barcodes.

Your own garden? True. Everything else you just listed actually does, at least on the ingredients (for prepared food).

The more general point is that food in the US has to be at a certain level of quality to be sold, literal barcodes aside.

I would pay something like $5 per day for this if it was a real person who looks at my photos, categorizes them accurately, and enters them into MyFitnessPal.

I won't trust an AI until it has a massive database and a million users. Until then, it's just not going to work well enough. You also need to convince users why they should use your app, instead of MyFitnessPal (which already has a massive database, including barcodes.)

I have considered something like this as a business model, although I was thinking of nutrition coaching instead of mere logging itself, but thats actually a really great idea.

There's nothing stopping you from having a human aspect do the training and verification behind the scenes.

It's how companies such as Expensify were able to handle their receipt OCR gracefully for such a wide variety of invoices and receipts.

I'd err on the side of saying this is a more palpable sales approach to selling it as AI. Letting your customers know that while AI is running the show, things are being monitored closely and fixed by real people when misclassifications arise.

Ive been running the underlying nonprofit (infino.me) for about 3 years with this ambition to study gene-exercise-nutrition interactions, with very little interest in making any money at it. But this thought of having a paid service for people who don't want to go through the asspain of logging .... this does have me intrigued. Everyone tells me they want an effortless meal logging app ... and photo snaps might be the closest we can get short of having something like google glass persistently monitor intake.

I wouldn't pay $5 a day. I would pay $30 a month. Mechanical turk, with a note that the picture needs to contain some sort of scale (and a request to manually scale the item if you didn't include one and it's not obvious), would probably take you pretty far.

I remember there was an app like that a couple of years ago, that didn't really take off. Don't remember the name or story.

I agree with your suggestions for human assessment because of problem posed by food calories varying significantly due to different preparations.

Take for example soda, if I gave you a picture of soda in a glass could you tell it was diet or regular? You might scoff at such an edge case but it quickly becomes more common when looking into food perpetration techniques. This is why caloric estimation is a really difficult and causes restaurants to not list their calories as the calories of a meal do not equal the sum of it's parts.

All these solutions are common as people want a signal to tell them to stop eating, but these are insufficient as people will simply ignore it due to hunger cravings (as happens on diets). Any nutritionist service in addition to detailing calories would need to incentive the patient to recognize the need to lose weight or setup a helpline.

How is that " the calories of a meal do not equal the sum of it's parts"?

That seems to violate the laws of thermodynamics.

Cooking changes the caloric content of food. Not all of every ingredient gets all the way to the diner plate.

Biology is not thermodynamics. Humans are not perfect combustion engines.

He may be talking about cooking. This could involve chemical reactions, or facilitate / slow the absorption of nutrients, therefore changing the effective caloric intake from various ingredients.

Thermodynamics should add up once you account for heating, cooling, evaporation, enzymes, waste, etc

> causes restaurants to not list their calories as the calories of a meal do not equal the sum of it's parts.

In my area, restaurants list the calories for each menu item, exactly to the extent required by law (chain restaurants are required to public calorie counts), plus some promotional "under 500 calories" or what-have-you for diet-targeting places.

It would be difficult to account properly for ingredients such as butter and oil which are hard to see in a finished dish and drastically affect the calories.

But people would probably still pay as long as it's a good faith attempt, since they can't necessarily do better themselves, so it would be just as accurate and save time.

And with geolocation, you could start to figure out where people are and then you'd know exactly the calories if they're at a chain restaurant.

Yup! The app as it stands does geolocation and factors this into account for nutrition lookups, but I could certainly do this automatically without requiring a check in.

Impossible. How can someone tell whether you cooked with butter and oil, for instance?

Sending notes with the image maybe? Combined with a clear estimate-based response it might be okay.

For me I would be ok with knowing weather the meal was in the 500 or 800 range.

Built using those newfangled convolutional neural network things. Trained caffe on a set of about a million food related images. Can't believe how accurate the classifier is...

UI needs work. Would love feedback on how to make food logging more intuitive.

Any info on the network itself (number of layers, input resolution, pooling, dropout etc)?

Stock Cafenet, trained from scratch.

why caffe ? why not something like h2o or tensorflow ? just curious.

because Im a newb. Looking forward to expand my repitoire!

Is not Caffe actually GoogLeNet? So technically it is Google architecture.

You could probably outsource enough of the UI work to Fiverr to make a really decent looking app.

Things like Main screen, App Icon and colour palette.

My color palette was carefully engineered to induce vomiting and weight loss! :p

Laughed for a disconcertingly long time at this.

Wish you luck with the app, really cool idea!

Your app kinda sucks now, but the detection is great. You could also do food identification as a service...

I'm trying to use fatsecret/myfitnesspal for counting calories. Something like this could make it much easier!

To me, a better workflow would be: take pictures of everything you eat and classify later (at night, maybe on the PC). Of course, this software would simplify the classification, the user would only fix any errors and adjust sizes.

This workflow would be well suited to your app, which operates on a server? (my understanding).

Congrats, I understand the relevance of AI for simple apps now ;)

Ive been thinking of that 'classify later' workflow. I think of it kind of like facebooks photo tagging, and it might even be better as web app that you get email reminders for to complete entries.

It would also work nicely for background geolocation services, since if the photograph was taken near a restaurant the service could extrapolate those things.

Anyways, I might make this project opensource and build a nice community around it. Depends on whether or not that model would best support my research goals.

Great! And think about using the new data from the users to improve the system (that should probably be in your terms).

How well does it work on plates with multiple foods "kinda mixed"?

Thats another huge design challenge Ive faced, specifically because it really complicates the data model and UI.

Implicitly, since the app presents a list of possible identified foods, and the user can easily check multiple boxes, thats straightforward, but then having the app pull each of those items AND ask the user to estimate portions for each .... gets a little messy.

So I put that complexity off for v2. That use case in particular would also be best off for a web-interface, I think.

What about dividing the whole plate weight by the foods detected? (in Brazil there are many restaurants where you pay by weight, thats why I'm suggesting..)

Theres a company in the bay area trying this approach actually! They look really interesting: https://getsmartplate.com/

Cant compete with that awesomesauce!

I would love a blog post about that!

How much experience with machine learning did you have before you started this project?

Cut my teeth on the netflix prize several years back. Took ML in my graduate coursework, and have dabbled with it on and off since. I love the whole theory of matrix factorization and its myriad uses.

Very cool. What do you think helped you most? Would you recommend grad school?

Where'd you get your images? Food-101 plus something else?

We're working on a similar component for our commercial behavioral-economics-driven app suite, targeted toward patients with chronic diseases.

Do you use location as a way to filter the set of possible foods, as in Google's im2calories project/paper last year?

They do some pretty awesome depth calculation stuff too: https://www.google.com/?ion=1&espv=2#q=im2calories+type:pdf

Edit: oops, should have read further. I see that you pulled appropriate terms from WordNet and used ImageNet to gather training images from Flickr. Cool!

Someone should combine this AI with something like Google glass (so you don't have to pull out your phone), a basic calorie/nutrition filter, and a shock bracelet[1].

That way when I look at a Krispy Kreme donut I get immediate feedback!

[1]: http://pavlok.com/hello.php

I absolutely LOVE this idea .... :)

google glass is dead though right ?

In some niche areas it's really useful. Some people I know recently raised 17M to build software for surgeons. Looking forward to seeing more uses like this.

It changes month to month.

I'm dying here. Sign me up for your startup.

Let's call it: "American Dystopia"

There's a trump joke in there somewhere.

Electrification was called "the greatest engineering achievement of the 20th Century" by the National Academy of Engineering. - https://en.wikipedia.org/wiki/Electrification

With today's technology we can do it again.

I've been thinking of this exact thing for a few years now, good to see others are interested too. I think there is HUGE potential for an app that can accurately (within a reasonable error range) count macro-nutrients through pictures of food, but I think that is WAY more difficult than you would imagine. How can you train a classifier to tell how much butter is in a dish? How can you account for food that is literally underneath other food?

Maybe a combination of spectrometer, weight, and photos would be a future solution. I certainly don't have an idea of how it would be implemented, but I think using data from all 3 of those sources could be a good step.

Another angle is post-consumption. Ive always wondered if precisely monitoring body temperature might tell you something about consumption, since there is this 'thermic effect of food'. Would be hard to get right though ...

In the lab, we can use doubly-labelled water to precisely measure calorie intake and burn, but its so damned expensive. If only it were commercially viable.

About post-consumption could a turd-shot help to compute the calories in/out ?

That could be called hungryPOT ...

ah, good ole toilet humor

This has HUGE potential. Some of the comments here disregard the fact that AI can do a better job than humans. An expert chef looking at dishes will make an educated guess on the ingredients and quantities that went into a dish and will have a better idea of the caloric content. The AI could learn a similar skill, identifying whether a dish contains pastry, buttery sauces, fatty meat etc etc.

You have a business opportunity here if you train that on fresh vegetables. Many people have trouble with these issues:

1. Knowing what the hell a (example) rhubarb is or looks like.

2. Knowing if it looks fresh or about to go bad.

Being able to aim a camera slowly across an isle with it getting highlighted would solve first problem. Spinning vegetable around in front of the camera would could solve second. So, try that and make it a paid app.

You know, its funny. being a former fat guy, Im not entirely familiar with all of my vegetables. I took this app to a farmers market recently and it was actually helping me identify a few things ...

Had never seen a beet before un-cut. Ashamed of my ignorance ;)

I live in a semi-rural area. Gives me a bit more experience. I still don't know what all these things are in the supermarket, though. So, I can't judge. ;)

That said, you should try training it on different kinds of produce that look very similar. One example I recall is Zucchini vs Cucumber as the stem at one side is the giveaway. Another is Nappa Cabbage vs Romaine Lettuce. Only two examples I could think of off top of head that would confuse people.

Pretty nifty, but please consider allowing creation of an account with a simple email address, or even anonymously. I really don't want to have to connect my Facebook or other social-networking information to your app in order to use it.

Thanks for allowing one to try it out without logging in via another service; it'd be great to be able to actually have an account.

Email accounts are definitely on the long todo list.

This is not compatible with my device which is an LeTV X500, running Android 5.0.2. Is there any reason that it might not work? I'm in the UK, if that might be the reason.

Im new to cordova (the toolkit I used to build the app), and I did notice that the number of supported devices is kind of low for my build. I may have included a plugin that narrows down the pool. I need to dig into this.

Sorry about this!

This happening for me as well with a Moto X Play running Android 6.0.2

Looking forward to trying the app once I can install it though, looks neat

Found my bug! Try again. Way more droids are supported now.

This is happening for me too. Samsung Galaxy S3 on US ATT running Android 4.4.4 Cyanogenmod 11.

I think I resolved this problem. Try again!

It's installed! Good job. :) I'll try it out at my next meal and let you know how it goes.

Hey Subcomos, great work! We've been working on something pretty similar, an app called Logameal (only available in the UK). We have run into (are running into) a lot of those UI issues too and would be glad to help. Let me know if you'd like to chat.

Your UI looks great!

I remember the latest Android camera app being able to extrapolate distances in the app (thus simulating photos with large apertures/shallow depth of field).

I wonder if you could use this visual distance algorithm to somehow judge portion sizes/plate sizes?

Haven't thought of that! What I have considered is letting users put something like a quarter or ID card in the field of view for that purpose.

Aside from that though, I figured Id do portion estimation down the road should this preliminary version take off.

Its not that bad to let the user estimate it too.

Good thinking. Assuming you don't have the portion size data, you may be able to run your data set through Mechanical Turk or hire an oDesk worker, and you could build a training set..

Nice work! Another big user group of stuff like this is diabetics who need carb estimates to adjust insulin doses.

I earned my PhD in diabetes research at the Mayo clinic. My whole goal for this project is to cure that damned illness.

Aim high ...

Isn't it strange that in [1] the beginning of the "Normal" BMI has more risk of death than the beginning of the "Obese"? I'd even venture to say that the "Overweight" range has, in average, lower Mortality Risk than the "Normal".

PS, another plus for the Metric System. The Imperial System form includes Feet+Inches, while the metric one only needs Centimeters for the height.

[1] https://www.infino.me/join/

Its been contested in a few papers now that overweight people can outlive normal weight, according to these BMI subcategories. But I am not surprised that things can be fuzzy, since ultimately its a continuum.

The finding that people with an "Overweight" BMI have a lower rate of all-cause mortality is consistent and has been called the "obesity paradox." A very interesting study came out this month that addressed it.[0] Basically, they did a prospective study of 55,000 (mostly elderly women) where they measured their body fat percentage directly using DEXA scans. It turns out that when you control for body fat percentage, it's actually better for all cause mortality to have the highest BMI category and the lowest body fat percentage. Of course, the measure "BMI after controlling for body fat percentage" is kind of unintuitive, since BMI is supposed to be a crappy proxy for body fat percentage. But basically, as long as your body fat percentage is low, being bigger is healthier than being smaller.

0: http://annals.org/article.aspx?articleid=2499472 (this paper used to be available, because I read the whole thing, but I guess it's paywalled now)

Thanks for posting this! I wonder if part of this paradox is also explainable because people with high muscle mass will tip towards the high end of the scale.

Very cool!

BMI is a terrible metric - it's just a function of weight and height and a terrible indicator for health - most professional atheletes would have high BMIs due to a high muscle to fat ratio (muscle tissue being denser than fat tissue). It's not abnormal to be "overweight" - "obese" at 10% body fat depending on what sport you're doing. (long distance atheletes are normally quite lean, sprinters are bulkier)

It's really not a terrible indicator for health despite this argument getting trotted out every time the subject of BMI comes up on the internet. Obviously, as you pointed out, there's individuals for whom BMI is inaccurate but they represent a very small portion of the whole population. At a population level, it tracks very nicely with health outcomes.

If you're a powerlifter or a sprinter, don't change your habits just because your BMI is high. On the flip side, if you sit behind a desk all day, don't blame a high BMI on the metric being inaccurate.

The metric is very convenient and a useful rule of thumb which is therefore perhaps useful for some proportion of the general population: especially for those with limited medical knowledge and/or equipment - but it's really not a niche minority who is unaccounted for by the BMI.

Muscle density and bone density, amongst other factors, will vary by ethnicity enough that researchers are discussing the value of one singular set of windows for the BMI[0], and although I don't have evidence I would also say that it really doesn't take much exercise for a person to skew their BMI by a noticeable amount - I would venture the top 30% of all exercisers would see their BMI affected by their habits.

So while there is value in the BMI for being easy to collect its data and make recommendations based on its value, in this day and age you'd really expect a more granular and precise measurement for use in research and disseminating health information.

It shouldn't be such a big leap for smartwatches / fitbits to measure body fat %, for example.

[0]http://www.ncbi.nlm.nih.gov/pubmed/19221673?dopt=Citation

The low end of normal includes people with eating disorder (anorexia used to have very high mortality rates) and some people with drug addiction.

Note that for women the low end of normal is probably sub-optimal.

Most people in Europe measure their height in meters plus centimetres.

> PS, another plus for the Metric System. The Imperial System form includes Feet+Inches, while the metric one only needs Centimeters for the height.

Don't be absurd. There's no reason one couldn't choose to input inches only, or decimal feet, or metres, decimetres and centimetres, or decimal kilometres.

First thing I tried: logging /two/ Granny Smith apples manually.

I had do the entire process twice because there is no (obvious) way to change the serving size to 2.

If you click the 'no barcode' button to pull down a fatsecret nutrition record, there is a servings pulldown that is displayed. Its kinda hard to see. Working on the CSS

Please do blog about the technical details, if you could post a walkthrough of setting up and training the net it would be great!

Someday! Right now I need to put the fire out in my server room. Didn't quite prepare for HN traffic ...

That is very cool subcosmos! I am also very interested in both cooking+nutrition and AI (see for example my web site cookingspace.com). My contact information is in my HN profile - shoot me an email if you want to talk about AI, food technology and business, etc.

This should be in use at those self-registry machines at food markets. I imagine that you place the food item on the scanner and it will automatically register it or ask for human input when something unrecognized is found.

Would decrease cheating perhaps?

Where did you get your training data set? I'm trying to build a network to recognize faces from ID cards and match them to faces/selfies. Onenof the hard things has been to get a training data set.

Check out wordnet! I think they have face categories.

This sounds like it could be used for the wrong purpose for sure.. Are you working for some nosy employer?

Employer? No - we are automating paper applications for financial institutions. This includes a photo ID and a photo.

That is checked manually today.

Haha nice! I was assuming someone was going to do some web screening before hiring people.

That's very interesting indeed. Are you using any API?

Fatsecret API for nutritional information and Foursquare for checkins, but installed and trained the neural network on my AWS instance for the prediction stuff.

Is it possible to implement the myfitnesspal API in the way user can scan the food and it gets entered in their food log on MFP ?

I've been emailing myfitnesspal for years for access to their API .... emails go ignored.

I think something like this could get really good at estimating calories too. I'm saying within +\- 50 calories.

I've heard that Google is working on that problem. They have way more resources than my one-man part time nonprofit operation, so I look forward to how awesome their thing will be when they finish it.

Will probably opensource this, since I want to grow up a nice research community around quantified self fanatics.

The classification looks good, congrats!. Where did you get the million labeled images to train your classifier?

Wordnet! Got a list of URLs from there and then hit flikr with some python.

Is there a writeup about how you approached this? All I see are a couple of app store links.

Ive been meaning to make a blog for some time now, since my nonprofit also has some preliminary research results with the genetic basis of obesity. Just havent had the time really. It will happen though. These neural networks are amazing and worth sharing.

This is great. Where are you hosting your caffe model? Just AWS with API Gateway?

Its on the instance right now, but I could certainly post it to github at some point. I think I need to retrain it though and swap a few categories out. Theres a few animals in there ;)

Also, been needing to train on non-food items to weed out abuse.

How much hardware and how much time was used in training?

Took a single AWS instance about a month. Not too bad.

How does it measure the glucose content of that sauce?

Probably gets it from FatSecret?

It does! They've been gracious enough to let me use their data since I'm running a nonprofit experiment.

Another app that is similar to this is called tuingle http://tuingle.com

where did you get the annotated training data ?

wordnet: https://wordnet.princeton.edu/wordnet/

I can't find anything related to images... Could you expand on this?

Oh dear, I meant Imagenet! http://image-net.org/

Awesome! Thanks! :D

halonewstza.blogspot.com

Reply on news.ycombinator.com