Show HN: Bitesnap – Deep Learning Meets Food Logging
getbitesnap.com
getbitesnap.com
How does it handle differentiating different types of bread, which have differing carbs?
How does it handle a thick layer of butter or another fat put on the sandwich in the Avocado Toast example, which would presumably be below the visible avocado?
A long time ago my friends and I offered a manual version of this as a service via sending pics / emails to us and us then manually going through and guessing. It worked well enough, so I have high hopes for a ML version!
My biggest pain point doing it manually came from pics of things like pasta where I couldn't really guess how much oil was in the sauce.
You can definitely get far with just estimating the macronutrients from a photo, and the absolute accuracy matters less than consistency in measurements over time.
We don’t nail everything yet but we allow users to refine the predictions. So in your example we might predict bread and let the user pick the type.
> How does it handle a thick layer of butter or another fat put on the sandwich in the Avocado Toast example, which would presumably be below the visible avocado?
We don’t predict portion sizes yet. At the moment we give a sane default and ask the users to adjust it. The next time you eat the dish we bring back the past meal so you don't have to specify the details again. We’re hoping to start predicting some of those details once we get enough data from our users.
Thank you for trying it out and the feedback.
This is a serious problem. Research suggests that one of the main causes of obesity in children is lack of ability to identify portion sizes or understand how much to eat.
Obviously there is a market of people who understand this well and want to track what they eat, but you are very likely going to be misleading a very significant amount of your userbase into making worse decisions for themselves.
this is one of those things that i believe you should have right from the beginning.
Would be cool if I could pull one out from my pocket, stick it under my plate, get a measurement then subtract whatever is left after I'm done.
1. Its not totally clear to me what the goal of the app is. Is it going to help me lose weight? Help me avoid unhealthy foods? Why am I tracking? Do I get to choose why I am tracking? Tracking is a big commitment, so I would lead more with what the benefit is, to motivate me to decide to track.
2. I really love the weight watchers approach of boiling everything down to a single point count. I have been around WW long enough to see them change the meaning of the points to incentivize different behaviors. For example, raw fruits and vegetables are generally zero points, even though they clearly have calories. High sugar foods are higher in points than their calories would suggest. I find a point system much more useful than a calorie system.
Overall, if your goal is to help people lose weight, I'd suggest you look at what WW has been doing in their app, and also in how they have changed their point system over the years. I actually think WW overall (including the meetings) is an amazing system.
Interestingly, I have gotten to the point that I basically know the points of everything I eat regularly. Originally, I loved the WW app because it was so comprehensive, but now I just use a tiny notebook and pen. Its a lot faster than messing with the app.
Our goal at the moment is to focus on making the logging experience as simple as possible. Weight loss is one of the main use cases but we have a few beta users who are logging for health reasons, trying to improve their diet and even a chef who’s doing it for fun.
We’d like to make the app customizable enough to fit most of those use cases. We don’t want to push calorie counting on everyone and have an option in there to disable the calorie and macro cards. As we add new features we’ll let users decide if they want them to appear in their feed.
We’re considering adding a simpler point system, maybe even one that adjusts based on your goals.
All of our nutrition data comes from the USDA right now. We don't recognize everything that’s in there yet but we map our predictions onto some of the “nodes” and let people refine the predictions to a more specific item.
We don’t recognize packaged products yet but plan on doing it once we have enough data. Barcode scanning is almost done and should make it into the app soon.
Which is why I'm sticking with MyFitnessPal. Also, I find that although it's tedious to keep count of calories in the beginning, once you get used to it, it becomes a game, and even fun.
For my situation, it's more about "damn, I ate 1000 calories over last week, oh wow lol it's because I got super stoned on friday and ate half a pizza, ok, so next week eat 2 eggs instead of 3 for breakfast to make up for it." This app definitely wouldn't work for a cut, though, because I need my measuring cups and spoons to do that right.
With Bitesnap, you can enter exact cups, ounces, etc. if you want to refine your calorie estimate. One of our goals was to build a flexible tool where it would be quick to get a ballpark number, but also possible to get a very precise number if you put in just a bit more effort.
We think that the visual side of things will be useful to many people, even with ballpark calorie estimates. It's a great way of developing mindfulness of what you’re eating and improving and maintaining your diet.
Also, as we have been working on this it’s been a really fun game to see what we can recognize!
My problem is, taking pictures is more effort than picking an item off a list, as current caloric counters do.
Most people eat roughly the same things on a regular basis, so they'll end up ticking away a meal before/after you have it and be done with it, with this, you'd be taking pictures while having your meal.
another use case is planning a day ahead, again, pics don't work here, can't take pics ahead of time.
And of course, the result from the pics have to be corrected, so the app learns, it seems easier to just get the item from a list immediately, without having to take a pic first, auto completion on an input works wonders (though of course you'd pick from a longer list)
Maybe i'm just old or not enough of a techie, or photographer, but for me typing a short text, even on a phone, is actually faster than taking a pic.
How can this be? Taking a picture is at most 2 taps, if it pre-fills 90% of your list (even 40%) it has saved you numerous taps.
Is there a way to save prior entries as meals? I am a boring person and I eat the same thing for breakfast 7 days a week. I would like to just add this with one click instead of selecting: Eggs... Spinach... Oatmeal... etc. MyFitnessPal has this and it is a great time saver.
One neat thing is that our model has figured out which features are relevant to this task -- so tomorrow, even if you eat eggs, spinach, and oatmeal in a different container or at a different place than you did today, we can still recognize that it's the same thing.
You can see this in action in the first shot of our demo video:
We have a blog post up explaining more about Bitesnap and why we built it. https://blog.getbitesnap.com/introducing-bitesnap-a-smart-ph...
I'd be more than willing to pay a monthly fee ($5 / month) for you to have someone confirm the details of my meals and label meals that your system doesn't recognize. I'd be happily paying you to build a higher quality training set because
1) I don't want to fill out the extra information (although your interface makes make that process less painful than it would be otherwise)
2) Paying would make me much more likely to be a consistent user
Anyway, excited to try it out and if you every try a paid upgrade I'll definitely be a guinea pig :)
I am about to get my SCiO unit which provides a means of sampling small amounts of food to determine the nutrition facts. The minor issue here is that it doesn't provide much in so far as what the total amount of carbs is, only the carb density.
I could see this product working alongside a SCiO type device that can get the macro assessment of of food you're going to be eating, but then get the nitty details by hooking into the SCiO data on the spot. If bread is detected, "Please get more accurate details on your meal by sampling your bread with your SCiO-type unit".
Great stuff! Keep it up!
- Very slick onboarding experience, especially compared to other calorie counters. Big plus here.
- There doesn't appear to be a way to add food outside of the current "meal." I'm sitting here at lunch time, but wanted to add what I had for breakfast --- instead I've just eaten a very large lunch.
- The current database of foods seems pretty slim. No entry for my African Peanut Soup, for example, which is available in both LoseIt and MyFitnessPal.
- How will you deal with things like sandwiches, where many of the ingredients may be totally hidden from view? Guess "sandwich" and let me pick what's on it from a sensible list of sandwich ingredients? Same goes for soups, or stews, or anything that can be visually similar with a wide range of possible ingredients.
Over all a good start, and some much-needed innovation in the calorie-counting app space.
Glad to hear that you liked it
> There doesn't appear to be a way to add food outside of the current "meal." I'm sitting here at lunch time, but wanted to add what I had for breakfast --- instead I've just eaten a very large lunch.
Yeah we had a bunch of people asking for that the past few days. We should have that fixed in the next release.
> The current database of foods seems pretty slim. No entry for my African Peanut Soup, for example, which is available in both LoseIt and MyFitnessPal.
All of our data comes from the USDA right now. We’re going to add barcode scanning soon and that will include another 70K items. After that we plan on making it easier for users to add new things by OCRing the nutrition labels and computing the nutrition values from ingredients/recipes.
> How will you deal with things like sandwiches, where many of the ingredients may be totally hidden from view? Guess "sandwich" and let me pick what's on it from a sensible list of sandwich ingredients? Same goes for soups, or stews, or anything that can be visually similar with a wide range of possible ingredients.
For more complex items we have these “builders” that let you quickly adjust and add common ingredients to things like sandwiches, salads and soups. As we get more data we’ll use ingredient correlations and predictions to make the suggested additions more accurate.
The app also learns to recognize your past meals so you quickly copy the information for meals that you eat often.
Putting it out there: I would pay a lot of money for a consumer tech wearable or even implant that would track calorie consumption in the background.
Having this integrated into google glass or spectacles would be great.
1. Tag the location (if you go to McDonalds and take a picture of a Big Mac you'll see that you're at McDonalds and you have a picture).
2. Then, to get your "nutrition" info you have to manually specify what you're eating.
3. What you're eating would then be matched to a database that would provide the nutrition information. The picture basically would be there just to show you what you ate.
---
This looks WAY better than that.
About 10 years ago I created a cooking web site that also uses the USDA nutrition database (cookingspace.com) and I just recently started working on free iOS and Android apps that will use the improved analytics code that was originally used on my site.
I am playing with the Android version of your app right now - it so far has done a good job recognizing food items pulled out of our refrigerator. I also like that as I take a picture that it does not add the image to my local pictures (since these are automatically instantly backed up to OneDrive and GDrive).
I'd be curious about the calorie detection. I'm wondering if it's using some kind of weighted sum of image segmentation proportions, or doing end-to-end deep learning.
Anyway, cool product, love to see where it goes!
We haven't tried going directly from image to calories yet, and I'm not sure that we ever will. Instead the plan is to do end-to-end portion size prediction for some of the classes. Segmentation would be cool but it's really hard to get the data for it.
By the way, plotting images with matplotlib is a pain. Try using HTML with base64 encoded images instead. Something like this should work:
def base64image(path_or_image, prefix='data:image/jpeg;base64,'):
s = BytesIO()
get_pil_image(path_or_image).save(s, format='JPEG')
return prefix + base64.b64encode(s.getvalue()).decode('utf-8')
def show_images(paths_or_images, predictions=None, sz=200, urls=False):
from IPython.core.display import display, HTML
predictions = predictions if predictions is not None else []
img_tags = map(lambda p: '''
<div style="display: inline-block; margin: 2px; width: {sz}px; height: {sz}px; position: relative">
<img src="{b}"
style="max-height: 100%; max-width: 100%;
position: absolute; left: 50%; top: 50%; transform: translate(-50%, -50%);
border: {bsz}px solid rgba(255, 0, 0, {pred});"/>
</div>
'''.format(b=p[0] if urls else base64image(p[0]), pred=1 - p[1] if p[1] is not None else 0, sz=sz, bsz=5),
zip_longest(paths_or_images, predictions))
display(HTML('<div style="text-align: center">{}<div>'.format(''.join(img_tags))))For myself, and for many others, calorie-intake tracking was/is one of the last hurdles jumped before weight loss/maintenance efforts really achieve great effect. It's such a pain (time-consuming, tedious) to do it manually, especially if you have any reasonable amount of variety in your diet.
That's one of the main reasons why we ended up working on this. I was pretty overweight as a teenager and lost over 60lbs in one Summer by really paying attention to what I ate (and exercising). I tried using a few of the calorie counting apps but they felt like a chore and really nudged me to use packaged products since I could scan the barcode to log them.
One other area of feedback - while the onboarding was slick, I felt the hours of activity to the labeled "level" seemed a bit off. For example I do a high-intensity workout almost every day of the week for over an hour either: strength training or cardio and the level of activity for 7 hours per week only put me at "lightly active" (I forget the actual terminology and cannot restart the onboarding screens without uninstalling). I was just curious how you came up with the activity scale.
do they mean CNN's for image classification and/or recognition? Does the app estimate the distance and the portion size and if not, how feasible would that be?
Would it integrate via HealthKit perhaps?
I'm suspecting this is a concierge MVP of sorts...if not consider me quite impressed. The last time I checked food identification (or portion sizes really) was a fairly hard problem. Edit: Guess the food identification isn't that hard anymore. Yikes times are moving fast :D
https://github.com/Microsoft/CNTK/wiki/Object-Detection-usin...
I mentioned this here as well: https://www.reddit.com/r/MachineLearning/comments/5ol7od/d_w...
The benefit of pictures seems to be that I'm forced to think about what I am eating before chowing down.
1. I like the design, and it works very cleanly. Nothing appears to be heavy, and the UI is intuitive.
2. It only does one thing - track food. I think this is it's biggest strength. It doesn't do fitness or anything else right now, which it shouldn't.
3. The default goal to lose weight is simple to use and I think captures the predominant use case - by calorie reduction. I think the goal breakdown with consumed/remaining should be the most prominent UI element on top with the breakdown by calorie type being second. Having the challenge details (# meals X days) at the top isn't data I need each time I open the app. I can see why that would be a design challenge.
4. I have only used it a day so I can't say how well or not it displays trends about calories/nutrition over time, but I know that I would like to be able to break things down more.
5. The biggest challenge I think you have is with CV. Correct me if I am wrong, but my guess is that you are trying to use users to do reinforcement learning on your Deep Vision Nets. I am a Deep Vision guy myself (which is why I downloaded it by the way) and my guess is that you are going to have a hard time doing training this way. Here is why:
A. If the results of the object classification are good enough to always be result #1 (because it's obvious you are using a probabilistic return set (imagenet?)) then over time people will be annoyed at having to select the object/food in addition to taking the picture.
B. If the results are not good, then people will get annoyed with having to take the picture AND ALSO enter the food type. They will just resort to entering it manually each time. For example I made vegetable curry for dinner, and didn't take a photo because I knew it wouldn't know what it was.
So as a result, your training set will stagnate and won't learn any better than if you did it with a team of people. If you want it to really learn you're going to have to incentivize or force people to always take a picture and always tag it. Even better if you can have them bound each item right?!
By the way, crowd sourcing Machine Vision training is I think the right way to do things (that's what we do with interior home objects FYI).
I look forward to seeing iteration here. Best of luck.
We're not doing any reinforcement learning, we just fine tune the net as we get more data (and occasionally train from scratch when we add a lot of new classes).
In regards to (A), we plan to start skipping the selection steps for predictions that we're really confident in and will just add them by default. I think once we have enough data we might even be able to predict what users will eat before they take a picture. I eat practically the same thing for breakfast every day of the week so it could just log it for me without requiring me to do any work. Same goes for things like coffee shops, we don't ask for location right now but if you always get the same thing when you walk into a coffee shop, we could just log it for you based on the fact that you were there.
B. We keep track of our predictions and what users end up logging so we can tell what our weaknesses are. When we train new models we prioritize the weakly performing classes, especially if they're popular among our users.
Can you let me enter my height and weight in metric please as I had to use Google to convert.
We're working on supporting more countries, but localization can be difficult -- we have to worry about things like foods and preparation methods that are local to a region, etc.
I'd be interested in using this if I can get the data out. I've been posting everything I eat and drink to my own website for the past few years, sometimes with photos sometimes just text. I'd love to have a better workflow for doing that!
I noticed that the image of the phone with app on the homepage takes a few seconds to load on a mobile connection. Might want to optimize that for faster loading.
It was just clarafai -> keyword search in USDA database -> log though.
Looks cool!
It might help to add a mail subscribing list to your website to capture early oversea adopters.
Does the app allow you to take a picture of the aftermath to account for the unconsumed left overs ?
We don't predict portion sizes yet. We could try adding that once we do.
I'm guessing there is nothing specifically US centric in the app, any chance of opening up region support and sharing the love with your mates down under?
Cool service btw. I remember a company that did this with Mechanical Turk lol.
It looks damn cool, but I'm really sad to not being able to try it just because I don't have the right IP address :-(
https://goo.gl/forms/WQ2VOJwRsn9yWTfC3
We'll just add you to testflight or google play beta.
Meanwhile another app with the same name already exists and can be downloaded in Switzerland - http://bitesnap.appstor.io/ - thought this was the one; very buggy bad experience. Oh dear.
Can understand not wanting to launch in non-English speaking app stores, for the risk of attracting negative reviews. But still for this app, would it really hurt? The marketing right now is to the English speaking audience...
Frustration all around
edit: just saw https://news.ycombinator.com/item?id=13484472 - OK fair enough :-/
Only imperial units though :(