Show HN: Nutrient insights through your grocery receipts | Hacker News Reader

Show HN: Nutrient insights through your grocery receipts

nutri.adrianstobbe.com

162 pointselchead81 comments

Nutri is still in beta and the GPT-powered results are sometimes inaccurate. The nutrient information accuracy is good to get an overview, but there are still outliers at times. I'm looking to improve the accuracy through food databases. Furthermore, I'd like to add additional tips for combining / preparing food to improve its nutritional value. For example, iron absorption is improved through vitamin C, so combine chickpeas or leafy greens with lemon. Or combine beans with rice to get all amino acids.

On the UX side, I'd like to integrate a QR code on the desktop version to easily upload receipts through the phone. Furthermore, it would be great to have analytics over weeks on nutrient improvements over time. Nutri could also be a great accountability partner to track items high in sugar / processed foods.

What do you think?

Open articleView on HN

This is cool. As you said, my recommendation would be to not use ChatGPT to get the nutrition data, but use one of the databases that have good APIs such as USDA's Food Data Central [1]. This is domain where there's structured data and it's important that the output is deterministic, so a LLM is not a good choice.

I once started working on something similar, but the goal was to suggest a list of ingredients for me to buy for the week so that I had a good total nutritional values. I don't work on it anymore though.

1: https://fdc.nal.usda.gov/index.html

I'd second this, I just gave it a try and many of the results in the receipt breakdown did not seem to align with the nutritional information available from other sources.

Also, if it's going to give protein advice it should probably recognise meat - unless it's aimed solely at vegetarians & vegans. My receipt had a whole chicken, a lamb shoulder, and a couple of different kinds of fish - but it said I was getting 0 protein and recommended quinoa.

Haha, love its passive-aggressiveness. Lowkey pushing people to plant-based diets :D

This is how we eventually get Soylent Green.

“Fun tip, you have plenty of protein right at home!”

> I once started working on something similar, but the goal was to suggest a list of ingredients for me to buy for the week so that I had a good total nutritional values. I don't work on it anymore though.

Cronometer has a feature that does this – it checks to see where you aren't hitting your daily targets for macro/micronutrients and then suggests foods that would allow you to hit those targets. Quite nifty.

Yes, that's what I ended up using, too! The goal for my project was just to build something as a side project (I was experimenting with Deno and Fresh at that time), not necessarily because I couldn't find an existing alternative.

I tracked my intake for 1-2 weeks with Cronometer and it helped me see what kind of minerals and vitamins I might be missing. I mean, these things are "ballpark" only anyways, and I wouldn't have the discipline to track my every meal. But good to get a general idea - kind of like a check-up, just doing it from time to time.

I had this database in mind too, thanks! Btw Bard has deterministic output as I learned

I had the same goal and developed FoodCoach (TheFoodCoach.app). It tracks nutrition (not calories) based on a picture of a grocery store receipt.

The high level goals are to increase protein, increase fiber and decrease sugar. Super simple. Tracking takes <5 minutes per week, for the entire family. With the receipt it recommends foods to buy more or buy less to help the user hit their protein, fiber and (low) sugar goals.

I used no-code Appgyver and have been very disappointed in its robustness. The backend is all hooked up. Everything works, but I've learned I'm not good at marketing. There many better (and free) sources of information to obtain nutrition data.

This is pretty cool! I built a prototype of something similar last year: starting with your family's macros and meals, it fetches recipes and a list of ingredients that fit. It's structured as a traditional optimization problem so you can, for example, minimize the length of your shopping list or meal prep time. https://planmyrecipes.com/

This is very cool! Some visual feedback on the website: the vertical padding between the images and text looks a little tight/off, at least on mobile. I think tweaking aspects like that would make it look substantially more “legit”.

Thank you very much for the feedback - I'll take a look! I'm planning on adding to the tool this year, so I'll fold this in!

Sounds excellent. It would also be super useful for people with IBS/FODMAP like dietary issues. It could show you what ingredients may be causing flare ups (fructans and galacto-oligosaccharides and polyols)

Good idea! Do you know people with this condition who would want to exchange about it?

the r/FODMAPS sub is pretty active.

The main issue I see though is that if you're going to include processed foods, things will get complicated.

The names for various ingredients change in some countries ie: Sorbitol is also known as glucitol, D-Sorbitol, Sorbogem and Sorbo .

I use this example because Sorbitol is a sugar alcohol that affect FODMAP types.

You might do well having a chat with the Monash University, they have done a lot of work on this and have even published an app for people to check and see what sensitivities particular foods might have.

https://www.monashfodmap.com/

That database might be crucial for your site.

I was going to build something similar but now that you’ve already done the first part, maybe you could run with this idea - can you get GPT to generate small stories where characters eat the stuff in the grocery list and gain superpowers/are able to do big things?

My kid loves superheroes but every meal has to be accompanied with a story about why that day’s food item is good for him. I wanted to use GPT for this. Start by reading grocery bills, then have GPT understand the bill and come up with stories since it likely knows what health benefit each item in the bill confers.

That is a strange request for the OP. I doubt they're going that direction.

Assuming the number of meals you make is limited, you can use ChatGPT directly for this. Just ask it to make stories for your meals.

Cool idea. I've actually just taken a pic to my plate, threw it at gpt4 and asked it to generate a story. It worked nicely.

As others already said you can do this out of the box with ChatGPT. Take a picture and it will do a good enough job with what it identifies.

As a PoC, ChatGPT is fine for this. A kid would expect a more polished product which he himself can operate. E.g., maybe the UI could show images of items in the grocery bill, show images of superheroes he wants to include in the story, text to audio to narrate the story, history to see previous stories if there is any particular story he really likes…

I’ve been surprised by how good ChatGPT is at this, I took a picture of squid ink and mushroom pasta, which I made last night, and identified the entire meal accurately, including the garnish on top. If you take a second to think about this, it’s actually incredible.

I can’t think of too many other meals with that palette.

Stories are also the only copyrightable part of recipes.

I don't think it would handle my January receipts well. 1: 50 lbs flour, 25lbs sugar 2: 12x 3oz parmesan cheese, 26x cat food (wet), 60lbs potatoes 3: 5x 50lbs layer pellets, 50lbs black oiled sunflour seeds, 50lbs whole corn, bail of straw 4: 1400lbs pig food 5: 20 gallons raw milk Two are from the feed store and one is from the dairy.

Not something I need, but it’s a really cool idea, especially for people struggling with nutrition.

One feature suggestion would be to accommodate specific diets (keto, vegan, etc.) and generally allow excluding food (e.g. I hate the taste of beet roots and eggplants, so I’d always exclude those).

One thing that’s not clear, is if it understands multiples, I uploaded a receipt [0] with 2 feta, 2 broccoli, and 3 arugula, but I’m not sure if it recognized that? I also don’t know how well that can even work without some calculation check, for this store (REWE) the amount is always below the product, for Aldi it’s the opposite https://i.imgur.com/YvqJCle.png

[0]: https://imgur.com/a/Xhw3Acs

[1]: https://i.imgur.com/YvqJCle.png

Thanks for the feedback and suggestions. I can't open the images but I get the problem. Will check on that

Super interesting project. I also started once a project to index food and their ingredients via gpt. The inaccuracy let me abandon the project. But never tried the new gpts for that.

One great resource is also: https://world.openfoodfacts.org/

This starts from a fundamental assumption that one person shops exclusively for only their own food. The vast majority of grocery shopping is done for a group of people (a couple, or a family). Those people aren’t eating an exact 50% even split of every item purchased.

There’s a “How many people do you shop for” button.

Yes but that assumes that you eat the exact same ratio and quantity as your 4 year old child, for example.

Sometimes it feels like my 4 year old eats more than me. The ratios are definitely different though.

That is an example. Correct. What’s the issue?

It isn’t telling you how many calories you consumed. It’s just insights into the nutrition from the food you consume.

And that's key. Calories are important, but most people will naturally eat less calories as they shift from carbs+sugar to foods with higher protein and fiber content.

In Germany round about 41% of people live alone. It's also easier to build. OTOT you probably also don't have a lot of families that optimize for these things, while single people do. I see a lot of good reasons to build this for single people first.

No need to be so negative about something that apparently doesn't apply to you.

This is a real problem. The good news is that if its not healthy for an individual, its not healthy for their family. A diet comprised of no fiber, no protein and high added sugar is always unhealthy. Families tend to eat similarly. Its very rare to have someone eat well and someone to eat unhealthily.

It also excludes going out to eat. That being said, if someone is concerned about their health enough to use an app like this, they SHOULD be eating at home more. They only way to know what is in your food is to make it yourself.

> Upload your grocery receipts for veggie tips to make your diet complete in nutrients!

Isn't that in reverse? How about you give them options to build a - balanced and highly nutritional - items for a shopping list?

I've had similar idea with supermarket receipts about 10 years ago, but had trouble with implementation — OCR APIs were very enterprise-oriented and unreliable. However, instead of nutrition, I was thinking more into automatic recipes, and re-ordering online, getting feedback about the stuff you bought to build your taste profile and recommendations.

If you have good data of all the households purchases, you can do so much good stuff with it, the problem is getting this data reliably.

Good point! What would be most valuable to you from this data? Recipe suggestions could be one thing, but I guess most people build their shopping list from recipes. Re-ordering is probably something that retailers will try to make easy.

This is awesome! I wanted to do something same but was going with the approach that people would be incentivized to enter/map nutritional information for line items on their receipt...and then implicitly contribute that to the community (like how MyFitnessPal works.)

Ultimately there arent that many grocery stores and there arent that many popular items, so you'd get an 80/20 approach.

Also curious about the GPT approach, are you worried about the cost exploding with a GPT approach?

I see, also similar to Yuka. That can indeed work well for an established user base. I'd focus on food databases now, but reporting nutrients for items as a user is interesting for the future! I was actually surprised about the GPT cost. The HN launch was about 3 dollars :) If you are interested about the future, I added a form to collect feedback and stay in touch with users :) https://forms.gle/AN7e85MwqehPTQoE6

The idea has merit. Aside from the demo itself, I'm more interested in the business opportunity.

You could partner with grocery stores. The user would have an account, not tied to any one particular grocer. It would track all of your grocery purchases, and like your demo, give you nutrient insights as well as recipes and maybe coupons and money-saving tips. Lots of stuff you could do with that.

There's a company already pursuing this (disclosure: my current employer built their site/app for them), but it starts with the diet definition and then creates shopping lists, with callout links to online grocers.

https://sifter.shop/

I was also thinking about monetization, but was more thinking of a SaaS for consumers. Do you happen to know the grocery industry to know more about their interests for the app? I mainly see it as a way for them to place ads in the app

If I buy a pound of sugar, doesn't it matter how long it takes me to use that sugar in order to have any relevant insights? If I buy a pound of sugar a week, that's a really different implication than buying a pound of sugar only once per year.

Averaging over a pre-defined time period smooths out variations like this.

Really cool. I checked also a receipt in Romanian and it worked, it translated the ingredients.

This fails with cash payments, e.g., at farmers' markets. If I were in the market, I'd prefer an app that analyzed pictures of my actual meals instead. Less work for me, and it represents my actual intake. Such an app may already exist.

It just can't be done for meal photos -- there's no way of knowing if the mashed potatoes have just a little milk, or two sticks of butter. Or if the tomato sauce is pumped full of sugar or not. Items that are visually identically can have vastly different nutritional profiles.

Not to mention trying to guess what's inside of those dumplings or ravioli...

Start with the picture and maybe a verbal description, presenting an estimate of the ingredients and allowing corrections.

I should stop talking because this is too much work for me either way. Cooking is enough work without all this!!

So if I go grocery shopping, I may purchase a few prepped meals which I eat in one sitting. Then I have a couple sticks of butter. I use one over the next six months, and the next one is stored for after that runs out. I've got some shelf-stable snacks that I use intermittently over the next 2 years. Stuff in my freezer for a couple of weeks.

Overall, my groceries are never consumed at the same rate, so how can this possibly track daily nutrition?

That's a fair point! I can add a feature to remove items from the receipt. The option to manually add items bought earlier already exists. It's not about daily nutrition, but to understand the overall nutrition over let's say a week.

Awesome!

Can you customize target intakes? Say, Food Pyramid / Food Wheel / etc.

And vitamin B12 is crucial...

Sure! If you want to have a quick chat about it, shoot me an email at hi@adrianstobbe.com Happy to see how I can adjust it

Cool!

Sounds cool. Does it understand only receipts of products available in the US ?

I just tried it with a screengrab of a past online order from a grocer outside the US and it seemed to work: https://imgur.com/a/cZ2Beo4

Works with my German receipt: https://imgur.com/a/Xhw3Acs

If it's a finetuned multimodal vision transformer, it's probably multilingual.

When I click "Get summary report" on the demo page, a "Generating report [...]" spinner shows up for a second and the nothing happens. I'm on latest Chrome on Ubuntu 23.10.

Am I missing something?

Sorry, the traffic exceeded the API quota. You can try again now :)

I’d love to see a version of this based on additives and processed ingredients. These are big contributors to obesity and most Americans aren’t even aware they’re consuming such unhealthy food

just try to limit factory food you would be fine. There is no need for apps and stuff.

people who are conscious about this dont need an app ( its not at all hard to tell food is bad) and ppl who don't care won't use the app.

so there is no market for this kind of an app.

I would have thought the same until I found yuka (https://yuka.io/en/) and saw that they make multi-millions per year.

This is amazing, what kind of OCR are you using for the receipts? Or are you doing something else?

Not OP but gpt is multimodal and does an okay enough job for something like this workflow. Alternatively all the big cloud players and ocr/ml products to read receipts.

You could stitch all of this together for this product in a day or two.

Thanks! Yeah, I attached an image to ChatGPT and it worked great.

I see a lot of examples that use receipts themselves like this, but one idea I had that's kind of similar would be to look at just the "ingredients" list on product labels and parse those, like these examples (under the nutrition labels):

Example 1: https://i.imgur.com/MqpL6yh.png Example 2: https://i.imgur.com/3FSK0CD.png

However, using things like pytesseract and Google's Cloud Vision API returns mixed results, sometimes missing things, transposing lines, etc.

Any ideas on what I could do to improve being able to extract ingredients lists from food labels? Would I have to start looking into something like Vertex AI and training custom models?

Then again, as I'm thinking out loud, I realized if these tools can extract all the text pretty reliably, the order and place doesn't really matter if you create some extractor that's able to just pluck out which words are actual "ingredients" based on some master list or something.

Yes I think using a flavor of the cloud providers document tooling is probably optimal. Gpt vision is great for general recognition but I found it to be hit or miss when you started throwing too much text at it in an image.

If you can get image working via vision that’s great. On the cloud ocr side, I know tooling like Textract is good enough to generally provide output as if you were reading left to right. So in theory the text should not be that transposed or fragmented and nutrition labels are standard enough that you can probably pull the portion you want. On top of that, like you allude to, LLMs are pretty good and figuring things out.

I was also playing with Cloud Vision API, and it was good but found the formatting not always easily usable. GPT vision API is great for general recognition

i think it's a great start, i wonder if one day people will use a combo of grocery receipts and pharmacy prescriptions receipts and then compare and offer suggestions on what products to cut out

I suggest you add restaurant receipts being scanned.

If they contain the meal name, you can do this :) Otherwise you can also manually add the meal name. I'm however not sure about the accuracy of entire meals yet

Hey, how can I contact you?

Hey, you can reach me at hi@adrianstobbe.com :)

whatever you do don't sell it to insurance companies...

Nor governments.

However, I do think the stores likely already have such data about you, if you take advantage of any of their offers to track you.

It's a little disturbing to see this slow normalisation of mass surveillance, one little thing at a time.

Hell, knowing what people actually eat and comparing it to their actual health would be extremely helpful, especially combining with a massive data-set and big data.

Food studies are currently based on recall... and peoples recall is horrible. "What did you eat last Monday for breakfast, and how many grams of it did you eat?" The inputs are garbage and the outputs are garbage. Linking actual food intake to healthout comes would be VERY powerful.

Does it work with European receipts?

See example

The co-op that I belong to actually emails me PDFs of receipts of the groceries that I buy there (to reduce waste). It’d be really neat if this could accept PDF receipts as an input option.

That's FoodCoach

Reply on news.ycombinator.com