AI Restaurant Menu with RAG
wandb.ai
wandb.ai
Sigh. Am I the only one who likes to hold a physical menu and browse it? I never go into a restaurant knowing what I’m gonna get… I like to see how the menu is themed and decide from there. The UI on a physical menu is a vast improvement over any digital menu I’ve seen.
Dining out is a human experience, and I deeply resent any systems that demand my attention be funneled away from my dining companions and staff, into a wholly inadequate digital kludge. There should never be a reason for an electronic device to be at the dinner table.
QR-code-based menus are absolutely and uniformly unfit for purpose. It’s a question of information and navigation: it’s intuitive and natural to navigate the physical layout of a printed menu, associated as it is with the stages of the meal and complete detail of all options at once. As you point out, I’m going into the menu not with the intention of seeking a specific taste I have in mind, but with the intention of hearing the full range of the restaurant’s proposition and recognizing what resonates with me. It’s a recognition task more than a retrieval task.
Compare to tapping and swiping through tabs full of pages of digital listings to remember what you meant to order when the time comes.
I admire the creativity and technical effort involved with this project, but I can’t imagine anything worse than having to play a guessing game with an LLM to hide-and-seek what the kitchen serves.
I feel like the whole enterprise understands the value of hospitality differently than I do: a human hospitality professional can apply the full range of their judgment, context, intuition, and humanity to something like this type of menu recommendation. And a skilled menu designer can provide an efficient survey of what’s on order and set the tone. Even in its ideal implementation, what problem does this AI-ification actually attempt to solve or add value to?
I can’t imagine how you’d factor AI into that though. Ultimately I generally know what I want, and writing a story about it isn’t going to help.
- the tablet is provided by the restaurant and knows your table number, etc - doesn't require registration or personal information - isn't used for payment
The only reason online menus are taking off is cost reduction and the fact that they can change the menu more easily (and the prices of course)
The menu is a huge portion of setting the ambience and theme of a nice restaurant.
At the risk of being extra curmudgeonly, have you ever dined someplace where they wheel a dessert cart round? There’s kind of a lightweight vicarious thrill you get just from eyeing the physical tantalizing plates, even if you don’t end up eating one.
Compare that to “here’s an empty text box. Whisper your very specific wishes to the genie and maybe it’ll materialize something in the neighborhood of your demand, probably not exactly what you asked for; and it’ll hide all the cool stuff we do that isn’t directly relevant to the demand we made you work to formulate (and probably can’t fill exactly).”
Based on my dinner earlier, a good example would be if you could take “who has that Ethiopian dish with chicken and bits of bread” and return a place listing only “Doro fitfit”, which would require more effort with a traditional search engine since none of those words are commonly present in the menu and you have to have more than basic Solr-style synonym expansion to expand “Doro” to “Doro wat”, which itself expands to “chicken stew”, and expand “fitfit” to “shredded injera”, which expands to “a type of bread”. Obviously that _could_ be solved with the traditional stack but that starts to get more involved so I could see an advantage for making it smoother.
The majority of humans aren’t adventurous, and prefer convenience and consistency, and not having to think.
Other thing is, when the menu isn’t available, there’s a decent chance I won’t even go there when I’m craving something specific. Like — I’m craving wings, but not every pub near me has it, so I do a very quick check before I go there.
But, to your point, the nuance that some chefs place in their food descriptions tells a lot. Only a human can tell you which pairings are important to a particular dish, a machine can't. A bit of both would be nice.
If you’re at a restaurant and it is uncertain, the answer is likely no, purely based on the scale that restaurants require.
In Malaysia, it’s common for the menu to be entirely digital - you scan a qr code, and you’re then presented with the menu, from which you then order and often pay directly on your phone. Some places remember who you are and show your previously ordered dishes at the top of the menu.
Unfortunately, I think a lot of restaurants drew the conclusion that customers don’t want online ordering rather than that customers want fast and smooth ordering. I suspect some of that was also unrealistic expectations from the ad-tech industrial complex promising additional revenue from things customers don’t want, too, based on how aggressively some of those pushed you to create accounts and allow tracking before you could order anything. Someone not blinded by greed would make the pitch to do that kind of thing after ordering when you’re not trying to do something else and have an idea about whether you’ll even want to come back.
Additionally, a full vector store is overkill for a menu with <100 items. You can just keep a numpy array in memory with vectors for that and use a matmul for RAG.
... what? Tell me more.
This would be much harder with basic vector database+RAG. You would at least need HyDE (hypothetical document embeddings) to act like a real waiter.
I find your project very interesting though, and the walkthrough is nice!
Imagine the generated documents based on that query was like “Our shepard’s pie is the perfect spicy meal for sharing with friends without being filling” (Obviously this is wrong, but it’s an example of a statistically possible document)
Then the following similarly search would also be searching for shepherds pie.
I don’t think throwing more LLMs at rag problems is the answer.
As a user I sometimes want a "buffalo chicken quesadilla" not disparate buffalo wings and quesadilla recommendations.
I believe Elastic has a mechanism that supports both? Haven't tested it yet.
Curse you! I just started eating dinner, but now I want what you listed instead.
Roasted buffalo chicken thighs, smoked gouda, and sautéed onions, bell pepper (and jalapeño or other peor is optional, there's a smoked canned pepper -- I cannot remember the name -- that could be especially good here).
This is pretty straight forward:
1. Place chicken thighs (bone in, skin on) on a roasting sheet, liberally salt. Roast in an oven at 400F for 30 to 40 minutes (start checking at the 30 minute mark).
2. While the chicken roasts, dice an onion, a red bell pepper, and a green bell pepper and sautee in oil of your choice (bonus points for using the drippings from the chicken thighs). Don't cook them all the way through.
3. Cook down canned black beans with a bay leaf, 1/4c of fat of your choice, salt (to taste), cumin (to taste). They're done about the time this concoction resembles a spreadable paste, about 10 minutes over medium heat.
4. Shred the thighs in a bowl, consume or set aside the skin. Add your buffalo sauce if choice. Bonus points if you made your own Buffalo sauce.
5. Heat a cast iron skillet over medium high heat, or a George foreman style press. I'm assuming these are non stick, add enough oil that there is a fine sheen. We don't want to fry the quesadillas (maybe next recipe).
6. Assemble your quesadillas directly in the pan, in this order: cheese (shredded or slice thin is fine), veg, meat, beans, cheese other tortilla. The last layer of cheese is important and will help glue the thing shut. If you're using a skillet, put a lid on it to tap heat and encourage the top cheese to start melting.
7. When the cheese on the bottom has melted and the tortilla is starting to brown, we flip. Get a plate and slide the quesadilla onto it. Then take the skillet and put it on top and then rotate both. If you're using the press, get a second plate and flip.
8. Wait for the new bottom side to brown and melt. Finish with cilantro, lime juice, and something approaching queso fresco. Greek yogurt, sour cream, goat cheese, anything like that will do.
Enjoy!
now I also want Buffalo chicken quesadillas
[edit] Chipotle! It only took an hour to remember.
I prefer the approach of standard french restaurant: 2-3 options for each category of appetizer, main course and desert. Don't paralyze me with choices. Too many options also mean the establishment has no focus - they won't be all good so it's a lottery.
If I have question I will ask the waiter. And I am as asocial and introverted as any common nerd.
Just use an picture of the menu as part of the prompt and you will make everyone's lives easier. I'm skeptical that asking questions about the menu has much real utility.
But LLMs and RAG in general couldn't be further from crypto in terms of utility.