Google's AI doesn't understand restaurant menus
shkspr.mobi
shkspr.mobi
People have pointed out recently how Google search seems to struggle as sites on the internet turn more and more into apps rather than standardized documents and they just go and search on reddit. Having a standard to encode semantics seems honestly necessary at this point if you want to keep things interoperable.
You could probably recreate exactly that phenomenon today, however complicated by the fact that legitimate good-intentioned creators have to behave in much the same way as spammers.
Combine that with Google Analytics, they know how long you stayed there.
They also know if you went back, clicked on another and so on.
Google correctly realized early on that this idea that content creators would correctly tag and structure everything is a software developer's pipe dream. The "semantic web" failed precisely because the real world is much messier than that.
1. https://developers.google.com/search/docs/advanced/structure...
> Having a standard to encode semantics seems honestly necessary at this point if you want to keep things interoperable.
The trick is with the incentive structure.
Metadata is very common on academic paper pages, because Google scholar doesn't index them if they don't have it.
However it's also commonly just wrong.
More forward-looking restaurants manage this all themselves as part of their digital strategy, but it’s still a small percentage and disproportionately located in the US.
I’m not saying Google doesn’t also scrape or use other sources, I’m sure they do, but this is one of those situations where the whole system is broken. Tbh one of the bank shot benefits of having all of these digital delivery services is that some restaurants are using aggregators that can also publish menu data.
As far as the authors idea about markup for menus, that’s great, but highly improbable for a bunch of reasons: most restaurants don’t update their menu frequently, dishes are often difficult to represent structurally, POS systems are often modeled differently than the printed menus, etc.
I actually went to the web site just to see, and it's worse than I thought. Even their Western menu, the stuff random Westerners think is "Chinese food" is presented as JPEGs of photographs (sometimes out of focus) of the physical menu, which is itself strewn with typographical errors and mysterious annotations.
So, to get even the bad text an actual patron has in the physical restaurant you need to scrape the site, download the images, and successfully OCR from low resolution out-of-focus photographs. It's not impossible but good luck to you, and at the end the results will still be pretty unsatisfactory. "Frind pok" is actually what they wrote, they meant Fried Pork, but that's not an OCR error it's really what they paid to have printed.
However while the restaurant's manager might care, as I understand it her husband is the hard core chef who ensured it's a success, why should he give a shit? Presumably the errors are just in the text for stupid barbarians like me - many of them don't even order from his real menu anyway. His taste is what matters, nobody comes to the restaurant because of the typography or web site design, they come to eat his food.
The last one is a problem because real family restaurants do want to raise prices/be more upscale too, but none of their customers will let them because they expect banh mi to be $3.
Additionally, I have it in my head that not bothering to fix menus shows a certain admirable pragmatism. "Frind pok" is not correct but it is correct enough.
So, Google and these other companies, the option is - build it yourself and try to do better, or buy data from the companies that do this at varying degrees of quality, or don't have menu data at all. Except the last option, people _want_ menu data, it's one of the most common things people want to know about a restaurant.
Instead eveyone is busy building their own little feudal kingdom and they call it platform even if it's actually just a toll booth
So I’m sure you could sign up yelp and google and Uber eats and everyone else for a common data standard, but you’d then still have to go chasing the restaurants to go put that information in a system somewhere.
We haven’t even really been able to convince businesses to put their opening hours online, it’s still such a problem that one of my interview questions at google in 2018 was “name as many ways as possible you might be able to discover a business’s opening hours online”
Menus are about an order of magnitude more complex than that - it’s a tough thing to get restaurants to do.
Yet their hours are wrong, and different, on google and bing.
if I own a business there is no singular place I know of where I can publish my opening hours. My laundrette here can't publish the fact that they will be closed due to sickness except by placing a sheet of A4 at the door.
"A structured representation of food or drink items available from a FoodEstablishment."
Clearly the issue is that no one has convinced FoodEstablishments of the business case to spend tech dollars publishing their menus.
The waiter brought them heated plates.
Turns out that the cheapest thing on the menu was a heated plate service.
One day while in a border town in the south near Laos, my wife and I were in a suitability weird and humid restaurant with a slow ceiling fan keeping us a bit cool. On the only other occupied table sat a bunch of police with what looked like the local police chief due to the hat on his head but otherwise naked torso. They were just getting drunk so we couldn’t point at food and order. We asked for a menu. I pointed at 6 random things. They gave me a funny look. I then asked for 2 beers in Chinese. They confirmed “two beers?” with an inquisitive look. I confirmed. Then I asked for rice and remembered how to ask for spicy cucumber, a delicious side in china I had come to love. Eyebrows were raised.
Shortly thereafter out came two beers, spicy cucumber, and 6 mocktails in tall sundae glasses with umbrellas and curly straws.
The police table almost died laughing. Good times :)
When it comes specifically to restaurant menus in the US, most seem to be manually transcribed by the restaurant staff. The items and prices are correct (but often out of date), and food descriptions faithfully reproduce non-native spelling/grammar mistakes. In addition, I almost always see user-uploaded photos of the menus.
This does not point to a difference in Google’s automatic parsers or in the level of Google-generated content; it seems that US users contribute to map and PoI content a lot more.
I wonder why this is. My guess is that there are far fewer staff members at Google curating crowdsourced content outside of the US, which makes non-American users much less likely to contribute, since their contributions will appear much more slowly, if at all. I’ve contributed my own corrections to PoI data in the US (e.g. opening hours updates) and seen it reflected on the map in a few days. This probably wouldn’t happen elsewhere in the world.
To be fair, sometimes neither of these exist. A place in some parts of the EU may be open whenever the owner feels like it. You can't even approximate opening hours and holidays unless you actually ask the person over the counter when they're actually open.
I can't point to an example or cite a source (as this is just a guess), but maybe US users unknowingly contribute via Google Photos doing OCR (and other analysis) and combining it into Maps data, while Google is more careful about running AI against every photo taken by EU users (and using it to help in ways that go beyond exclusively the UX of the photographer) for data privacy compliance reasons?
Outside of curation, I think Maps lacks polish from non-US devs, and that results in weirdly unconvenient maps for a lot of cities, leading to people using it and contributing less to it.
For instance train station mapping (where are the entries/exits) is a feature available in some local map services and is a big quality of life improvement in europe or SEA cities, but never made it to Google Maps.
Same for the lack of multi-story building mapping, where there’s only a single shop for a single address, which can still work out for shopping malls (they have their own site), but is crazy for densely packed neighborhoods. Looking for restaurants in Paris or Tokyo through Google Maps is just frustrating.
Another bias I've noticed is US sites "collapsing" opening hours as if there is no siesta, ie opening hours are just displayed as 10am-11pm.
You'd be lucky if 50% of the restaurant supplied data is accurate, 40% is out of date, and 10% is actively incorrect. Personally I'd guess that the ratio would be more like 20/60/20.
Then why trust the site at all if it fakes metadata?
This fuzzy we-know-better algorithms has wrecked Google search.
If restaurants were rewarded with actual updated menus on google, you can bet the restaurants would care about creating the micro data, but it's a waste of time.
All of this without the mom-and-pop restaurant owners lifting a finger. It gives them a competitive edge over their competitors. All of this to say: Google doesn't care - but GrubHub, UberEats, and the ilk do care
There are lots of problems with scraping-based approaches.
One, yes, you need some really good tech to scrape data from menus, which, even though they are “structured”, next time you’re at a sit-down restaurant, pay attention to all the subtle discrepancies in formatting between different sections/categories on the menu.
Two, if the menu isn’t html, but is an image or a pdf upload, now you need some strong OCR on top.
Three, the website is generally not likely to be current with what’s actually on offer in the establishment itself. Specials, seasonal dishes, or items that are out of ingredients (“86’d”) will still appear on the menu. That’s going to lead to complaints, refunds, or generally bad customer experience from whoever’s consuming your data / using it to buy food.
Four, you’re going to want to to be paid for all this tech and customer support you’re electing to intermediate between the end purchaser and the restaurant, as a service, and so you’re going to tack on some fees and either jack the price up on the consumer or try getting the restaurant to pay you a finder’s fee, cutting in to their already narrow margins.
Five, if you’re trying to provide ordering service and not just menu data, you still need to submit the order into the store itself, somehow. Which either means calling it in, robo-submitting an online order (if you’re lucky), or sending a courier to place the order and wait. And then, on the other side, whoever’s taking orders for the restaurant has to punch in the request to the register to actually complete the transaction. Which means the system you really want to talk to isn’t the website, it’s the point-of-sale.
Good luck with all that.
Source of bias: I work for a company that helps restaurants enable online ordering and POS integration so they can pay much less in fees and focus on making exceptional food.
To some degree the real issue is that each restaurant can change hours (or menus) at a moments notice, and at many places, the staff and management is not super computer savvy. So no one thinks to update these sources of info, and/or they don't know how. Google has the added data (from tracking phones) of how busy the restaurant is at a given time, but that is presumably some sort of moving average over time, and not necessarily current or accurate.
When I first saw the headline for this post, I thought it was going to be about a related issue: even if AI is really good at understanding general spoken/written languages, the names and wording of menus is its own weird thing. If you're then trying to auto-translate that to another language it can be next to impossible. Ethnic restaurants in different places which supposedly speak the same language can have all kinds of spellings and ways of describing the same dish. In the US we call a long sandwich a sub, grinder, hero, or hoagie (to name a few), depending on where you live etc. Or the same name can mean wildly different things.
For example, if I search "similar pairs of sentences", it replies with "similarity between two sentences". No, I want a pair of sentences (A1, B1) that is similar to another pair (A2, B2). I am not talking about similarity between sentences, but between pairs. The distinction is lost on Google. And they claim to be using transformer neural nets for search. Pfft!
That is a good idea, especially for visually impaired users - but why can't we have both? I sometimes like seeing the fancy fonts, and images if I have never eaten that dish before, etc.
Example:
https://www.google.com/search?q=stanford+dish+hours&ie=UTF-8...
On the site it says "April-August Public Access Hours: 6:00am-7:30pm"
And I see the same on Google
I can confirm, I am working on information extraction and 90% is the glass ceiling for current models. With much labelled data you can get higher scores but they don't generalise from one vendor to another.
On a slight tangent: bear in mind also that Google is not party to whatever agreement there is between venue and customer, which is why it's so foolish that one frequently sees people asking questions on Google Maps as if they are directly communicating with the venue. Eg people ask "Can i get a child's cot in my room" - all it takes is for some joker to say yes, the naive asker to proceed to the venue and then find out that they offer no cots, at which point naive person has no leg to stand on (venue quite rightly says we've no idea what you're talking about).
Do they? The article says there's "no way for a user to contact Google".
However, putting that aside: Venues/business owners can contact Google and i suspect would get a slightly better response rate (not saying it'll be perfect or even super easy for an owner to get through but they can as there are often links indicating "Are you the owner of this business?")
I can't imagine this creating much of a barrier - Google would stop if told to by the restaurant but then they might not list the restaurant, and what restaurant would see that as worth it?
An average burger price in a given city would be different. There the intent would be to assume profit margins are consistent (albeit perhaps not hyper-thin) from city to city, and thus get a sense of the relative costs of burger-related inputs (chiefly: labor, real estate, utilities, taxes, and transportation).
By necessity, it'll be "pour it into this big stew of linear algebra" or nothing. Any kind of sensibility is simply impossible when you're as big as google!
or a crazy synthetic meat burger, how much did the first one of those cost, wasn't it in the millions?
What I have found occasionally useful is the (presumably automatic) categorization of photos into a menu category. Those at least generally let me know whether something's up to date (especially useful the last couple years as restaurant concepts have fluctuated wildly), and are very helpful for places that don't have much online presence - street food stands, pubs, local fast food places, that kind of thing.
Menu layouts are all over the place, so using OCR was trivial, but trying to figure out what went together and what modifications were valid on which items was a complete mess. Shrimp cocktail w/ cheese, anyone?
The above applies to all web walking robots with non trivial capabilities.
This is not at all to detract from the hard work and accomplishments made... its just still fairly easy to confuse even the most advanced AI. I guess its impressive that its only easy if you know how they work though, might take the average person awhile to find a way to trip them up.
Further than static difficulties, there may be a purposeful reaction to make it more difficult for the online world to “digitalize” the non-digital.
Good luck abstracting a reactive world.
E.g. "chicken" vs "locally-raised, hormone-free poultry"