Hacker News Reader

Comment by jdm2212 | Hacker News Reader

Xoogler here... my favorite Maps spam story is how there was a cron job that ran every (I think) 6 hours to analyze and clean up suspected scam listings [1]. The scammers figured out when the cron job runs, and set up their own cron job putting up their listings again in the minutes after Google's finished.

It's a well-known problem on Maps and has been known for years. It boils down to: scammers are smart and resourceful and very good at making their scam listings look like real listings.

[1] if you're wondering why they didn't do the filtering at submission time, there was a good technical reason for this at the time that I don't remember anymore

View on HN

That is a great story, at Blekko my first attempt at getting rid of people trying to exploit the index was to ban any address that did more than 100 queries per second, that got a bunch. Then they start showing up again, at 99 queries per second :-). So I lower it to 50, same pattern but now 49 qps, then set it to 10 qps, they would limbo in under the wire again. They would also start using bot nets to send their query from different PCs in different places (all the same query though, typically looking for ecommerce or wordpress templates with known exploits) We would nail them by finding all the machines that made the same query, then we could just start killing any query that asked for more than page 10 of the results, etc.

For a while I pondered if I could use my ability to change their behavior by changing my rules, to get them to actually create a set of behaviors that would unmask their C&C servers but not enough free time for that.

You could start sending fake results after they exceed the ~100 queries, they will probably stop exploiting it.

Yeah, "hellbanning" scammers is a good idea, instead of giving them a simple 401 code. Increasing the cost of their scam is the best mechanism.

This is a good idea, actually -- make it impossible to distinguish a banned request from a succesful request. At least that could delay the discovery period, as would randomizing the request limit.

I've wondered if a web server or proxy in something like Erlang would be ideal there, where there is little server-side penalty in dragging requests out progressively longer.

With the lightweight processes, the cost per should be very, very low, and basically make it easier to Turing tarpit the scammers.

There are public records for businesses. I just learned from another comment that locksmiths are state licensed.

Wouldn't those data be useful for fraud detection and prevention?

Are the records all of the following: 1) structured 2) available via web API 3) consistent across jurisdictions 4) reasonably up to date

I'm guessing they're not. Google has had hundreds of really smart engineers with an effectively unlimited infrastructure budget working on Maps spam for a decade. If it were easy to solve once and for all, they would have.

There are at least a dozen companies that aggregate and resell that data. I'm familiar with LexisNexus (nee Seisent). I'm certain Google could do it too.

This is not a technology problem, its a process problem. Vast majority of submitters could be cheaply verified and for small number of potential spammers, there are several options:

1. Require submitter to be legally traceable and identifiable. Establish fines/fees for misuse of your property and charge their bank accounts for cleaning fees. Blacklist them for life.

2. Once a submitter is identifiable and legally traceable, charge exponentially higher fees for number of submissions unless they want to show up as chain business in which case charge reasonable fee for validation and background checks.

3. Detect and avoid autonomous mass submissions except for highly trusted entities such as big chains.

Google Maps is not optimizing for "percent of listings that were completely accurate at time of submission". That's how paper maps (and to a lesser degree, the Yellow Pages) mostly worked. There's a reason we stopped using them.

What Google's doing is harder and more nuanced -- they want to accurately capture the world as it exists around you right now. That means understanding what parts of the world around you you care about and understanding what level of accuracy (as opposed to promptness of updates) they should optimize for. Given Maps' extraordinary success as a product, they've done this quite well. But, yes, it has failure modes that users should be aware of and that Google tries to combat without compromising the main goal.

You are begging the question, explaining their approach as a success instead of explaining how it is a success.

Why not have both? Still allow users to submit their own listings but (like twitter/Instagram) have an additional verification process which gives business a "blue tick" if verified.

Copied from another comment of mine:

> The Maps mobile app has had this for at least 3 years -- they ask you questions about places you know. You can also suggest edits on Google Maps which are sent to others for review But not that many people use either feature.

Google is trying to build maps without human verification, because humans are expensive, and to some degree because humans can be tricked too.

Accepting content without verification certainly has a side effect of getting businesses on board the instant they open, too, but that's just a happy accident.

Google is fine with expensive (they sunk many billions into Maps over 10+ years and only just started monetizing it). But humans don't scale. If you want to have ten thousand data points about every business on Earth, you can't get there by relying on humans. You need to automate everything from top ot bottom.

Perhaps scaling to Google-size is the problem.

Humans can scale, it's just that Google chooses not to use them. Anything that requires a human in the loop is something they won't do.

Customer service, not going to happen. Having Youtube Kids actually have appropriate content for kids, unlikely. Being a trustworthy source for businesses on maps, no way.

Wikipedia seems to be an example of humans scaling.

Google products have up to 2 billion users. It's nearly impossible to scale any human support to this size.

Because it's actually billions of users. It's a billion of users for Google Maps. It's two billion of users for Youtube. It's nearly a billion for GMail. And the requests for those different products have to go to different support personnel.

In areas where there's significantly less users they do provide human support, for example in GCP.

If it's impossible to provide human support for a system, and if it's possible to automatically exploit the system to affect a nontrivial fraction of users, then perhaps the system shouldn't have billions of users?

There's resilience in diversity.

> then perhaps the system shouldn't have billions of users

How do you imagine Google Maps achieving that? Telling people "oh, sorry, can't serve you today"?

I work for a product with big numbers too. We have humans reading the support queue (with a lot of engineering to prefilter and group for bulk replies and what not). People may or may not like our support, or may complain it's not as good as it used to be, but it's there.

Frankly, we couldn't have as many users as we do without a human connection in support. Users tell us what we're doing wrong, and what we need to do that we're not doing, and where we need to improve --- but only if you listen to them.

> Frankly, we couldn't have as many users as we do without a human connection in support.

Different companies and different products are different. It looks like many Google products are good enough to attract many users without support.

Google Maps does allow any user to submit edit request for place details (opening hours etc) and flag suspect entries. I guess one could be upset about this attempt to get users to work for free, but personally I don’t consider exploitative because (a) verifying local business is actually far easier for local people compared to anyone Google could hire and (b) Google Maps is such a useful product that the distinction to, for example, Wikipedia is less obvious than the simple for-profit/non-profit designation makes it out to be.

> What Google's doing is harder and more nuanced -- they want to accurately capture the world as it exists around you right now.

I challenge that. I do not believe it is true. Google wants to be the go-to directory for spatial queries so that they can display their ads and make tons of money. The quality of the data is irrelevant after a certain level, as long as there is no viable alternative or people just blindly trust Google. Google Maps' data is quantity over quality by a wide margin.

Google is a huge organization. It is possible for Larry or Sundar to have a 30 year plan that involves Maps making money while every engineer working on Maps day to day (including Director and VP-level people) cares deeply about product quality and has the freedom to actually put a lot of resources towards that.

I agree about quantity over quality -- the product strategy boils down to "collect lots of data and build models with it". In the short term, each new feature has middling quality data with middling quality models. But over time they get better and better. It's just Maps keeps adding new features that are at that early, kind of crappy stage, and you forget how much the really polished ones used to suck.

> Vast majority of submitters could be cheaply verified and for small number of potential spammers

Yeah, okay sure. I'm sure all the engineers working on this product at google have never considered anything you wrote before.

If there was a cheap, easy way to block spammers and meet all their other constraints, I'm sure they'd have done it by now.

No matter what you do, if there is money involved, the scammers will eventually work around whatever you have in place to stop them. It is a constant, never ending game of whack-a-mole.

This is laughably ridiculous and not scalable at all. Stores close and prop up all the the time. Not everyone thinks of signing up for google maps. And besides where is the manpower to do verification of millions of stores every day? It’s very easy think something is easy until you actually think of the real world details.

What if you required 7 hours for a listing to go live?

The Maps folks played (and are almost certainly still playing) this game of cat and mouse with the spammers. Google comes up with better defenses, and the spammers come up with cleverer spam. I just found the cron job iteration of that cycle particularly memorable and amusing. I don't want to share anything else because I'm sure they still use some of the strategies they used back then, and it's better not to publicize them.

there was a good technical reason for this at the time

I gather there's a number of selfish business reasons for it instead. Google no longer gets my benefit of the doubt in these situations. They won't solve it, because solving it costs money.

Reply on news.ycombinator.com