Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

I deal with so many folks trying to scam our search engine every day I can't imagine the hordes at that gate here.

Reddit's response seems to be to take away the cheese in order to make the site less appealing to the rats. Its one strategy.

So far I've found that making it seem to the bad actors that nothing has changed is most effective at keeping them from getting worse. The first time we tried returning a straight error code for a robot search we got to see how fast a robot could send requests (pretty damn fast!) but we can send them the same 10 bogus serps again and again and again and they will chew their robot cud all day.

View on HN

So far I've found that making it seem to the bad actors that nothing has changed is most effective at keeping them from getting worse.

Right. The quality of feedback information is one of the biggest factors in development cost. The bugs you can't reliably recreate are the biggest problems, sometimes by one or two orders of magnitude.

This should be used as a weapon in the security battle. Seems like not enough people use it.

In all likelihood, the scam initiator is not the same person as the scam implementor, so "making it seem to the bad actors that nothing has changed" is likely to inflate their costs tenfold. This also works when the scam is entirely the work of one person, but it's especially effective when multiple parties are involved.

Some following this thread might find Tarpits interesting if they don't already know about them: http://en.wikipedia.org/wiki/Tarpit_(networking)

Do you guys know of any other techniques that waste the malicious actor's time making it think everything is a-okay?

Disciplined users - (http://meatballwiki.org/wiki/UsAndThem)

This usually fails because someone isn't strong enough to not respond. Or someone engages because it's a fun conversation for them. Examples of this on HN are some of the political discussions (eg Palestine) where people really should just flag and ignore but often people engage.

    Seems like not enough people use it.

Well, since this is a tactic that works best if the bad actor never finds out about it, isn't it possible lots of people do it without ever talking about it?

Relevant Coding Horror http://www.codinghorror.com/blog/2011/06/suspension-ban-or-h...

isn't it possible lots of people do it without ever talking about it?

I hope it's the case.

>So far I've found that making it seem to the bad actors that nothing has changed is most effective at keeping them from getting worse.

Reddit does do this, through shadowbans. Apparently, they felt that they weren't effective enough in discouraging antisocial behaviour.

As far as I know, shadow/silent bans only apply to users, not domains.

Does anyone know if reddit has tried extending silent bans to domains?

I once submitted a link from my blog to Reddit. It didn't complain, but the submission wasn't viewed by anybody, being totally absent from Reddit and I was able to see the submission only if I was logged in.

I then talked to the moderators about it and they said that my link submission history contained multiple items from my blog, so the automatic spam filtering kicked-in, with my domain being blacklisted. Because the submission was good and ontopic, one moderator proceeded to whitelist my domain again.

I don't know if the key for this silent ban was just the domain name or the combination of (username, domain name), but it was a silent ban.

That isn't a shadowban. That's just the regular spam filter. A shadow ban is a slightly different approach. A shadowbanned user, when logged in can see his posts and comments as if they were there normally, but only he can see them. Other users don't even know about the existence of them. (Mods can see shadowbanned user comments and posts in their subreddits, but normally don't approve them because each post/comment has to be approved manually.)

Yeah, but I was able to see my submission while logged in, as if nothing happened.

How is a robot used to scam a search engine? Are they stealing your results on a per-search basis?

Sometimes. Sometimes people try to build databases of pages about people, you know search for "Kennedy, Alice" and then get all the pages we have, then "Kennedy, Bob" ... "Kennedy, Charles" etc. People then try to offer "Find out about <name>!" types of services on their web sites. Or some wordpress theme will have a sql injection vulnerability and we'll get robots trying to find every site we know about that run that particular theme. There is lots of stuff you can do when you have a copy of the Internet, not all of it 'good.'

Now we do sell API access to our index, but not for stuff like this.

Thanks. Interesting.

Trying to scam Blekko alone, or all search engines?

All.

When I was at Google there is a new hire class that talks a bit about security and 'bad' queries. There is an admonishment in the class notes to 'not' experiment with those queries unless you were a member of SecOps. They claimed they would notice, and they would track you down.

So to answer the meta question, all search engines have this problem. I know this is true because of the new hire class at Google (largest search engine) and because we (Blekko) offer our index through API access to a number of smaller brands you may have tried and by doing that we see what they see in terms of search queries.

Generally there is someone in an operations role at those partners with whom I can share notes and pass along the latest fashion in terms of scams.

By bad queries, you mean if you query "how to buy links without getting notice", Google will track you down, find out who you are, and try and get your sites penalized?

No, I mean queries that look for things like exposed credit card numbers or passwords, or vulnerable software. Web crawlers don't discriminate in what they index and inexperienced or inept web administrators sometimes put things out there that they shouldn't.

Criminals attempt to exploit that, using search engine results to figure out likely targets.

There are also people who try to exploit keyword searches and the advertisers who buy advertisements on them but that is less obviously criminal.

Interesting so Google has an "Internal Afairs" dept who audit themselves ie if some one looks up celbrities details internaly either for curiosity or to sell to the tabloid press.

I know that big telcos have internal security goups and British Telecoms had/has a ferocious rep.

BT got even stricter after some one got a temp job and looked up the x diretory numbers for one of the Queens residences.

After that they started doing posative vetting for team leads on major projects - that is the equivelent of TS clearance in the USA

Reply on news.ycombinator.com