Hacker News Reader

Comment by oconnor663 | Hacker News Reader

Is it possible to configure a robots.txt file to accomplish #2?

No, robots.txt is for well behaving bots like bing bot and google bot, not bots that will spam your forums (and Git repo apparently).

But if the issue is SEO bots then robots.txt would block the search engines thus meaning the spam content is of no importance (it's effectively private) and doesn't cause issues for the main sites SEO (nor help the spammers).

To be honest, I'm not certain of the purpose of the spam.

Some portion of it would end up in the search results, sure.

But I don't know if there's some secondary benefit to, say, a casino showing a link coming from my site even if my site has a robots.txt saying that the address for that link isn't to be directly indexed.

Is there such a benefit? If not then I'll just set up the robots.txt and observe whether that does indeed solve the problem. But I'd much prefer to just set up the permissions I know I want on my own running instance than spend time making inferences about the reasons bots are abusing my instance's inputs.

It doesn't stop the spam from being created, but it does stop the spam from ruining your site reputation in search results was the suggestion I guess.

But the goal is to prevent spam in the first place. I don't think these bots will verify robots.txt to see if the spamming is effective. They just spam anything they can get their hands on.

They probably don't have general code that spam any form, it's more likely that they have code specific to GitLab CE instances that knows to post snippets. If GitLab changes their default configuration so that those snippets are no longer indexed by Google, the spammer are likely to stop using that GitLab CE spamming script, after a while.

That's right. I'm talking about SEO spam. Basically anything that has a url where the content includes input from the user will be spammed.

I'm fine with, say, the spammers hammering the main repo's merge requests and issue tracker. Those are things any healthy project will check regularly-- I'm even fine just pruning the spam there by hand (and historically I haven't gotten a lot there anyway).

But I don't regularly look at the global view of snippets, and I don't want to regularly prune the global user list for SEO spam in the user profiles. There's no good reason most FOSS projects need those things to be publicly viewable, anyway. But AFAICT Gitlab's admin settings only have a single setting that affects all these things across the board. So if you make snippets viewable only to logged in users, then nobody can clone from the main repo without logging in.

It's quite frustrating, and Gitlab shows no interest in disabling or hiding features like snippets and user profiles.

robots.txt is nothing more than a request. It's essentially a "no dog on lawn" sign in the yard of a vacation home.

Reply on news.ycombinator.com