Robots.txt for the NYT has a specific exclusion for an 1996 news article

I've added a special exclusion to a robot.txt file for a specific article. It was some years ago while in college. The article in question was about the presentation of an assisting professor who had some kind of misunderstanding with the campus newspaper and therefore the article wasn't especially positive in tone. Couple of years later I was the sys admin of the newspaper website and a letter arrived in my university email. The professor had found that I'm responsible for the website and had sent me a tearful story about how this article is ruining her life, because it is the top Google result for her name, and how she had spent thousands of dollars on scammers who had promised to change that, and she was asking me to remove the piece. Long story short, I forwarded the case to the newspaper editor at the time and she agreed to let me add a line to the robot.txt.

Edit: newspaper -> newspaper editor

Just for future reference: adding a URL to robots.txt will not necessarily exclude that web page from Google, especially if it has already been indexed.

To reliably exclude a URL from indexing, you have to serve a “no index” instruction with that URL, either in a meta tag or an HTTP header. And for this instruction to be read, the robot has to visit that page! So disallowing the URL in robots.txt can actually be counterproductive to de-indexing it.

Google also offers a tool specifically for removing URLs from their index in Search Console.

Here is the article: http://web.archive.org/web/20091128124216/https://www.nytime...

So does Wired and other news sites.

https://www.wired.com/robots.txt

It's concerning someone working for the news doesn't understand why.

This is their power of ruining lives forever.

This is a new thing, it should be taught, it's not hard to understand.

I don't know why journalist think they deserve respect when these things are not fundamentally in their ethos.

There should be a better process than robot.txt and some news sites are doing better. Europe has brought in laws. But if journalist want to be thought of as more than writing blog spam, they need a better answer to this.

I have a specific exclusion in my robots.txt file, and also a cron-scheduled grep of my logs to see if anything actually hits it. I don't care if they do or they don't, but it's a way for me to exclude specific bots that don't honor my robots.txt file.

It was about a rape case. The Internet Archive shows the article was updated in 2008 to say the case was dismissed and sealed in 1997.

Diplomatic immunity extending to the point of sealing the case and suppressing reporting?

Scathing Glassdoor review for his current company entitled "Just Another One of Tarik's Victims" among a sea of obviously fake ones (yes I know GD is bs)

the US Department of State (DoS) during 2012/2013 in its robots.txt[1] excluded around 9577 documents which leaked into archive.org (already pre Snowden). The robots.txt file now is OK but not sure if content is still on archive.

[1] https://pastebin.com/raw/RE2tpyR3

  8<-----------8<-----------8<-----------8<-----------8<-----------

  #!/bin/bash
  snapshots="20120713050942 20121013154343 20121010165822 20120921054221 20130413152313 20130113162428"
  # orig source http://state.gov/robots.txt but also on pastebin in case they delete it:
  wget --output-document=robots.txt http://pastebin.com/raw.php?i=RE2tpyR3
  for x in `echo $snapshots`
  do
    for i in `cat ./robots.txt|cut -d ' ' -f2 | tr -d '\15\32'`
    do
      if [ -e `basename $i` ]; then
        echo "$i already fetched"
      else
        wget https://web.archive.org/web/$x/http://www.state.gov/documents/$i;
      fi
    done
  done

So much for the right to be forgotten. And this is a dismissed case.

Reminds me of companies posting copyrighted material in comments only to file a DCMA takedown.

If you can’t get NYT to remove it, maybe you “know someone” who can.

Using robots.txt to avoid search engines indexing the page is the most stupid thing you can do. Not only it's not mandated by law that search engines have to follow the rules in the file, but also you are giving to the public a known file where you put all the URL that you don't want to be public. And everyone that wants to get some information on a site the first thing that goes to see is the robots file.

The correct thing would be to serve pages with the appropriate HTTP header to disable indexing. Of course search engines are still not obliged to follow the header, just as they are not obliged to follow the robots.txt file, but you are not leaking more information that you need.

Really, robots.txt file is only useful to reduce the load on the server by crawlers, it shouldn't be used as a protective measure!

Here's a relevant Radiolab episode about the right to be forgotten.

https://www.wnycstudios.org/podcasts/radiolab/articles/radio...

This was one of the very rare cases where even wikileaks took down one article in the global intelligence files, after the state dpmt complained. A very high profile case. In that specific country everybody knew, what the diplomate of the other very specific country did.

(2019)

A follow-up on one of their article. More of it. Please.