Comment by franze | Hacker News Reader

Comment by franze | Hacker News Reader

I coded a few robots.txt parsers i.e.: (these are just the ones i cared to publish) https://github.com/franzenzenhofer/simple-functional-robots-... https://github.com/franzenzenhofer/robotstxt

and even though robots.txt seems like a very, very simple text based protocoll there are unanswered mysteries

the biggest mystery is user agent groups and comments

i.e.:

  User-agent: googlebot
  User-agent: bing
  User-agent: yandex
  Disallow: /

so i am disallowing everything for google, bing, yandex; easy enough, but:

  User-agent: googlebot

  User-agent: yandex
  Disallow: /

means that googlebot has no instructions, but yandex is disallowed all.

if a whole line is commented out

  User-agent: googlebot
  #User-agent: bing
  User-agent: yandex
  Disallow: /

is a commented out line a blank line or a non existing line?

if it is interpreted as a blank linke, the disallow only counts for yandex, if the line is non existant, it counts for googlebot and yandex.

i like simple things, but sometimes the complexity is between the lines.

  User-agent: googlebot
  #User-agent: bing
  User-agent: yandex
  Disallow: /

why would it be interpreted as a blank line? If you remove everything after the #, that includes the new line characters at the end of the line. Leaving:

  User-agent: googlebot
  User-agent: yandex
  Disallow: /

here the problem comes in, as people write

  User-agent: googlebot
  User-agent: bing    #behaved-badly
  User-agent: yandex
  Disallow: /

if # removes everythin afer the #, then it would be

  User-agent: googlebot
  User-agent: bing    User-agent: yandex
  Disallow: /

which results into the whole line " User-agent: bing User-agent: yandex" beeing thrown out as malformed, so only googlebot would be disallowed.

It's ambiguous, and this is why significant whitespace can be so frustrating. Unless it's specified, some people will interpret a full-line comment as a blank line with a comment ending it (`^#.$`), and others will interpret it as you have (`^#.?\n`). Neither is obviously correct (even if it's obvious to you).

Edit: I don't know how to escape in HN formatting. Obviously there are italics where literal asterisks should be.

  *** You can just use three asterisks. ***

* You can just use three asterisks. *

   Unfortunately you need something after them though. ***

Unfortunately you need something after them though.

Thanks, that's good to know!

If you indent it two spaces, I think that should solve that problem:
*(`^#.$`), and others will interpret it as you have (`^#.?\n`)*
(Not sure where you intended those asterisks. I made my best inference.)

Yeah I was hoping to keep them inline, but indenting surely would have helped. I intended them after the dots (`.*`, dot asterisk in case I get the escaping wrong, is 0 or more characters in regex).

What's the solution? Semantically it makes sense if it counts for both googlebot and yandex.
Have you observed one/both behaviours?

i am consulting a lot of aggregators, and we coose to interpret sometimes as this, sometimes in the other scenario. the way i interpret the spec, it is not defined behaviour.

Reply on news.ycombinator.com