Hacker News Reader

There is an RFC, but in practice the format is super informally specified. You'll encounter

# comments

// comments

/* comments */

plain text lines as comments

TeX-markup

script tags

HTML

PHP code

search-engine specific directives (Yandex has a few strange ones)

Somebody screwed up and made their robots.txt in a rich-text editor like wordpad.

That's an RTF file that was saved as a TXT (without removing formatting)...