I think you got the wrong RFC number:
http://tools.ietf.org/rfc/rfc3938.txt
No mention of robots.txt, nor a section 6.
http://tools.ietf.org/rfc/rfc3938.txt
No mention of robots.txt, nor a section 6.
RFC 3986 - Uniform Resource Identifier (URI): Generic Syntax