You joke, but it's not that far:
http://cpedia.com/robots.txt — No disallow on /wiki
Except, maybe, that mahalo's "content" is far better scraped.
Except, maybe, that mahalo's "content" is far better scraped.
I'm not sure what a crawler is supposed to conclude in that case.
I don't see a lot of cpedia content on Google yet. The only links I can see are clearly due to the threads here and on sites like Reddit.
User-agent: *
Disallow: /imgsrv
Disallow: /search
Disallow: /pref
Disallow: /suggestIt lets you in regardless of the username/password combination. I just logged in with this:
username: you morons
password: this defeats the whole damn purpose
I wonder if any of this gets logged at their end.