Hacker News Reader

Comment by tuckerconnelly | Hacker News Reader

Kieran! Kieran (OP) is a friend of mine, highly competent, and I highly recommend his comprehensive legal guide to web scraping: https://mccarthygarberlaw.com/a-comprehensive-legal-guide-to...

View on HN

Thank you sharing the link. I find it amusingly appropriate that viewing this legal guide requires agreeing to Terms & Conditions, but appreciate that it’s one of the shortest Terms & Conditions I’ve ever seen easy to read and understand within seconds.

Haha! Given that the article is about scraping, I figured some prudence with respect to people copying and pasting my work somewhere else was warranted!

Just for fun, here is the robots.txt from that site:

    # Default robots file version:2
    User-agent: \*
    Disallow: /calendar/action\* 
    Disallow: /events/action\*
    Allow: /*.css
    Allow: /*.js
    Disallow: /\*?
    Crawl-delay: 3
    Sitemap: https://mccarthygarberlaw.com/sitemap_index.xml

Thanks, Tucker!

Reply on news.ycombinator.com