Robots.txt for the NYT has a specific exclusion for an 1996 news articletwitter.com·270 pts·mikeortman·100
Show HN: Asdfasdf.co – emails from the internet's gibberish form fillsmedium.com·78 pts·mikeortman·31