Isn't the assumption always that accessing a web page programmatically is allowed unless specifically asked not to in robots.txt?