Ledio··on Ask HN: Is there any non-web-development jobs out there?We're hiring at Splunk: http://www.splunk.com/view/careers/SP-CAAAAGG
Ledio··on Ask HN: Is there a great open source crawler?Nutch has a full on web crawler, a lot of features, and it scales pretty well. You can white list or black list URLs as you see fit, and filter out unwanted content.