Hacker News Reader

Comment by tocomment | Hacker News Reader

Just write yourself a Ruby script that downloads a webpage. Then parse the downloaded content with Ruby's equivalent of BeautifulSoup (a forgiving HTML parsing library).

That's pretty much all there is to it. You'll want to manually inspect the HTML from the page you're scraping to find patterns. For example all courses are in <li> tags, etc.

Let me know if you need more help. But I think as a first step write a program to download a web page.

View on HN

I've never written anything like this... Is there a function like CURL that downloads the page as a string from a URL?

  require 'net/http'
  require 'uri'
  output = Net::HTTP.get(URI.parse("http://www.google.com"))

curb is a lot faster than net/http

require 'curb' c = Curl::Easy.perform("http://www.google.com) puts c.body_str

I wanted to thank you for your help. I wrote a script to scrape over 30,000 courses and textbooks from Arizona State University successfully in a couple days.

So apparently accessing a secure URL isn't straightforward... No login is required, its just an HTTPS but it won't go in curb.

Wasn't an SSL problem but a cookie problem...

Premature optimization much?

Reply on news.ycombinator.com