Hacker News Reader

Comment by bdclimber14 | Hacker News Reader

I've never written anything like this... Is there a function like CURL that downloads the page as a string from a URL?

  require 'net/http'
  require 'uri'
  output = Net::HTTP.get(URI.parse("http://www.google.com"))

curb is a lot faster than net/http

require 'curb' c = Curl::Easy.perform("http://www.google.com) puts c.body_str

Premature optimization much?

So apparently accessing a secure URL isn't straightforward... No login is required, its just an HTTPS but it won't go in curb.

Wasn't an SSL problem but a cookie problem...

I wanted to thank you for your help. I wrote a script to scrape over 30,000 courses and textbooks from Arizona State University successfully in a couple days.

Reply on news.ycombinator.com