Comment by dchuk | Hacker News Reader

Comment by dchuk | Hacker News Reader

Parent Full thread

While on the "how HN works" topic, I have a question:

I'm working on an HN app reader idea that I think is unique (more to come) that requires regularly getting the front page posts and comments. I wrote a script to do that through the Firebase API but holy shit it ended up needing thousands of requests to get all the data for the current state of the front page.

So instead, I wrote a scraper script to produce the same thing, just with 30 requests (1 per front page post). (Trying to turn these damn <tr>'s into a recursive comment tree was a quite the mindfuck btw)

...is it ok to scrape HN? I see nothing in the robots.txt to say it shouldn't be allowed, and I actually feel better making 30 scraping requests rather than XXXX API requests.

EDIT: Also happy to receive suggestions. I'm coding in Ruby and don't see an obvious way to access the Firebase DB directly (quite frankly I don't even know if what I just said makes sense) so any help is appreciated.

The algolia api has more bulk operations, so you can get more than one response per api request. Also if you look around, there are some datasets where people have downloaded all the HN API json responses, and have made them available, so you could use those for local development and testing.

As far as ethics go, it's probably better to use the API than scraping. Sure you're making more requests, but I think the firebase servers and cdn have far more ability to handle a lot of requests than the HN server does. If having that many requests would put firebase out in any significant way, then I'm sure they would put a query load limit on the HN api.

HN is behind Cloudflare; if you're not logged in the pages can be cached so I wouldn't expect small-scale scrapping to have any effect. Cloudflare might stop you though.

It sounds like your struggles are with Firebase more than the actual scraping part, so I'm not sure how useful this will be. But there's a marvelous demonstration of Clojure's ability to render and serve a scraped digest of the HackerNews homepage.

https://github.com/swannodette/enlive-tutorial/blob/master/r...

The final product is about 17 lines of code. Which is quite compact, sure—but if it looks like Perl, who cares?

Thankfully, it looks nothing like Perl. The code is deeply intuitive and evangelizes the idea that sending an object over a network can often be defined as a composition of functions resulting in the creation of the very object that we're sending. Whether that object is an HTML document, JSON, or whatever, the alluring premise is that these can all be forged from a single bulk-concatenation of strings returned by functions being unrolled in an inner-loop. It looks practically effortless. Might give you some ideas for batching those 30 GET requests. (Or maybe not—I've never used firebase before!)

Very cool! Always wanted to get into Clojure, I just don't code for a living anymore so only do it in my free time so I tend to turn to tools I know. Bookmarked though!

The Firebase API is the way to go. I built a small app to do exactly as you described [0] and wrote a blog post about it [1]. Drop me a line if you need any more info.

UPDATE: Using the Firebase feed means you don't have to do periodic scraping. You can simply set a listener for the FB change() event and the API will basically tell you when there is fresh information from the front page etc.

[0] - https://tophn.info

[1] - https://hackernoon.com/tophn-a-fun-side-project-built-with-v...

I can't seem to find anything that would let me do what you did, but with Ruby. I'm not opposed to using Node, just prefer Ruby more.

For Hacker News Daily I make requests to https://hacker-news.firebaseio.com/v0/

IIRC I made this change at the suggestion of someone at YC (maybe dang, but it might have been before he took over HN) so I think it's very likely that this is the route they would prefer you to use. But if in doubt, send them an email; dang is very responsive.

Hmmm.

What would you recommend for the instance where someone's emailed hn@ycombinator.com and hasn't gotten a response?

I suspect it's something I said, I'm just unsure exactly what. I asked about the posting and API ratelimits, the reviewer idea mentioned at https://news.ycombinator.com/item?id=11662380, source release (notwithstanding the voting system, which is fine), and other curios. I don't think I said anything offensive.

I have successfully gotten in touch in the past, so I'm just not sure what to do. The email was sent on the 27th of April; I just checked my spam folder (I must admit I didn't until now...), but I don't think gmail would have deleted any incorrectly-filed replies, since they'd be less than a month old.

I'm not complaining or trying to create drama, just curious what to do next. My fear is that it's not possible to answer the "what did I do" question due to social context and so forth (a hole I've fallen into many, many times due to communication perception issues).

Update for the record: I was, very happily, completely wrong.

dang got in touch to let me know the delay was due to backlog and that he'll properly reply when he gets the chance. Absolutely fine by me; I'm super glad to know my email was fine :D

Check out my implementation of comments parsing from the html, which is used in HNES: https://github.com/ibejoeb/HNES/commit/054fc4e626137aadef6f0...

Nice! I basically did it the same way. I'll push my code up to a public repo soon and let you know to compare.

Happy to help - email is in my profile.

Am I crazy? I don't see an email in your profile.

It is in the 'about' field.

It wasn't at the time your parent posted. It is there now. That, or we're experiencing folie à deux.

hckrnews.com ?

Reply on news.ycombinator.com