Hacker News Reader

Looks like a good idea.

Is the idea to crawl locally? How do you crawl - sequentially or in parallel?

Thanks. We first grab all browser history from your existing browsers and slurp it into a single sqlite database. We do this sequentially but since it's sqlite-to-sqlite it's usually very qucik.

For the full text portion we send out a request to the URL for the full HTML, distill it (similar to "reader" mode in some browsers) and add that distilled content to the full text index.

The full-text generation happens with some concurrency but we intentionally didn't want to spam anyone's server so it's limited and takes a while to populate.

Yup! Specifically we extract the full text for each website 8 URLs at a time right now, but you can still search over the titles and URLs while the full text is being collected

how does the crawling works, if the URLs content is behind logins or similar?

Hey wuyishan, right now it isn’t able to retrieve the full text of content that’s behind login walls, but we are looking at ways to allow it to handle those sites as well

Reply on news.ycombinator.com