Hacker News Reader

paramawon Show HN: Till – Unblock and scale your web scrapers, with minimal code changes

You can connect your local browser to Till. But, you need to have your browser to accept a custom CA(Certificate Authority). Instructions here: https://till.datahen.com/docs/installation#ca

paramawon Show HN: Till – Unblock and scale your web scrapers, with minimal code changes

We don't have a hosted service at the moment. It's all run on premise. The API link you mentioned is for a Till instance to validate the token and for usage stats reporting for each user. The Auth token validates each user's free and premium features and limits based on usage stats more specifically total requests counts and cache hits counter. We don't record nor track anything related to the requests made on Till.

paramawon Show HN: AnswersEngine – A flexible web scraping platform for Ruby devs

Hello, I'm a co-founder here. I've recently built a web scraping platform for Ruby developers. It is intended to solve the pain points of what I've experienced over the years of doing web scraping for my clients.

Things that are just a hassle to do, are already taken care of:

Proxy and user agent randomization, so that you don't get banned easily.

Parallel scraping, that allows you to scrape at scale.

Page Caching. Imagine your scraper already runs for several days, and it fails.... would you rather re-download the pages all over again? or would you rather just parse from the cache?

HTML parsing using Nokogiri, and/or any other gem.

A pretty granular RESTful API, so that you can integrate it with your app.

Currently, it's in Beta. Feel free to join. I'm giving away $100 free credits, so that you guys can test it out. Any feedback, is greatly appreciated.