Show HN: flyscrape – An expressive and elegant web scraper

github.com

14 pointsphilippta5 comments

Hey, quick question: Is there a way to handle async behavior? Like in puppeteer we had something like waitFor helper to wait while loading the page. I couldn't find a similar thing in your readme. Or is this framework for only static pages?

At this time it works with server-side rendered / static pages only and does not execute any JavaScript of the website.

The data extraction function of your script also runs aside the crawler, not inside the page.

This looks interesting, how would you compare it with colly? I’ve used colly for a while and overall am pleased with it, but things can always be improved.

Colly is a great library for builing scrapers if you are familiar with Go.

Flyscrape a cli tool that lets you define your scraping logic with a declarative JavaScript API, while it takes care of issuing requests, caching, rate limiting and more.

Ah lol somehow I totally missed the fact that you write it with JS. That is a neat approach.

Reply on news.ycombinator.com