jancurn

I think around 1/3 of companies in our batch were remote, but as far as I know, now it's almost entirely remote to accommodate more companies.

We're still here :)

- at the moment we don't store the HTML content of the visited pages (except of the last one), so the only way to determine if something changed is to run the 'pageFunction' on each page again and compare the results. This can be optimized in certain situation, e.g. you can crawl a product listing and only go to product details page if some basic property changed. Saving HTML for each page is certainly possible, but after the crawler finished loading a page, running a pageFunction adds very little extra overheads.

- if a page cannot be loaded for any reason, a detailed description of the error will be present in the JSON results. We want to implement a limited number of retries for these pages, for situations the error is just temporary.

- certainly, if your crawling strategy cannot be expressed using simple pseudo-URLs, you can use the low-level 'interceptRequest' function to control exactly how each new page navigation request is handled (enqueued/ignored), tell the crawler which URLs refer to same pages and shouldn't be visited again etc. You can also enqueue arbitrary pages to crawl using 'context.enqueuePage'. In fact, you don't need to use pseudo-URLs at all and control everything from your code.

morph.io is a great tool but it requires a non-trivial setup on your machine in order to get things running. We wanted to enable people to create scrapers with no prerequisites.

Currently, there is no way to distinguish traffic from our crawlers, but of course, let us know at support@apifier.com and we will blacklist your websites from our crawlers.

Cool, if things go well, we'll be hiring soon :)

You're right that most of the time you don't need to use JavaScript.

But look at Google Groups for example - there is an infinite scroll to get all the topics, posts are also loaded dynamically, so you have to wait some time to get them.

In the SFO flights example you have to deal with pagination also using JavaScript.

We wanted to build a powerful tool which can crawl and scrape almost any website out there. It's slower, but you can use bench of our nodes to do it in parallel.

You could definitely do that, but then you might also need to:

- implement a mechanism to find and click active page elements, track the browser actions

- recompile PhantomJS to support POST requests

- implement a page queue with checks for duplicate URLs

- implement some parallelization and failover mechanism for case PhantomJS crashes (it does)

- possibly implement support for infinite scroll

- setup your own pool of proxy servers

- setup a database to store the results

and finally make the whole thing simple to setup and use :)

Many thanks! You're absolutely right, we'll publish the price ASAP and also provide some long-term guarantee for users who will depend on our service. BTW do you think pricing per GB downloaded is reasonable or would you prefer some flat monthly fee?

Great to hear that, let us know if we can help in any way!

Thank you! The user JavaScript code runs in the context of the web pages in the same restricted environment as normal web page's JavaScript. Also, the crawling processes are sandboxed.

I forgot to add: when you reach 1 GB we'll get in touch with you.

TBH we don't have the pricing defined yet, because we don't know what will be our server costs. In a few days we will know more and put up the prices.

Thank you! The crawler is currently based on PhantomJS, there is a pool of worker nodes that distribute the crawling to more machines. We use Node.js + MongoDB backend and Meteor for front-end.

Hello HN! Today we’re launching what we were building for the past couple of months. Apifier is a hosted web crawler for developers that enables them to extract data from any website using a few simple lines of JavaScript. We built it because we realized that many existing web scrapers trade off their ability to scrape complex websites for the "simplicity" of their user interface. We thought: we are programmers and we already use JavaScript for client-side development, so why not use it for scraping?

Please have a look at the service, play with the examples and maybe set up your own crawl. My co-founder jakubbalada and myself will be around here to answer your questions. We'd love to hear what you guys think!

Many thanks for your advice, this was the first thing I tried, unfortunately without any success...

I've had also quite strange experience when departing from the Ben Gurion airport in Tel Aviv two months ago. Neglecting the fact I got a very thorough security check for no apparent reason, have been asked questions like "Did you like it in Syria?", and got my pants wiped with a bomb detectors, the most humiliating thing that happened was that they asked me to unlock my iPhone and then walked away with it for a good few minutes.

I have no idea what did they do with the phone, did they read my emails? check my photos? installed some spyware? I have no idea, but I'm sure if I rejected to unlock it, I would be missed my flight. I felt like crap after that.

Interestingly, they didn't care about my second phone, an old Nokia, probably iPhone is more popular.