- at the moment we don't store the HTML content of the visited pages (except of the last one), so the only way to determine if something changed is to run the 'pageFunction' on each page again and compare the results. This can be optimized in certain situation, e.g. you can crawl a product listing and only go to product details page if some basic property changed. Saving HTML for each page is certainly possible, but after the crawler finished loading a page, running a pageFunction adds very little extra overheads.
- if a page cannot be loaded for any reason, a detailed description of the error will be present in the JSON results. We want to implement a limited number of retries for these pages, for situations the error is just temporary.
- certainly, if your crawling strategy cannot be expressed using simple pseudo-URLs, you can use the low-level 'interceptRequest' function to control exactly how each new page navigation request is handled (enqueued/ignored), tell the crawler which URLs refer to same pages and shouldn't be visited again etc. You can also enqueue arbitrary pages to crawl using 'context.enqueuePage'. In fact, you don't need to use pseudo-URLs at all and control everything from your code.