very informative, thanks! i coincidentally just posted a blogpost about the other complementary approach using pure HTTP requests without headless browsers: https://news.ycombinator.com/item?id=21770576