Hacker News Reader

Or how you end up with a search engine for passion projects, open-source projects, community hosted sites and more.

Treat paywalls the same as ads.

How could the crawler discover the pay-walled content? If sites explicitly allow the crawler to enter, what would be their argument to prevent that the search engine shows a cached version, especially if that's the search engine's policy from the start?

That said, search engine for pay-walled sites with integrated micro payment isn't a bad outcome. What else could create an alternative to the ad-based business model?