Hacker News Reader

While I kind of agree with the rule of 3, in the example given, I would just rename BaseScraper to ScraperWithFormLogin when encountering the 3rd instance and not derive the 3rd from anything (or create an abstract BaseScraper) - there's still a high likelyhood there will be another scarper with form login.

View on HN

Why not just pull it out into a function, "loginWithForm(session, 'user', 'password')"? Why is it interesting on a class level that the scraper is using a form based authentication method? What if a page is switching between two authentication methods just for laughs?

I think you've hit it on the head. Mostly you shouldn't be trying to codify what a scraper is, you should be collecting little utilities that are useful for building scrapers. The more lego-like the better.

Reply on news.ycombinator.com