Hacker News Reader

Nowadays I would jump straight to selenium (or similar), as most websites feature AJAX or similar, so need a full browser.

Then, you don't actually do anything with selenium, click a button / link, or anything interesting.

View on HN

While it's true I often end up needing something like selenium, it's way more heavy handed and I usually reach for it last. It doesn't scale as well, harder to troubleshoot IMHO, and more libraries and dependencies to deal with in a language where that's already not great.

Agreed, this is why on the Selenium paragraph we link to this article that go much more in depth https://www.scrapingbee.com/blog/selenium-python/

Reply on news.ycombinator.com