Hacker News Reader

I've tried this for the first time recently in 10 years - it's really become a miserable chore. There are so many countermeasures deployed to web scraping. The best path forward I could imagine is utilizing LLMs, taking screenshots and having the AI tell me what it sees on the page; but even gathering links is difficult. xml site maps for the win.

View on HN

Literally step for step what I spent my weekend putting together. Here's the preview blog I wrote on it. https://blog.bonner.is/using-ai-to-find-fencing-courses-in-l...

Only step you missed was embeddings to avoid all the privacy pages, and a cookie banner blocker (which arguably the AI could navigate if I cared).

Awesome! Things have gotten so bad this is the only alternative. I tried building a hobby search engine then quickly gave up, but did imagine how I would do the scraping!

Reply on news.ycombinator.com