Hacker News Reader

I hate to be this person, but... it's scraping, not scrapping. You're scraping the information off the page regardless of what its structure is.

Yes. Converting docs to markdown and using them in claude projects, for example, makes a big difference.

Yeah, I'm not sure what the point of markdown is here either. I would expect that anything that looks remotely like a URL will be collected and scraped no matter what format it's in.

Context windows for LLM inference are limited. You can't just throw everything into it -- it won't all fit, and larger amounts of context are slower and more expensive. So it's important to have a carefully curated set of well-formatted documents to work with.