Hacker News Reader

The winds may be changing, but those who don't fear resorting to piracy will always be sailing smoothly. We will just have more walled gardens and more illegal offerings, leaving normal people stranded and thirsty.

View on HN

Unfortunately things like a 500gb dump of sacredtexts website is not often found on torrent trackers or other 'warez' sites. Anna's is pretty great for written material that has been published offline, but even the wayback machine and archive.org have limited full scrapes aside from the published ones by the kiwix team.

Such megadumps are only really interesting for people who train LLMs anyways. No way you can ever consume that for personal development. Individual sources are much more prevalent and useful to everyday life.

It would be interesting for a personal project to do some search and summarisation project, e.g. with RAG.

Get the data, make embeddings, categorize and tag the data, then do search and have LLM summarize findings.

:)

Reply on news.ycombinator.com