Hacker News Reader

Comment by tooltower | Hacker News Reader

> Rather than downloading our dataset in one complete download, they insist on loading all of MusicBrainz one page at a time.

Is there a standard mechanism for batch-downloading a public site? I'm not too familiar with crawlers these days.

View on HN

There isn't. There never was one, because vast majority of websites are actually selfish with respect to data, even when that's entirely pointless. You can see this even here, with how some people complain LLMs made them stop writing their blogs: turns out plenty of people say they write for others to read, but they care more about tracking and controlling the audience.

Anyway, all that means there was never a critical mass of sites large enough for a default bulk data dump discovery to become established. This means even the most well-intentioned scrappers cannot reliably determine if such mechanism exist, and have to scrap per-page anyway.

> turns out plenty of people say they write for others to read

LLMs are not people. They don't write blogs so that a company can profit from their writing by training LLMs on it. They write for others to read their ideas.

LLMs aren't making their owners money by just idling on datacenters worth of GPU. They're making money by being useful for users that pay for access. The knowledge and insights from writings that go into training data all end up being read by people directly, as well as inform even more useful output and work benefiting even more people.

And rarely cite their sources, thus affording the author not so much a crumb of benefit in kind.

Which is irrelevant if you're truly trying to "pay it forward".

That is the core of my observation: people claim to publish to benefit society, but push come to shove, they care more about getting credit and having oversight over who is benefiting, to the point of refusing to publish further (and sometimes unpublishing things) if that credit/control isn't given.

The problem isn't in wanting these things - it's in not being up-front about it.

Except the output coming from an LLM is the LLM's take on it, not the original source material. It's not the same thing. Not all writing is simply a collection of facts.

> turns out plenty of people say they write for others to read, but they care more about tracking and controlling the audience.

I couldn't care less about "tracking and controlling the audience," but I have no interest in others using my words and photos to profit from slop generators. I make that clear in robots.txt and licenses, but they ignore both.

They're not talking about downloading the web pages. The data is available in a bulk download: https://listenbrainz.org/data/

Reply on news.ycombinator.com