Hacker News Reader

Comment by varenc | Hacker News Reader

Can someone help me understand the underlying motivation behind this?

It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.

What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?

View on HN

They are poorly implemented by the "fuck you I got mine" crowd. They will get stuck doing things like trying to run through a calendar that could theoretically go back to the beginning of time and all the way to the end of time. And because that calendar might change, it gets scraped for every inquiry made to the poorly implemented AI system.

Similarly, I've never understood the economics behind the constant rescraping that is flooding the internet or really what's triggering it.

It can't all be agents reacting to user queries. It's confounding how much CPU and bandwidth is getting flushed down the drain.

If you have enough money to burn you don't care about efficiency. I personally blame VC model. Efficiency and profitability doesn't matter. Just make line grow up. And when same mindset is applied to those companies with money to waste... Yeah waste will happen.

My laymans understanding is that poorly written scripts get executed and then owner comes back occasionally to check that there is 'content' in their database. They're not sitting there troubleshooting their thing beyond 'it's getting data' because, for similar reasons to their scripts being poorly written, their strategy is 'get data'.

Nuance, complexity, and an awareness of 'other people' do not exist in their worlds.

The most recent data on the internet for advertising, intelligence, etc.

And a lot of bad scrapers.

Reply on news.ycombinator.com