Hacker News Reader

Comment by znnajdla | Hacker News Reader

Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.

View on HN

This is not meaningfully static content. Look at the charts in TFA. There is a combinatorial explosion of distinct URLs which the crawlers can and do request.

They allow you to diff commits, which is an awesome feature.

But if bots a crawling diffs between all possible commits it's crazy.

CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.

No CDN blocks bots. Some claim to. None do.

Did you read the article?

“[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”

From the HN guidelines <https://news.ycombinator.com/newsguidelines.html>:

> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".

What if it's really obvious they didn't read the article?

I did read the article. It just didn't occur to me that their combinatorial explosion of diffs was scrape-able. To be honest that sounds like an scrapers tarpit / honeypot now, because there is no value in scraping trillions of diffs. Sounds like the issue could be fixed by putting the diffs in a frontend app, not scrapable by URL, only by clicking around the app.

Reply on news.ycombinator.com