Hacker News Reader

Comment by belval | Hacker News Reader

> “I don't understand why the onus is on me to add a new header to my sites opting out of this tool,” Eden said. “Please can you change the default behaviour so that it will only work on sites which set the X-Robots-Tag: YesAI?”

Not respecting robots.txt is idiotic and website maintainers are right to be pissed, but that's not a good faith proposal. People don't have to opt-in to Google crawling their site with a special header, Google just crawls everything unless you add noindex[1].

[1] https://developers.google.com/search/docs/crawling-indexing/...

View on HN

In their defense, most websites want to be indexed by Google.

"New services shouldn’t expect to be welcomed in the same way, "

Why exactly? So megacorporations can keep owning everything of value?

+I don't see how downloading all the images on a website is disruptive.

If you can't handle each of your images being downloaded once something is really wrong with your website

I guess how I feel is, if there are 50 new services that spring up overnight with different opt outs that would be annoying for people to deal with. If the website operator isn’t getting value out of the service they might just block you or put content behind an paywall and we all lose. =/

Also, the article claims the scraping generated so much traffic that it got reported to the owner as an attack. I’m not claiming anything other than what’s being reported. They should have at least throttled their application to reduce the load.

If the owner didn’t notice the scraping we might not even be having this conversation.

> Why exactly?

Because when the megacorporations were just corporations the web was young.

Now the web is old, and what was interesting and novel at the small scale is tedious and taxing at the large scale.

It's much the same as how it's now a death march to write a new standards-compliant browser, or how you can't really found a new country unless you want to start a bloody war or carve a chunk out of Antarctica.

Sometimes, the rules have changed by the time new things come along.

> being downloaded once

the issue isn't "downloaded once" but "downloaded at once". agressive crawlers often turn into dos attacks

Reply on news.ycombinator.com