Hacker News Reader

Comment by explain | Hacker News Reader

robots.txt is meant for automated crawlers, not human-driven actions.

Every automated crawler follows human-driven actions.

Conversely, every browser is a program that automatically executes HTTP requests.

Yet they respect a lot of things meant for machine to machine interaction. Like server return codes, cookie negotiations, and CAPTCHAs if they behave a certain way.

So they sometimes hit bollards and turnstiles made for other types of code which executes HTTP requests. So they're bots basically, but better (or suitably) behaving ones.

Browsers let you visit websites without regard for robots.txt.

Welcome to "Context".

It must form the search index somehow. That is prior the human action. Simply it would not find the page at all if it respects.

I remember in late 90s/early 2000 as a teen going to robots.txt to specifically see what they were trying to hide and exploring those urls.

What is the difference if I use a browser or a LLM tool (or curl, or wget, etc) to make those requests?

But how did you find those sites that had the robot.txt to begin with? LLM must somehow find the existence of those pages and store that information before they can crawl them further or mark as acceptable source.

I am a human so I can visit other sites with links or from word of mouth or business cards or literally anywhere?

LLM finds out about it from me, when I ask it to go to the link.

You don’t accuse browsers of “somehow find[ing] the existence of those pages”. How does a browser know what page to visit?

The user tells it to.

If I prompt an LLM “go to example.net and summarize the page” how is that any different from me typing example.net in a browser URL bar?

That is certainly true. But that is not how these work 99% of the time. This post was originated by "search".

I think a distinction needs to be made between ingesting for LLM training and ingesting / crawling because a human asked it to during an inference session.

I have been talking about the latter, agree the former is abusive.

careful, some of those are honey pots or trip wires

Let's say you had a local model with the ability to do tool calls. You give that llm the ability to use a browser. The llm opens that browser, goes to Google or Bing, and does whatever searches it needs to do.

Why would that be an issue?

So, do you mean LLMs are human-like and conscious?

I thought they were just machine code running on part GPU and part CPU.

I think they mean that it's a tool accessing URLs in response to a user request to present to the user live - with that user being a human. Like if you used some webpage translation service, or non-ML summarizer.

There's some gray area though, and the search engine indexing in advance (not sure if they've partnered with Bing/Google/...) should still follow robots.txt.

Yeah, that seems to be a big distinction. If I tell my AI to summarize the headlines from my three favorite news sites every morning, it's just carrying out my request same as if I'd clicked to them, so that seems fine.

But if I say, "Search the web for a low-carb chicken casserole recipe that takes squash and cottage cheese," then it's either going to A) send queries to a search engine like Google, in which case robots.txt already should have been respected, or B) check its own repository of information it's spidered before I asked the question, in which case it should have respected robots.txt itself.

There’s a human using the LLM. In a live web browsing session like this, the LLM stands in for the browser.

Would you believe that humans turn on traditional web-crawlers as well?

if a human triggers the web crawlers by pressing a button, should they ignore robots.txt?

If a human triggers a browser by pressing a button, should it ignore robots.txt?

Are you arguing that these are equivalent actions?

The entire web was built on the understanding that humans generally operate browsers, and robots.txt is specifically for scenarios in which they do not.

To pretend that the automated reading of websites by AI agents is not something different…is quite a stretch.

I see it as very different. I the human want the data from that request. I am using a tool to get it for me.

Should I not be able to execute curl to download a webpage because the "understanding that humans generally operate browsers"?

> I the human want the data from that request. I am using a tool to get it for me.

Isn't this a bit of an oversimplification, though? Especially when the tool you're using completely alters the relationship between the content author and the reader?

I hear this argument often: "it's just another tool and we've always used tools". But would you acknowledge that some tools change the dynamics entirely?

> Should I not be able to execute curl to download a webpage because the "understanding that humans generally operate browsers"?

Executing curl to download a webpage is nothing new, and compared to a traditional browser, has about the same impact. This is still drastically different than asking an AI agent to gather information and one of the pages it happens to "read" is the one you were previously navigating to with a browser or downloading with curl.

If you're a content creator who built a site/business based on a pre-LLM understanding of the dynamics of the ecosystem, doesn't it seem reasonable to see these types of "readers" differently?

No, whether I curl it, or I use a browser, or an LLM, it is essentially ALL the same, unless of course the LLM crawls it by itself, without human interaction.

If the scale bothers you, block it, just like how you would block any other crawlers.

Other than that, we all wanted "ease-of-access" (not me though), and now we have it. It does not change anything.

What if the crawlers are faking their identity (As they are doing it right now)

Well, how do we deal with it in terms of DDoS?

It's reasonable for the content creator to see it differently, but I don't think it's reasonable to expect everyone around the content creator to contort any new approach to the needs of the pre-existing business model.

I agree. This came up in terms of copyright either, or who is pressing the shutter and who owns the copyright to the photo taken. I personally think that the copyright belongs to me, because I, a human, made the detailed prompt, the tool just generated it. Do I not own the copyright if I make something using Photoshop? As far as I know, I do. So, how is AI any different that needs human action (i.e. be prompted)? Because it is better than Photoshop? That is not a good argument, IMO.

In practice, robots.txt is to control which pages appear in Google results, which is respected as a matter of courtesy, not legality. It doesn't prevent proxies etc. from accessing your sites.

Reply on news.ycombinator.com