Hacker News Reader

So, do you mean LLMs are human-like and conscious?

I thought they were just machine code running on part GPU and part CPU.

I think they mean that it's a tool accessing URLs in response to a user request to present to the user live - with that user being a human. Like if you used some webpage translation service, or non-ML summarizer.

There's some gray area though, and the search engine indexing in advance (not sure if they've partnered with Bing/Google/...) should still follow robots.txt.

Yeah, that seems to be a big distinction. If I tell my AI to summarize the headlines from my three favorite news sites every morning, it's just carrying out my request same as if I'd clicked to them, so that seems fine.

But if I say, "Search the web for a low-carb chicken casserole recipe that takes squash and cottage cheese," then it's either going to A) send queries to a search engine like Google, in which case robots.txt already should have been respected, or B) check its own repository of information it's spidered before I asked the question, in which case it should have respected robots.txt itself.

There’s a human using the LLM. In a live web browsing session like this, the LLM stands in for the browser.

Would you believe that humans turn on traditional web-crawlers as well?

Reply on news.ycombinator.com