Hacker News Reader

Comment by adriand | Hacker News Reader

Feels like only a matter of time before these crawlers are blocked from large swathes of the internet. I understand that they’re already prohibited from Reddit and YouTube. If that spreads, this approach might be in trouble.

View on HN

I doubt those crawler rules will be honoured for long.

I wouldn’t even be surprised if a law is passed requiring sites to provide equal access to humans whether accessed directly or via these models.

It’s too important an innovation to stall, especially considering the US’s competitors (China) won’t respect robots.txt either.

TBF OpenAI in particular bought access to Reddit. Otherwise yeah this is my main confusion with all of these products, Perplexity being the biggest -- how do you get around the status-quo of refusing access to bots? Just to start off with, there is no Google Search API, and they work hard to make sure headless browsers can't access the normal service.

They do say "Currently, deep research can access the open web...", so maybe "open" there implies something significant. Like, "websites that have agreements with OpenAI and/or do not enforce norobot policies".

Client-side browsers that crawl for users (and prompt for logins or captcha as needed) won't be as easily blockable

Anyone selling anything would want to remain crawlable if people use this to research something that could lead to a purchase.

Not necessarily. Southwest airlines doesnt allow itself on price comparison sites or Google Flights.

Amazon listings are blocked from google shopping and other price comparison sites.

Your point is completely valid, but... Southwest now has an arrangement with Google Flights to allow their listings there.

they finally bowed mid last year https://www.nerdwallet.com/article/travel/southwest-google-f...

> Amazon listings are blocked from google shopping

I see Amazon results there all the time. 3 of the visible 8 sponsored results are Amazon, in the non-sponsored results an Amazon listing is either first or second in every category.

While people might attempt that, it's going to be an arms race, just like ads vs adblocks. There's already multiple crawlers that present fake user-agent when their original one is blocked. Temptation of more data is just to irresistible to them

I suppose there is an equilibrium, where sites that penalize these types of crawlers will also get less traffic from people reading ai citations, so for many sites the upsides of allowing it will be greater than the downsides.

> Feels like only a matter of time before these crawlers are blocked from large swathes of the internet.

How would you know its a crawler?

This is trivially bypassed by OpenAI asking the user to take control of their computer (or a sandboxed browser within it,) then for all intents and purposes it’s the user themselves accessing your site (with some productivity/accessibility aid from OAI.)

Big Tech Podcast listener?

Reply on news.ycombinator.com