Web Scraping Using ChatGPT – Complete Guide with Examples | Hacker News Reader

Web Scraping Using ChatGPT – Complete Guide with Examples

proxiesapi.com

Side note: Add this to your robots.txt if you don't want ChatGPT to scrape your website:

  User-Agent: ChatGPT-User
  User-Agent: GPTBot
  Disallow: /

Sources:

- https://platform.openai.com/docs/gptbot

- https://platform.openai.com/docs/plugins/bot

Well this has little to do with the article here, ChatGPT is not accessing the website itself, I even think that this feature is disable at the moment.

They opened browsing feature for Plus and Enterprise users 2 days ago.

Side note to your side note: the official beta ChatGPT browser plugin’s user agent is Python/3.9 aiohttp/3.8.4 (I just tested it on a domain that I control)

And like a sibling comment says, any plugins/agents that are trying to scrape will have various fingerprints.

If you're feeding the specific nodes to scrape, iterating and bug fixing what is Chatgpt doing other than giving you someone to talk with while you code?

Call me when I can ask a LLM to pull structured data in CSV form from website X and deliver it to me each morning. And it does it.

This doesn't seem too difficult of an ask. Let me see if I can scrape something together.

https://agenthub.dev - not my project (but I have a similar project in my bio if anyone is interested)

Reply on news.ycombinator.com