Show HN: LLM Scraper – turn any webpage into structured data

Great work! One of the things that would be incredibly useful/interesting would be generating a reusable script with an LLM, instead of just grabbing the data. In theory, this should result in a massive cost reduction (no need to call the LLM every time) as long as the source code doesn’t change which would make it sustainable for constant and frequent monitoring.

The biggest trick here is going to be costs. I have gotten scary openai bills feeding websites into gpt-4 due to cost scaling with content size.

Interesting, so it takes a screenshot of the page in playwright and then asks the LLM to parse the image and find the values corresponding to the keys in the schema? How expensive is it to run it per webpage? Does it sometimes hallucinate, and if so, have you tested how often?

The LLM Scraper seems like a highly useful tool that can transform any webpage into structured data. This could be a significant advancement for data analytics and automation processes. I'm looking forward to seeing its practical applications and effectiveness.

Operating modes are input, yes? Handling JS sites would be a huge improvement. Part of your plans?

Great work! I’ve worked on the same problem and used LLM to extract feed into structured data (in my case have to use a more affordable model like GPT3.5 for a Saas app, looking at llama3 now)

Have you thought about automatically extract schema?

Oh neat, I'm working on the same thing with python and playwright.

I'm finding that the latency with web LLMs is a pain in the ass and hoping to switch to llama3 after I get that set up with function calling.

The problem is not the scrapping with llm. You need to solve the underlying problems like antibots, captcha, all these issues prevent the scraping at scale

This is so sick haha and I'm loving this new norm of providing local options by default :)