CurlyQ: Command line helper for curl and web scraping | Hacker News Reader

CurlyQ: Command line helper for curl and web scraping

github.com

67 pointsingve19 comments

I've been looking to automate some web processes and I think this is just the tool. Brett has a long track record of providing terrific tools for free as well as some top tier paid apps.

I've created a few similar tools for link scraping: https://github.com/chapmanjacobd/library#usage

- library links-extract: extract inner links from pages (stdin, local files, or remote sites)

- library links-add: extract inner links across from multiple pages (paged lists of articles/forums) into a SQLITE database

- library links-update: fetch new items for each link in a links-db (just added this yesterday)

You can use the same filtering across all subcommands. For example you can filter based on text between links:

    pip install xklb
    library links https://en.wikipedia.org/wiki/List_of_bacon_dishes --path-include https://en.wikipedia.org/wiki/ --after-include famous

You can use all of them with the same --cookies-from-browser interface that yt-dlp provides for pages that are locked behind a login.

Alternatively, copy and paste:

    library links --local-html <(xclip -selection clipboard -t text/html) --after-exclude paranormal spooky horror podcast

I wish someone made a similar command line tool that does screen scraping, is powered by AI and could simulate clicks as a real user.

I just finished up a Grub crawler for https://mitta.ai. The result is that you can screenshot a page and then run it through OCR, then push the text into an LLM, or embed chunks of it: https://www.youtube.com/watch?v=QadlImAKzCE

While it won't do clicking, the system also has a processor that can fetch the HTML (or other files). It would be an afternoon's work to add something like CurlyQ to it. I was kicking around adding an ffmpeg processor to it...

We’re working on a tool that “screen shares” with AI and allows both looking at what is on screen when you’re sharing and acting on it (simulate clicks and other). What’s your use case? Happy to share more about where we’re at with the project. It’s quite a lot to build to get right for many kinds of apps and to be multi-os but we’re pretty close. No CLI though, would love to hear more about your use case there. Feel free to email me diamond@augmend.com

My use cases:

1. Fill any form with intelligent data (from orders, through plane tickets to tax forms). Think of ChatGPT but for forms.

2. Research any person and company using various tools (e.g. LinkedIn, Google, job boards). Do it at human speed so I won't get blocked and can observe it on a screen. Provide me with a summary of text and list.

3. I want to be be able to script it and hack to whatever I need. API and CLI.

> I wish someone made a similar command line tool that does screen scraping, is powered by AI and could simulate clicks as a real user.

How much are you, personally, willing to pay for such a tool?

Because, minus the AI[1], I'm pretty certain that such tools exists at different price points.

I'm not sure what the AI would do for such a tool, but I'm sure that the companies making such a thing are busy working hard to incorporate some form of AI into it.

[1] What would the AI in such a tool do?

The brainer price would be $20 / month. $50 / month if it was terrific.

The trick has to be really useful without too much work on my end. Combine screen scraping and simulating real users with some mix of AI.

My use cases:

1. Fill any form with intelligent data (from orders, through plane tickets to tax forms). Think of ChatGPT but for forms.

2. Research any person and company using various tools (e.g. LinkedIn, Google, job boards). Do it at human speed so I won't get blocked and can observe it on a screen. I can provide a script on how to do it (e.g. check LinkedIn, this board for that, etc.) Provide me with a summary of the text.

3. I want to be able to script it and hack it to whatever I need. API and CLI.

> [1] What would the AI in such a tool do?

Solving captchas. In particular, not only solving the challenge itself (recognizing crosswalks, etc) but also handling input (moving the mouse, alt+tabbing, etc) as if it were a real human.

I was recently reading the Cloudflare turnstile announcement https://blog.cloudflare.com/turnstile-ga and it has the following links about this

https://uncaptcha.cs.umd.edu/papers/uncaptcha_woot17.pdf

https://www.vox.com/22436832/captchas-getting-harder-ai-arti...

https://arxiv.org/abs/1903.01003

In those excerpts:

> The prevalence of free speech-to-text services has made it easy for bots to solve audio CAPTCHAs as well, with a recent study showing bots can accurately solve audio CAPTCHAs in over 85% of attempts. We’re proud to state that Turnstile is WCAG 2.1 Level AA compliant, while eliminating the need for audio CAPTCHAs as well as visual ones.

And

> With all of our emphasis on how easy it is to pass a Turnstile challenge, you would be right to ask how it can stop a bot. If a bot can find all images with crosswalks in grainy photos faster than we can, surely it can check a box as well. Bots definitely can check a box, and they can even mimic the erratic path of human mouse movement while doing so. For Turnstile, the actual act of checking a box isn’t important, it’s the background data we’re analyzing while the box is checked that matters. We find and stop bots by running a series of in-browser tests, checking browser characteristics, native browser APIs, and asking the browser to pass lightweight tests (ex: proof-of-work tests, proof-of-space tests) to prove that it’s an actual browser. The current deployment of Turnstile checks billions of visitors every day, and we are able to identify browser abnormalities that bots exhibit while attempting to pass those tests.

So, this is a cat and mouse game, the cat is using AI, the mouse is probably using AI in some form or another too.

We really are gonna end up with servers and clients evolving AI challenges and solvers respectively to the point where humans won't be able to solve them anymore haha

I can build it. Out of curiosity, how much would you be willing to pay for such a tool?

> I can build it. Out of curiosity, how much would you be willing to pay for such a tool?

You beat me to it by just a few seconds! :-)

$20-50 / month

Though I'm a picky and very demanding user who likes to use API, etc, you have been warned :-).

Coincidentally, my current side-project is a tool to read, and then search, HTML input.

Searches expressions are a subset of the ones accepted by `querySelectorAll()`.

Well, you can't say that and not link the project!

> Well, you can't say that and not link the project!

I wish I had something to link, but I only started it yesterday and it's just a single file right now.

Will maybe post a broken version in a comment later today. I've time-boxed this side project to not more than 2 days, and today it's day 2

Well congrats on starting it! I often don't start the things I have in mind.

Can you use it to do something useful already? Then, if you don't have further plans, consider committing and pushing to Github. Two files is enough, one for the thing, another for the license.

> Can you use it to do something useful already?

Unfortunately not. There's three steps:

1. Parse HTML into a tree.

2. Compile the search queries into a binary form.

3. Apply the compiler query onto the tree.

Steps #1 and #2 are done (with tests). You can see it here: https://gist.github.com/lelanthran/896a2d1e228d345ecea66a5b2...

Unfortunately, now it is the weekend, and with kids around I may not get around to step #3 until around Sunday :-(

Thanks for sharing.

Reply on news.ycombinator.com