Show HN: Headless Recorder | Hacker News Reader

Show HN: Headless Recorder

github.com

224 pointstnolet23 comments

Hey, this is cool we have a similar free project at https://www.testim.io/playwright-recorder/ - good job! Happy to see players in this space.

OP here with some background info. Headless Recorder started life as “Puppeteer Recorder” which was posted here earlier. However, we just added Playwright support and renamed the project!

Looks amazing, it is generally very difficult to find well-maintained, open source web scraping/robot process automation tools. Thank you for building this and congratulations on the successful execution!

Do you ever run into issues with puppeteer scripts where you do waitUntil: networkIdle2 and every 10-15 page loads it just sits there forever? There's a timeout parameter that you can pass too, but it seems to do nothing.

I seem to have this problem often and I'm wondering how common it is.

I remember this problem. Different framework though. It’s probably something buggy deep in whatever driver library we are still using. Our solution was to not use that function.

I found that I need to listen for error events inside the page, particularly http 401 and 400 and abort if these come up

I'd be curious as to why change from an easily remembered and branded name to a generic one?

Edit: not bashing, genuinely curious

It used to support just generating Puppeteer scripts. They added support to also generate Playwright scripts.

Sticking with the old name would make it sound like they only supported Puppeteer.

This is correct. We want to do more than just Puppeteer, so it made sense.

So it looks like this keys off of class selectors. Anyone know of a solution for cases where you have dynamic selector names? Frameworks like next.js (using styled-jsx) add random strings to class names at build to create classname uniqueness [0].

[0]: https://github.com/vercel/styled-jsx#how-it-works

EDIT: Looks like one solution is to put an ID on the elements you're interacting with. [1]

[1]: https://github.com/checkly/headless-recorder/issues/32

If the DOM structure is consistent, using xpath [1] may be an okay addition.

[1]: https://developer.mozilla.org/en-US/docs/Web/XPath

I imagine in the future we'll have ML-backed tools to give instructions like "Find the submit button", and that they'll be able to account for A/B testing and other things that get in the way of automation.

Maybe in the future we could have accessible controls which are accessible for non-graphical interfaces such as screen readers and automated systems.

Yeah, this is a cool idea. Came across a version of this the other day, though would love to see something open source [0].

The less "smart" way of solving this is to look at the contents of the element like Cypress' `contains` [1].

[0]: https://walrus.ai/

[1]: https://docs.cypress.io/api/commands/contains.html#Syntax

We use https://taiko.dev/ to do E2E testing, and it's basically that. It's very far from perfect but dramatically cuts down needing to use selectors or IDs

SikuliX [1] perhaps? It's an image-based automation tool.

[1] https://github.com/RaiMan/SikuliX1

I’m not familiar with puppeteer. Does this tool let you script a browser with all your logins and cookies intact (to access content behind a login)?

Yes it lets you script chrome, usually headless, but not necessarily. I don't know whether it can use existing cookies in an open browser, but I'd assume so. I use it with headless chrome, and set cookies manually.

Had a similar idea to this glad I can just use what you have created instead

Wait, there's still work to do: a Firefox version.

It generates Playwright scripts which automate Chromium, Firefox and WebKit. Ref: https://playwright.dev/

But Puppeteer works on headless chrome. So I'm not sure the benefit of recording in Firefox. Unless this other platform they now support works with headless Firefox?

I think this actually has a lot of applications outside of QA testing and for my own use cases I don't care as much about cross browser compatibility

Reply on news.ycombinator.com