Hacker News Reader

Hacker News

Comment by RyanShook | Hacker News Reader

Full thread

RyanShook

I’d like to know, what are people on HN using headless browsers for?

View on HN

You’d actually be surprised by the applications, I definitely was. Some I’ve seen:

- Screenshots

- PDF generation

- Functional tests

- Regression tests

- Scraping SPAs

- Crazy stuff like printing user-generated content for personalization products..

Among some of these, I've also built an internal microservice for analyzing URL redirect chains. Input a URL and it will provide me with all of the redirects it goes through and information for each one including method (html meta, 301, etc), timing, etc.

I’ve done some hacky work redirect analysis but just the final url. Thought of open sourcing?

Awesome! What did you use to go through the redirects? I’ve had issues with some browser automation only giving the final url.

I'm using Chrome Headless to analyze user-submitted pages on https://urlscan.io

I will save a screenshot, network request info, request content (for JS/CSS/HTML), cookies, log messages, JavaScript global variables etc. Annotate and correlate it all and you have a nice summary of what a certain website "does".

I am using them to run node-electron hosted WebRTC proxies, to enable UDP-like networking for a real time browser game.

interacting with an app which doesn't provides a public API and has too much frontend code to be worth reverse engineering, I just hack into the app's webpack bundled code and use it

I'm using splash [1] for scraping websites for personal automation purposes.

[1] https://splash.readthedocs.io/en/stable/

I scrape lots of data from services while logged in. Scraping is done with curl, but every so often cookies expire, so I use selenium to login, save cookies, then go back to curl.

Isn't curl fit to also do the login part and save the cookies?

Tons of stuff get loaded during login, hard to go through each one and code the proper urls and POST params etc, quicker to write login code.

PDF generation for us. Had been relying on phantomjs for years, but it has been a bear. Attempted to write something with PDFKit to avoid the overhead of hosting headless chrome, but the effort was too great for our small team. Using chrome we are able to build high fidelity reports with the same web components we use in the application's UI.

I'm using Puppeteer for website / transaction monitoring at https://checklyhq.com. People script their own checks and we run them for you on a cron like schedule and alert when things go off the rails.

Personally? Dealing with cloudflare and other bullshit WAFs that break the internet.

Recording of webrtc video conferencing sessions , pdf generation, spa test suites

https://www.prerender.cloud/ - server-side rendering API for single-page apps

Headless browsers are the de-facto standard for scrapping websites which have some protection layer. Ad-fraud is also a thing.

Automating logging into and data capture for financial/banking sites, building a new finance aggregation service.

Rendering high res vectors (SVG)

Built an DOM 2 png image api (hivis.io)

Personally, for testing.

Reply on news.ycombinator.com