Prototype: Puppeteer for Firefox
github.com
github.com
> Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.
Disclosure: Chrome DevTools docs guy
Then it gets upvoted to a more general audience, who ask "Puppeteer? what's that?"
At the end of the internship I handed over everything I'd built by that point, with the idea that they (Mozilla) would turn it into a finished project. But I never saw any activity on that front after I left, so I assumed the idea'd been scrapped. Feels very weird seeing this pop up out of the blue on HN today.
[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1523104#c1
[1] Complete with documentation comment generation; I found the result of running rustdoc over that crate super useful as an alternative to the official CDP docs site: https://www.spinda.net/files/mozilla/rust-cdp/doc/cdp/index....
Edit: I actually have no right to comment on what stalled your project and have no idea what happened there. I just wanted to discuss the fact that we need a good abstraction layer but seem stuck in debate.
Disclosure: Chrome DevTools docs guy, just speaking personally based on my research into the topic over the last year
I also used to build a desktop app that my team uses to download Alexa Flash Briefing metrics (because there's no API for that)
Puppeteer's full-screen screenshots & DOM manipulation abilities are clutch!!
Sometimes I want to directly control a browser to run a test or examine a UI component, but I still want to mock parts of my application. This is one of the biggest weaknesses with Selenium -- it takes a very narrow view of what testing is, so it becomes much less useful as soon as you're trying to do anything interesting.
Different applications call for different testing strategies. Selenium could be a low-level browser controller that allows you to write strict E2E tests, but instead it's a low-level browser controller that is almost fanatical about only supporting strict E2E tests, even when it means that basic actions need to be more complicated. I think that's a design mistake, but whatever.
If you're building a SPA, you can mock out the backend and control the backend mock behaviour in your Selenium test.
If you want to do something like request interception, you have to point your front-end at a proxy server and mock things there. If you want to test a file upload, you have to simulate typing into the dialog and then use a literal file on the disk. It's just pointless friction for most testing setups.
It's not that E2E testing is bad, it's that in some projects there's room for a Selenium-style tool across the entire spectrum from E2E testing to unit testing; particularly if I'm testing something like D3 code, or parts of a CSS layout.
It's all doable, it's just that Selenium makes it all needlessly difficult, because its point-of-view is that your front-end should be treated mostly like a black box, and that any mocking that does exist should be happening via endpoints and application connectors.
I like Selenium -- I prefer using an Open protocol over Puppeteer. I just feel like that protocol could use a lot more design work.
Selenium/WebDriver isn't a testing tool. It's a a W3C protocol for remote control of web browsers. I think once you get over that misconception, selenium starts to make a lot more sense.
If I'm remote-controlling a browser over a network, wouldn't I occasionally want to send it an arbitrary file upload? Why do I need to separately transfer the file using a different service, and then refer to it with the disk path?
The only reason I can think of is, "normal users couldn't do that." But normal users also can't control a browser over a network. There's no reason to restrict a control protocol to only things that normal users can do.
The selenium maintainers seem to have a very strong opposite opinion.
CasperJS was OK, even better with Nightwatch but Puppeteer is quicker, more stable and actively developed.
I've also noticed that sometimes (in CI, less than 5% of the runs) the installation or run will fail deep in chromium. It's actually spawning a chrome process and tunneling remote commands to the child process to do work, so, I guess the flakiness is (almost) excusable?
Restarting the test always resolves the issue, fwiw.
I too have experienced flakiness in CI where Chrome doesn't always seem to start or quit within a reasonable amount of time. I get around this in production with timeouts that give any lingering Chrome processes a series of increasingly aggressive requests to quit, but my hosted CI environment doesn't like it when I send kill signals to random PIDs.
I'm hoping that this Firefox client will be somewhat more reliable in that regard.
My conspiracy-laden mind veers towards the obvious but I'd prefer a better explanation.
The experiment was a great start to see what is needed to support Puppeteer. The next step is coming up with a well-integrated architecture. Find me on the web to talk to me about your use cases and ideas for Puppeteer.
If you want to record your own scripts, I also am actively developing https://checklyhq.com/Puppeteer-recorder, a Chrome extension.
Checkly looks like a great product!
Love recording macros
Selenium has it's quirks but when you understand how it works and it's limitations and user centric philosophy there is nothing that can stop you from writing rock solid stable tests.
The thing is I have to write these workarounds everywhere. What's more, I inject javascript code a lot to avoid nodes staleness caused by network latency. I believe selenium is not designed for writing lots of javascript. And javascript code in a string of python can not be linted, which increases debugging workload.
To this point, the cost of these hacks is too much in my project. I find puppeteer play nicely with javascript after trying. But the difference and reliability between waitFor() in puppeteer and WebDriverWait() in selenium still remain questionable.
I am wondering if using puppeteer can ease pains mentioned above.
> And javascript code in a string of python can not be linted, which increases debugging workload.
If you use PyCharm or any other JetBrains IDE you can use `Language Injections` [0]
> I am wondering if using puppeteer can ease pains mentioned above.
I'm not sure about puppeteer but Chrome DevTools Protocol [1] (I think this is what puppeteer use under the hood, but I'm not sure) may be interesting for you because it gives low level access to browser mechanisms like injecting code before page load, request interception or separate browser contexts between separate tabs.
[0] https://www.jetbrains.com/help/pycharm/using-language-inject...
I’m seriously thinking there is a cloud api play in the near future — a puppeteer Page is a pretty awesome container format ... a Puppeteer Page or BrowserContext per request would provide an awesome cloud-function-like programming model I think...
If it were me I would aim to merge the firefox puppeteer into chrome puppeteer but fair enough if they want to keep them separate.
EDIT: I guess it depends on how much of the code in the puppeteer chrome codebase is tightly coupled to chrome dev tools. Like if 25% of specific to chrome, then that would mean that 75% of the code can be shared between the chrome implementation and the firefox one, in that case I think merging and hiding the 25% chrome/firefox specific stuff behind an abstraction layer is the best option.
Or did you mean something else?
Back in the Mozilla days i was using Mozilla because it was very powerful and flexible despite being very slow (i remember watching Mozilla 0.6's dialog boxes draw themselves) and many pages didn't work with it because everyone only cared about IE (...and i'd put the RIIR crowd to shame with my "evangelization" :-P). Nowadays i use Firefox mainly out of inertia.
Perhaps if Google disables most of the stuff adblockers rely on, there will be again a reason to use Firefox.
A year or so after the fact, I can't even remember what I've lost. Couldn't have been that important.
Performance is ok but it isn't my main concern, i'm also concerned about features (after all lynx is faster than both of those browsers, yet it lacks a bit on the features side).
In any case, this isn't a hill i care to die on. I just do not see much of a difference between the two browsers anymore in terms of what they can do. I used to like Mozilla for its features (in fact i was really annoyed that they switched their focus to Firefox back in the day and left what they renamed to "Mozilla Suite" to die) and Firefox later for its (remaining) features. Funny enough now that i think about it, the reason was also "performance" back then and again i didn't care about it but i did care about the features lost.
I guess they kept on the same path and eventually Firefox will be a chromeless, featureless HTML terminal - not very useful as a browser, but it'll be the fastest HTML terminal :-P.
In any case, it still has a number of major differentiation points - more capability for extensions is even one of them, see their Facebook Container extension. But obviously the one Google will never be able to copy is a focus on privacy.
Disclosure: Chrome DevTools docs guy
"The Firefox Remote Protocol is a low-level debugging interface based on the CDP protocol."
Chrome has invested massively in developer tools for chrome and it's shows.... I think chrome is the best software ever written. The developer tools are so powerful that it'd hard to even know the full breadth of what's in there.
Firefox offers nothing to developers over chrome, which is sad.
Firefox has multi-account containers. Having different tabs logged in to different accounts is huge for me.
Firefox allows you to have each tab open in a different container, and they don't have to be tied to anything in particular.
I can have different tabs open to the AWS console for each AWS account I use. I can also switch tabs to change between different user accounts to test interactions between users on the site I'm developing.
No. Each profile opens in a new Chrome window. You can have as many open as you'd like.
> and each one is linked to a different Google account?
No. Local profiles have always been a thing.
Chrome's implementation has better UX for privacy IMHO; it's harder to accidentally open a tab in a different profile than expected, which I do _all the time_ in Firefox. In Chrome, Cmd+T opens a new tab in the active profile. In Firefox, Cmd+T opens a new tab in the default profile.
Vimium/cVim just doesn't match up and aren't under as aggressive development. For this keyboard-focused developer, there's not a good enough reason to be on Chrome. I know I am in the minority, but it's a point nonetheless.