mrskitch

I enjoyed reading the first few chapters, but wanted a PDF as well. I run browserless.io, so threw a quick script together to PDF-ify this. Gist is here: https://gist.github.com/joelgriffith/0e6a2a774317e845206ca8d....

You can pretty much just create a free plan and run this in our debugger, which will compile the chapters and return PDF. Pretty fun little challenge.

This seems decent in Theory, though I'm not familiar if GraphQL mutations can handle conditionals. Cool idea though, but that might be a deal breaker for a lot of folks

Thanks for that -- we did have quite a few good conversations over emails. I did read each personally and respond to them. Didn't justify the means.

Hey folks, Joel here, I'm the original maker behind browserless. You've seen us before on HN -- I'm very sorry for this situation.

We didn’t intend to SPAM you or send unsolicited emails, we just wanted to ask for feedback. Being an open-source/boostrapped service, feedback is really important for us, so that’s why we thought it might be a good idea to reach out to people directly. But now it's clear that that was a wrong decision. We stopped doing that and won’t do it again.

Sorry, it won’t happen again.

Wanted to say that browserless was on this podcast recently, and was great experience. I’d definitely recommend others do it as well

Just JS for now since it compiles and runs entirely in the browser

You’ve got it: the core service listens for inbound Upgrade http requests, starts a browser, then funnels the connection into the browser.

This debugger simply sits on top of all that, and puts the code/execution context in the browser versus the server. Cloud flare is simply detecting that our server IP is a known headless chrome instance, and is serving their bot detection check

You can use our demo debugger (the address is in this blogpost). It might not match your version of puppeteer exactly, but it’s a close enough proximate that it’s still valuable

Yup, There’s a lot going on here. Currently the tool uses a fixed IP for the running browser. That’s why you’re seeing that Cloudflare issue.

As far as the hovering goes, the canvas element is “mirroring” interactions back through to the underlying page. When Devtools are active, this triggers chromium to render hover effects in its GUI. This then gets sent back to the canvas element in the debugging page.

It’s a lot of network traffic and Synchronization... but once everything is setup it works fairly seamlessly

Sure! Puppeteer is a node-based library, and pretty much all the web-apps out there that let you run puppeteer code do it in an elaborate node sandbox. This tool gets around that by running puppeteer in your own browser, making it a lot faster and more secure

Just familiarity with puppeteer, eventually this tool will support both. Puppeteer has less “moving parts” for now

You can actually install our docker image or use the npm module:

https://github.com/browserless/chrome/blob/5627f1ef041ec23f3...

Hey, Joel here, mostly responsible for this tool. Happy to answer questions — one thing not well covered is getting puppeteer to run in the browser, especially a webworker. Can talk more about it if there’s interest!

This is exactly the same thought I came to with browserless.io. There simply wasn’t enough traffic to make informed decisions, and when there was it was really silly things (small copy changes and the like).

Eventually we just tore it all out, and never looked back. Improving the product and blogging about our findings are a win-win for us and the ecosystem at large, versus agonizing over traffic and data

> But from a legal perspective that is not even close to the same thing.

Very true. However, my rebuttal would be that the majority of folks reading this are of an engineering background, and seeing the words “no longer open source” can land quite differently.

> This means that Elasticsearch and Kibana will no longer be open source software.

This is categorically not true. The source is open, and will likely always will be. It’s not free for AWS going forward, however. Why is it that Amazon has such a hard time paying for stuff they use and commercialize? There’s no issues here with other providers (GCP, Azure, etc.), so clearly the problem lies with them. While they’re at it, they should also get off this “open and free” high horse they seem to be on. A few patches here and there don’t qualify as big time contributor status. If they want to show that they’re committed, how about the release their infrastructure code that runs all their services? That’d definitely go a lot further than “big bad Elastic changed their license and we’re defending users.” Get outta here with that nonsense, history shows otherwise with all the other tech that’s been ripped off.

I also don’t get all the criticism for Elastic doing this. They own the software, and they can do whatever they want. Should they have done this license from the start? Maybe, but it’s not exactly easy getting a project off the ground without some way to gain attention. If you’ve got no users, you’ve got to show at least what your code is doing, and picking software licenses is not exactly a straightforward task. They changed their license to fight back, and it’s entirely within their right to do so.

Hate to feel like I’m venting, but AWS is being the bully here and feigning that they’re pro-user, which is frustrating to witness.

I think dual licensing is possibly the best way to go, if your product mitigates a risk. It's why we went that way with browserless.io

Truth be told selling a small library probably isn't enough to make significant money on your product since it isn't solving a big enough risk. The risk of forking is fairly low, especially for small to mid sized companies. Larger companies, maybe so since things like security become much more urgent.

It's hard to separate people from their money, but I feel it becomes a lot easier if it offsets a potential risk they're not willing to take. Just my 2 cents.

I’ve seen some of these trends in browserless.io as well. Good thing you went full time back in January! I postponed going full time on browserless until just a few days ago.

I’m still a bit worried about how next year will shake out. Even though there’s some stimulus in place, there’s a cost to them that’ll eventually come to bear. Perhaps not next year, but years after.

I guess there’s always something going on. Good luck and congratulations on your success! Keep listening to your users, and keep innovating!

I suppose it's open core. Open Source is hard because you have to separate people from their money even though the code is "free." I'm not saying it's _the_ answer, but browserless.io isn't VC backed, and it certainly makes more money than most of the Patreon/sponsor models. I understand that it's not just a library, but a full-blown service, which makes it easier to market and sell.

Have you considered a hosted product? I have no idea what your audience is like, but that seems to be where we've had a lot of success. Happy to chat anytime as well: joel at browserless dot io.

My own opinion: I think there is as a marketplace. Years ago when I started browserless.io, I wanted to find a way to sell access to the core image of ours. Docker kinda has/had a marketplace, where you can buy access to curated and secure images, but it didn’t get any support or news on it. Because of this we went the open-code route and just sold licenses. Too bad because I’d much rather have had a marketplace to do this instead.

Wow this is almost identical to what we use at browserless.io. I’d be curious to hear how switching SSL termination went as that seems to be the hardest “reproducible” part of it, and copying over certificates is kinda scary.

The only thing I think we do differently is nginx’s amplify SaaS product for monitoring load balancers. Those are still kind of painful, but work well with WebSockets, which is hard to find support for.

Thanks for sharing!

This is how we framed it at browserless.io. Libraries generally aren't a product, and even though some of them do a lot, it's hard to charge folks for an `npm i`-style product. I tried a lot to look into this and it just didn't pan out. I think there's an opportunity for a library marketplace that does all of the curation, security checks, and other validation as it would provide a lot of value to organizations so they don't have to build all this themselves. Seems like all of this already exists, just not in a concise package.

On that, I feel like docker could provide a much much better storefront, and charge folks for their packages, but just hasn't. Did you even know they had a store? That's what their hub is supposed to be, but they don't charge for anything! How great would it be if they built out a marketplace?! This has even more potential, IMO, since images are generally a whole service and not just a library.

The opportunity and value definitely are there, and the need is there, it's just not something that anyone has tackled in a way that makes sense with software.

Yes: often the case is that JS does some kind of data-fetching, API calls, or whatever else to render a full page (single-page apps for instance). With Github being mostly just HTML markup and not needing a JS runtime we could have definitely gone that route. The rationale was that we had a desire to use our product ourselves, to gain better insight into what our users do, and become more empathetic to their cause.

In short: we wanted to dogfood the product at the cost of some time and machine resources

Thanks for the mention! I'm the founder of browserless.io, and agree with pretty much everything you're saying.

Our infrastructure actually does procedure for some of our scraping needs: we scrape puppeteer's GH documentation page to build out our debugger's autocomplete tool. To do this, we "goto" the page, extract the page's content, and then hand it off to nodejs libraries for parsing. This has two benefits: it cuts down the time you have the browser open and running, and let's you "offload" some of that work to your back-end with more sophisticated libraries. You get the best of both worlds with this approach, and it's one we generally recommend to folks everywhere. Also a great way that we "dogfood" our own product as well :)

Hey, I work on browserless.io, which supports both puppeteer and selenium.

Selenium uses a chatty HTTP interface, whereas puppeteer/playwright use WebSockets or pipes to communicate. Under-the-hood, however, Selenium is simply using chrome's devtools protocol to communicate with it. The way selenium does this is by another binary, generally a `driver`, that has the protocol "baked" into it and has the HTTP selenium API as its input interface.

This is all a long way of saying that puppeteer/playwright have a lot less moving parts, and are generally more approachable. Selenium _does_ have a lot more history behind it, better support across languages and frameworks, and is more stable but it's also much larger and "clunkier" feeling. It's also a lot harder to scale with load-balancers since, again, it's all over HTTP so you'll need some way to load-balance with sticky sessions.

Practically speaking they all do the same thing at some layer. Both are high-level APIs around the devtools protocol, it's just what higher-level interface you prefer and what your language/runtime is.

Thanks for the mention (browserless.io)! Really enjoying this site, let me know if you'd like a guest post and I'd happily add something. We've gathered a few weird tricks and tips as well. Definitely a lot that can be shared!

Yeah, you can definitely think of it like that

Hey folks, Joel here (creator of Quik + browserless.io). Quik came about as an attempt to stop spending time on IE support... but still support it when you have to. A vast amount of my "frustration" time as a developer was getting VM's to run IE to even test the site, let alone do the required repairs. Huge time-sink.

Anyways, happy to answer questions or talk more about it. Pretty fun/novel way of solving this problem!

I pretty much stole Mike's business model, and peppered in some cloud-y saas for browserles.io. I'm almost done with year 3, and will hopefully hit 500kARR this year.

Mike's story was a lot of inspiration for me, and I think licensing + cloud-ifying your opensource projects are a ton more viable than hoping for Patreaon or other platforms to pay you. Self-bootstrapped indiehacker style was great for me, but it does require a lot of up-front time and commitment in seeing it through. Upside is that it's not monopoly money being dealt with, and everything is on your terms.

Thanks for that! Appreciated! I have another person help with support now, but I really enjoy the dev stuff so I jump on those. It’s what got me here in the first place :)