Chrome 59 has cross-platform headless support | Hacker News Reader

Chrome 59 has cross-platform headless support

chromestatus.com

834 pointsvvoyer154 comments

This is fantastic. I'm using a combination of Chrome and PhantomJS for karma testing right now, for https://github.com/paypal/paypal-checkout and https://github.com/krakenjs/xcomponent. There are hundreds of tests opening up hundreds of iframes and popup windows, and sending a lot of cross-window messages, and that ends up being really memory hungry.

Chrome deals pretty well with garbage collection, so long as I'm careful to de-reference closed windows properly¹, and only uses a maximum of 150mbs. PhantomJS eats up almost 6GB of memory before it's done, which makes it almost unusable on machines with less memory or CI boxes. Travis is a no-go.

I'm hoping running Chrome in headless mode should give a nice speedup for our tests.

-----

¹ Turns out even a closed popup window or iframe keeps a huge amount of memory hanging around. Who knew.

I've been using post-robot and xcomponent a lot recently as I'm solving a bunch of similar challenges. Just wanted to say thank you for sharing your solutions and expertise.

Glad they're working out for you! Let me know if you encounter any interesting situations they don't support.

What do you do to ensure the resources for closed windows are released?

A combination of:

- Making sure all promises are fulfilled or rejected, so window objects don't get caught indefinitely in closure scope for any .then() or .catch() handler functions.

- Using WeakMaps as much as possible, when we have things that are tied to a particular window, like message listeners or response handlers in post-robot

- Manually clearing up any global references to windows when we destroy an xcomponent instance

Finding the references was the tricky bit. A lot of the effort was finding a leaky test-case, running in 100 times in succession, and deleting code until the memory graph was flat -- then figuring out what I'd just deleted that caused the leak.

The problem started manifesting as I added more and more tests -- so now I'm actually checking my tests' memory usage on the fly and failing if they cross a threshold. Hopefully that should avoid getting into this kind of sticky situation ever again.

https://github.com/paypal/paypal-checkout/blob/master/test/t...

We also did a POC with PhantomJS and found similar issues, as well as generally flakiness causing too many false negatives. Ended up not using it; I'm hoping this can simplify things are give something more solid to build on.

Same, ended up using ChromeDriver + XVFB on Jenkins. Works but it's still pretty slow.

We (Chrome) have reached out to PhantomJS to inquire if they're interested in collaborating: https://groups.google.com/d/msg/phantomjs-dev/S-mEBwuSgKQ/tU...

The DevTools Protocol is the primary API for headless Chrome, but we are excited for higher-level abstractions like PhantomJS & NightmareJS's API to manipulate the browser as well. Plenty of details to work out, but hopefully sometime this year you'll get a drop-in solution for some of your testing to upgrade from Phantom's older QTWebKit to the latest Chromium.

Thank you for being willing to work with them and not just rolling your own version of Phantomjs. I hope they respond positively to your inquiry

It was my understanding that Google's crawler literally is Chrome. Does Google have any plans to open source those parts of the browser to make integration easier? Maybe I was mistaken

That seems unlikely... It would seem they have two sets of crawlers... one that does typical/advanced scraping, and another that runs the JS (Chromium) and takes DOM snapshots. This is reflected by changing certain properties (window title etc) and seeing them reflected in Google's search results. A couple years ago, the lag was several days to a week behind on new content via as-rendered from the server vs. via JS.

Interesting! I find it puzzling that insider knowledge on how google search works never seems to leak into the public domain. Is google killing leaks in their search algorithms or do they pay the search team such ungodly amounts of cash that nobody has ever left?

Well, the point of their engine is to make it harder to game the system... everytime someone figures out a trick, someone takes advantage of it... that alone would discourage leaks.

We laypeople can only find out things via word of mouth or observational tests on assumptions.

> The DevTools Protocol is the primary API for headless Chrome

Why not the WebDriver protocol? That seems to be exactly what it's intended for…

WebDriver protocol has limited APIs. For example, no networking status. (WebDriver rejected to implement it, they told you to use Proxy for debugging)

A few hours later: https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuN...

After his announcement, Vitaly answered to the team: https://groups.google.com/d/msg/phantomjs-dev/S-mEBwuSgKQ/PQ...

> >Is there interest on your side in adopting Chromium as a runtime? There's some existing documentation [2] around the API and embedding, but admittedly, this would be some work.

> We are interested. But I am afraid not in the current state. Currently, PhantomJS heavily relies on Qt and QtWebKit. It's not that easy to adopt Chrome as a new runtime.

> But I think we could implement PhantomJS as a completely new (with the same API) project that will use Chrome - Phantomium!

Also see https://github.com/Vitallium/phantomium

We're using xvfb and selenium* for testing and a proper headless support would be a hundred time more stable than the self restarting framebuffer can't wait to move to headless chrome

*Yeah I know phantomjs is more cool these days but phantom doesn't support windows height so there's that.

+1 for xvfb

I replied to you on Twitter, but for the sake of discussion, there is a third option that may be worth exploring - using Chrome running in virtualized Windows Manager - https://www.alexkras.com/running-chrome-and-other-browsers-i...

Very cool. Gonna have to check this out.

Hello, former SDC coworker!

The official Selenium Docker image uses the same technique to run headless Chrome / Firefox.

From there you can just run

  docker run -d -P selenium/standalone-chrome

and you'll get something a lot more lightweight than spinning up a new Vagrant VM.

But can you run headless chrome on Chromeos? if not why not?

I've been testing Chrome headless extensively for the past few months, and while it's a good step, but it's not stable for high-volume or even diverse set of webpages.

Memory usage is pretty high, lot of heavy webpages result in crashes/hangs, there are many inconsistencies between features available in full version and headless, their debugging protocol has different APIs that work on headless/non-headless in Linux or Windows, and so on.

Of the bugs I've submitted, some have been fixed in the upcoming M59, so other critical ones may take longer due to their backlog. I suppose for now (maybe until M61-62), Chrome full with xvfb or even PhantomJS are better options. When you realize that Chrome is about the same size (by LoC) as the Linux kernel [1], you can't help but wish for a leaner & faster headless browser.

There seems to be some work going on building Firefox pure headless as well. Great overall, as long as all the browsers try to follow the RemoteDebug initiative [2].

[1] https://josephg.com/blog/electron-is-flash-for-the-desktop/

[2] http://remotedebug.org

Unfortunately, I have to agree, though they have been very helpful and vigilant on the bugs that have been filed. Most of mine have been fixed in a few days.

Hey James, I did come across some of your bug submissions, and our joint discussions on the headless-dev Google Group.

Yes, this exactly. I wrote Crabby [1] a few months back to schedule automated page testing using Chrome and webdriver but doing anything automated with Chrome is really atrocious. You can't expect to load more than one page every 10-15 seconds on the average 8GB instance and it occasionally crashes or otherwise stops working completely.

I ended up writing a simple check using Go's net/http library to do basic performance profiling but it doesn't measure DOM loading like the Chrome checks do. Such a bummer

What I really want is an easy, cross-platform way to collect the network timings for each object like you get in Chrome's dev tools network waterfall graphs.

[1] https://github.com/chrissnell/crabby

I've been successfully using Chrome headless in a 500MB Docker container for dumping the DOM for https://www.prerender.cloud/ for months (rendering a large variety of sites without restarts for weeks at a time)

Run it with:

  --js-flags="--max_old_space_size=500"

to force the VM to keep it GC'd below 500

Chrome v55 was a 30% memory savings, before that I used 1GB containers.

It's not perfect, but I am definitely pushing high volume (multiple tabs, concurrent activity) and I am not having any significant stability issues and I am pushing diverse sets of webpages.

What the best strategy that worked for you on high volumes, multiple tabs or multiple docker instances? I am wondering if multiple tabs is as efficient as multiple windows/instances.

Thanks.

I'm using more than one Chrome process so I can kill the processes every so often (e.g. after timeout or when they get stuck). Inside each Chrome instance I use 16 tabs, there might be a number of factors at play:

- Are you worried about same-origin pollution if you run multiple tabs from the same origin in the same process? If so -> Extra process - Do you have to take screenshots? You can only take screenshots of the tab that's in the foreground, so you have to activate it first to take the screenshot. This might fail if you have lots of tabs which roughly trigger at once.

You can see what I've built at https://urlscan.io btw.

That ought actually be:

  --js-flags="--max-old-space-size=500"

Hyphens not underscores.

A list of V8 flags can be found with:

  --js-flags="--help"

I think it may accept both the underscore and hyphen and just normalizes it.

This is correct, it does accept both, and the help lists it as underscores not hyphens. So the comment about it ought to be hyphens not underscores is incorrect. Either is fine.

When I went digging through the code for v8 I believe I saw examples of it both ways. I'm not dead sure though because I don't know very much C. I just checked because I was hoping they weren't actually using an invalid flag for all that time.

what Docker image are you using?

Thanks for that information. I've been putting off moving over to Chrome headless for https://urlscan.io and haven't had the time to do extensive testing yet. Right now Xvfb works fine for me. Still, I'm running 3 Chrome instances with 16 tabs each in parallel and have to kill the processes every so often because they get "stuck".

https://bitbucket.org/lindenlab/dullahan

I've been working on a fully open source Windows/macOS library (via Chromium Embedded Framework) that allows you to render pages to memory (and then of course to bitmaps, textures etc.) as well as inject synthesized mouse/keyboard/JavaScript events. It currently uses (what amounts to) Chrome 57.

Looks like this might make my project obsolete.

Thank you anyway for your efforts; I hope you interpret this as a validation of your having correctly identified a sorely-needed tool.

Thanks keville - I've poured my heart into this on nights and weekends after I'm done with my day job for a long time now but what you say is so true - hopefully we'll get access to a more robust solution compared to my modest hacks.

A lot of phantom js talk here makes me want to recommend http://ghostinspector.com

It's a phantom js (and other headless browser) web service. Using the site, you can quickly create different tests, scheduled tests, chained tests, keep screen shots, create videos of multi step tests, and have historical information of it all.

Can't say enough good things about the site.

Edit: also there's a great chrome extension that will record your mouse clicks and keyboard commands to make creating a test that much simpler.

I'm the co-founder of a startup that provides an enterprise solution in this space (https://functionize.com).

I agree with your sentiment regarding these types of tools. It has been a long time coming but this release and the tools available now are things I wish I had years ago when I was building my first company.

+1 to GhostInspector; I used them at a previous company a few years ago, and it was very useful.

They were just starting, but service was rather reliable, and their tech support was excellent (maybe because we were early customers). We used to run a bunch of automated tests for monitoring and compliance, archiving hourly screenshots over different builds for later comparison.

Also checkout https://github.com/cyrus-and/chrome-remote-interface for an easy way to fully control those headless instances

If you're using Go, https://github.com/knq/chromedp

It's also worth noting that the 57+ series has a nice embedded viewer you can use to view the actual viewport via devtools.

If you want great code completion for chrome's large devtools api with lots of different kinds of object then https://www.npmjs.com/package/chrome-remote-debug-protocol

It dynamically generates Typescript definitions for intelligence and type checking from their protocol.json files.

Vscode chrome debugger uses a fork of it.

I'm the author. /shamelessplug

WOW. That's fantastic. As I noted in a comment above, I wrote a tool called crabby that uses Selenium and Chrome headless to do automated page testing and to report results back to metrics engines like Graphite, Datadog, Prometheus, and Riemann. The biggest problem I have is the unreliability of chromedriver and the extreme resource consumption of Chrome + Selenium. It's really too much for your average public cloud instance if you want to test any more than, say, one or two pages a minute.

Do you know if chromedp can access any of the timing measurements?

Yes, you can access everything via the underlying APIs. chromedp is a relatively new project (only about 4 months old), so there isn't much yet in the way of high level timing / profiling, but we hope to add that to the code base when we have some bandwidth to do so.

Can you detail what is the embedded viewer and how/when you access it? Thanks.

With either Chrome in headless mode, or "headless_shell" (a minimal Chrome app part of the Chromium source tree), you first enable the remote debugging port (via --remote-debugging-port=9222), and then you can then simply browse to http://localhost:9222/. That web page will list the various Chrome "pages" (ie, tabs) which you can then click on. Clicking on those tabs will open the Chrome DevTools inside of Chrome, as a web app served from http://localhost:9222/.

This is the internal API that DevTools uses, and is what is referred to as the "Chrome Debugging Protocol" (ie, chromedp). Since 57+, the built in DevTools UI displays whatever the active viewport Chrome "sees" using the screencast APIs. It's just a PNG that's updated every couple hundred milliseconds with the output of Chrome's headless renderer.

Thanks, also did not quite understood what exactly was headless_shell: how to run it, pro/cons, when to use it versus chrome headless.

The documentation seems to be the source at this moment: https://chromium.googlesource.com/chromium/src/+/lkgr/headle...

Its a binary that I suppose runs the Chrome in headless mode, supports some command line options like --screenshot to take screenshots, etc.

I'm having a hard time understanding why its hanging on some runs, and how --timeout and --virtual-time-budget could help me with this.

If you're using c#/dotnetcore https://github.com/BaristaLabs/chrome-dev-tools

Will generate a project file you can use in your own solution. Generated protocol is customizable via mustache templates too

The feature I was most interested in when they announced this last year was virtual time. The Developer Resources link has it listed (https://chromium.googlesource.com/chromium/src/+/lkgr/headle...) but it's a broken link, unfortunately.

Mostly, I'd like to know how the control of the virtual time system would be exposed. Would it be through the C++ API, or could it be made available through the debugging protocol?

Any way of scripting this to automate button clicks etc? I use PhantomJS for this now but found it to be incredibly unstable for complex pages.

You can use ChromeDriver, the same as you would for headed Chrome.

Yes: https://github.com/cyrus-and/chrome-remote-interface/wiki/Tr...

But ultimately selenium and webdriver.io will do a better job at this

There is an excellent library called nightmare (based on electron). If you need print-screens you can use xvfb

or wait for this issue: https://github.com/segmentio/nightmare/issues/224

it should be noted that nightmare isn't safe for untrusted websites: https://github.com/segmentio/nightmare/issues/1060

If you're using Go, you might want to check out https://github.com/knq/chromedp ... There is also a similar package for NodeJS. Otherwise, you can use Selenium in other languages.

For complex pages, there is Kantu, which is based on Chromium: https://www.a9t9.com/kantu/web-automation

Not headless, but ideally suited for automating complex websites (date controls, etc).

Shamelessly bumping my project to produce a nice python API for the Chromium/Chrome-remote-debugger-protocol: https://github.com/fake-name/ChromeController

I'm trying to replace PhantomJS in my infrastructure with chromium. Not having to build my own chromium will be a very nice thing.

I've been trying to test audio and video with headless browsers (namely PhantomJS) but have experienced extreme difficulty, I wonder if headless Chrome is able to support/supports already HTMLAudioElement or HTMLVideoElement or any media interface that would make, for example, testing YouTube or SoundCloud embeds easier.

Related: I want to take screenshots of a few news websites for a little fake news project of mine, and most approaches return something completely different than what I'm seeing when I open Chrome.

Limited height would be better/ok (something like the first 3000 pixels).

Low volume / can be slow (30 seconds would be ok).

Those news websites many times have infinite scrolling.

I've tried:

- phantomJS (rendering sucked, tried every technique I could find to wait for JS to load)

- wkhtmltopdf (almost ok, generates a huge 30M image with all the height, no antialiasing it seems)

- https://github.com/gen2brain/url2img (this was the best so far, uses Qt bindings but not the latest version)

- actually run a headless browser in DigitalOcean with xvfb-run and take a screenshot: I failed at this

What I didn't tried was Selenium, because it seemed even harder.

How would you guys do it?

Create a chrome extension for yourself that automates the process

I use xvfb for selenium testing. It is really easy to set up and can take screenshots or videos while automatically browsing websites.

Thanks - I tried Selenium (on the desktop) with geckodriver now and it rendered well. The only thing is that long screenshots didn't work but there is probably a workaround for that.

Any chance for webgl here? Would be nice for automatic screenshots and webgl tests.

It is being discussed.

https://docs.google.com/document/d/1VTcYz4q_x0f1O5IVrvRX4u1D...

WebGL is supposedly a first class citizen of the browser and is used in a ton of pages, but it gets left out or deferred in many tooling packages. There are many tools that are nearly useless to us because they don't support WebGL. It's disappointing.

Oh my goodness I have been waiting for this day for a while - we ran into PhantomJS problems with keyboard/mouse eventing and the HTMLVideoElement for testing, this sounds like it should be the cure for our woes of having to hack around PhantomJS's deficiencies.

Nice!! Would creating WebRTC data channel be possible?

server side SCTP to client(p2p over SCTP/data channel) would be cool.

It's already possible. I'm using webrtc and node-electron to connect a golang server for my MMO project. I have a farm of 4 nodejs processes running under tmux acting as proxies for unreliable communications.

https://www.emergencevector.com

Which WebRTC library are you using on the golang server? Last time I looked, I couldn't find anything stable.

I am using electron-webrtc. You need xvfb installed. I was able to get this running under node, but only running the processes under tmux. You will also need to install some random shared libraries Chromium needs, but which node-electron doesn't install, but these are obvious from error messages.

Then I'm using simple-peer on top of that. There's also a library for UDP communications from the node process to the golang process.

What does this mean for Electron and other apps that depend on Electron?

There is an issue on headless here: https://github.com/electron/electron/issues/228

I am looking forward to cli tools powered by electron. Imagine the beauty of a grep writting in js, executed by a chrome binary. :D

I mean honestly Electron running React-CLI with React-Arg-Router is the only way to write a cross platform CLI tools these days.

Imagine the terrible performance of grep written in js, executed by a chrome binary.

Would this run as a chrome driver for Selenium? What is needed to make this work with Selenium?

It should be possible, but as a browser rather than a driver so you'd still need Chromedriver to glue Chrome and Selenium together. In Karma with karma-chrome-launcher[1] you can pass options to the browser using the flags option, eg;

customLaunchers: { chrome_headless: { base: 'Chrome', flags: ['--headless'] } }

I've not tested that yet but I can't really see a reason why it wouldn't work.

[1] https://github.com/karma-runner/karma-chrome-launcher

Yes, it does. Tried it out with our Selenium based test suite. Just have to pass in the `--headless` switch to Chrome

Almost one year old but there was a talk on headless chrome at the Blink conference (BlinkOn 6):

Video: https://youtu.be/GivjumRiZ8c

Slides: https://docs.google.com/presentation/d/1gqK9F4lGAY3TZudAtdcx...

More links: Headless Chrome architecture: https://docs.google.com/document/d/11zIkKkLBocofGgoTeeyibB2T...

Mailing list: https://groups.google.com/a/chromium.org/forum/#!forum/headl...

All of those links are on https://chromium.googlesource.com/chromium/src.git/+/master/...

Fantastic news, not a minute too soon! Can't wait to get rid of PhantomJS. Now if only this was a standard feature of all browsers...

Is it possible to install Chromium on a server without X environment? Last time I checked it requires a shit ton of dependencies.

PhantomJS has plenty unsolved issues(up to 1.7k+), a replacement instead of combination might be a better choice.

How fast is headless vs. normal? According to

https://developers.google.com/web/fundamentals/performance/c...

the chrome browser spends a decent amount of time on other steps such as parsing HTML. I wonder how much time could be saved by not rendering pages into pixels.

This page doesn't load for me (IE behind corp firewall). How does one drive/automate a headless browser? What kind of API is there?

Docs and a Youtube presentation...

https://chromium.googlesource.com/chromium/src/+/lkgr/headle...

https://www.youtube.com/watch?v=n6biclFh0i0&feature=youtu.be

I put together an example of how to run chromium with --headless driven by cucumber/capybara: https://dkastner.github.io/2017/04/21/headless-chrome.html

For those interested, Firefox is also going to support a headless mode. The current nightly supports headless SlimerJS on Linux and more platforms will come soon.

https://bugzilla.mozilla.org/show_bug.cgi?id=1338004

Looks like the launch bug is private? https://bugs.chromium.org/p/chromium/issues/detail?id=705916

Have people found many issues that come up in Chrome but aren't found in PhantomJS? We used to use a headless browser but switched to PhantomJS and haven't had any real issues.

(We should probably run under the real IE but jut haven't been bothered.)

I've run into some problems with some of the more ... creative jasmine tests I've done. Mostly, it's been around object mocks. I've found places where I can do things with Object.defineProperty() in Chrome that throw exceptions in PhantomJS :(

Phantom does not support more recent JS syntax/tweaks. We have an app that is aimed at more recent browsers only so can use latest ES6 features, and had to move from Phantom to in-browser tests (the alternative would be to use babel for transpilation, but then we wouldn't be testing the code that is actually released to users)

Can anyone confirm - would this work with a Flash/SWF application? i.e. could I use the headless mode to interact with the Flash Application to run some commands and retrieve the output?

I tried googling around but didnt find much to say either way...

It seems that Chromium 59 still can not be installed on Raspberry Pi, or anyone has done it?

It will be great to use this headless Chromium on Raspberry Pi to execute some routine web browser jobs.

Does it support the extensions installed on Chromium? Curious.

Does this mean I no longer need to use phantomjs for my tests?

How fast is the debugging mode? I tried the first debugging protocol when Chrome added it and it was very difficult to use. I assume this time is different?

Doesn't seem to work on OSX. Connecting to debug port from a different chrome brings up an empty page.

Running Version 59.0.3069.0 (Official Build) canary (64-bit)

same for me. Wonder if we're doing something wrong?

Has it actually landed in the builds yet, or just been marked as targeting 59 in the platform docs?

I tried to figure that out but was unable to

We are targeting 59. A fix for Mac OS X is on its way: https://codereview.chromium.org/2811633002

Thanks! It looks like that has been closed... is there any way to know what release # the fix is in, or when it gets released?

It'll be released in 59.0.3071.3

Works for me when using https://github.com/cyrus-and/chrome-remote-interface

Looking at https://developer.chrome.com/devtools/docs/debugging-clients, it seems we are missing a Ruby client. Last time I tried it gave me some headaches trying to talk to the websockets, but hopefully someone smarter than me can pick it up.

This is great if the headless mode supports the web extension API because it means that we can run our security tools almost as command line tools.

i use when they announced headless mode on linux, and built generating thumbnails from captured screenshots of websites and uncovering the technologies used on websites

and the api is available for free, https://github.com/letsvalidate/api

Can somebody explain to me what is this good for?

Thank you.

Headless browsers are frequently used in buildserver testing of web apps.

I'll be using it to generate printable PDF reports. It's a huge pain getting highcharts graphs to look identical in every browser's print view.

Is this confirmed to work with Flash? I know selenium did not support Flash so it caused some dev issues.

Given the rough comments on the Electron story earlier, this should be welcome by all.

Not sure if this can enable SSR (server side rendering) for any client side lib?

What is the use case for headless Chrome?

> Headless mode allows running Chromium in a headless/server environment. Expected use cases include loading web pages, extracting metadata (e.g., the DOM) and generating bitmaps from page contents -- using all the modern web platform features provided by Chromium and Blink.

Practically speaking, software developers will use headless Chrome to automate testing of product functionality. Today, developers use systems like Selenium or PhantomJS to accomplish this feat, but it's a painful process to maintain these headless browser execution engines. Adding headless support into Chrome means that developers can count on the presentation of their application on a given version of the Blink engine run within Chrome.

is this is chromium, too?

Please let this be capable of generating PDFs from HTML from the command line.

FWIW, you can do that now, using phantomjs (which is chrome) http://phantomjs.org/screen-capture.html

> FWIW, you can do that now, using phantomjs (which is chrome) http://phantomjs.org/screen-capture.html

It's based on webkit.

Once upon a time, but not anymore.

https://www.chromium.org/blink

PhantomJS is based on webkit (and an outdated version IIRC + a shitty JS interpreter)

specifically it's qtwebkit and an old version of it.

They recently released a beta version based on a newer version of QtWebKit:

https://bitbucket.org/ariya/phantomjs/downloads/

It's a useful tool (and huge thanks to those who built it -- and SlimerJS for that matter) but whenever I've reached for there's always been some issue to resolve, generally relating to the exact version and/or set of APIs supported. Headless mode (with PDF support -- which it looks like the latest version of the Chromium remote protocol does indeed have) built into a mainstream browsers is nearly guaranteed to be a smoother experience.

Also with https://wkhtmltopdf.org/ and http://pandoc.org/

Tried all of these and haven't found any as good as http://www.nightmarejs.org/

Thats just phantomJS though..

It switched to Electron under the hood a while back.

And, if you need much higher fidelity and control of HTML/CSS -> PDF, there's the fantastic Prince library, http://princexml.com/ (nonfree)

(I've been using Prince for over a decade, rendering everything from prescription labels, packing slips, receipts, resumes, books, and more. It's great.)

That's also an API service, http://docraptor.com, with a different pricing model that uses the Prince library for PDF rendering.

Tough crowd.

Yes: https://chromedevtools.github.io/debugger-protocol-viewer/to...

Any idea if you can configure the page DPI with this API?

Yes, the emulation domain has APIs for changing that.

https://wkhtmltopdf.org/

You can download the two tools wkhtmltopdf and wkhtmltoimage which use WebKit to generate pdfs/images.

Sadly wkhtmltopdf's version of Webkit is ancient :( No flexbox, no es6, etc...

I think it just means that the site is poorly designed. I think websites should work correctly everywhere where CSS2.1 and ES3 is supported. Otherwise some users won't be able to view those sites.

No paged media makes page formatting much too painful.

I'm the maintainer of wkhtmltopdf, and it's hopelessly out of date. There's still some bugs in the Chrome print-to-PDF support as support was added just a few days ago:

https://bugs.chromium.org/p/chromium/issues/detail?id=603559

Not sure if all the full functionality that wkhtmltopdf can be ported, it had patches to Qt/WebKit to enable that ... probably will need API enhancements in Chrome. Don't have the time right now, but I registered http://crhtmltopdf.org a while ago hoping that I'd get around to it.

The print to pdf command currently only support default print settings. Adding support for customized page size, header and footer, dpi, etc. is in progress.

Please see bug: https://bugs.chromium.org/p/chromium/issues/detail?id=603559 for updates.

Excellent news, thanks!

Not sure of your exact use-case, but mPDF does this well.

https://github.com/mpdf/mpdf

Is there an easy way to do the opposite? e.g. read PDFs as HTML (intended to be read through a remote shell), or text?

You might be able to tweak this pdf.js example to dump out its canvas element after it renders.

https://github.com/mozilla/pdf.js/blob/master/examples/learn...

Extracting text from PDF is not hard, though PDF only contains low-level formatting instructions so the result might not be nice, especially if the original PDF has any non-trivial formatting, like pull quotes, multi-column text, etc. If you don't care about that or the correct "flow" of text, it should be easy enough to just find all the Tj and TJ operators and extract their operands. You might also need to reverse some ligatures though.

Producing nice semantic HTML is much harder, though also easy if you don't mind every word in a separate absolutely positioned div.

Many PDF reader software already contains empirically tuned routines to infer the text flow and generate text files (because the software needs to handle Select All and Copy), but they often produce bad results.

But if you just want to read a PDF on a remote machine over ssh, the easiest solution might be just transferring the file and then opening it locally, or use X forwarding and open the PDF with a graphic reader.

I'm in the same boat. You might want to check these out:

http://coolwanglu.github.io/pdf2htmlEX/

https://github.com/JonathanLink/PDFLayoutTextStripper

http://tabula.technology/

https://docparser.com

Easy, not so much, depending on exactly what you want to get out of it. I did a project with this once https://www.idrsolutions.com/jpdf2html5/. Last I checked, they only supported it as a Java library that could generate rather nice looking and complete HTML pages from PDF documents. The output was great, but it was kinda pricey and difficult to work with.

On the opposite side of the complexity level, I have also used this http://www.pdfsharp.com/PDFsharp/ to extract bits of text from PDFs. It's free, but you only get access to the raw PDF text with formatting codes. It works fine if you just want to grab a short string, but you got your work cut out for you if you want to do anything more sophisticated.

If you're willing to wait until Electron releases a Chrome 59 -based build, I'll be updating https://github.com/mixu/electroshot which handles screenshots and print-to-PDF along with a bunch of other niceties.

Does it support paged media?

Especially it should adhere to CSS3 page break properties.

Reply on news.ycombinator.com