Show HN: Paper to HTML Converter
papertohtml.org
papertohtml.org
I’m one of the engineers at AI2 that helped make this happen. We’re excited about this for several reasons, which I’ll explain below.
Most academic papers are currently inaccessible. This means, for instance, that researchers who are vision impaired can’t access that research. Not only is this unfair, but it probably prevents breakthroughs from happening by limiting opportunities for collaboration.
We think this is partly due to the fact that the PDF format isn’t easy to work with, and thereby make accessible. HTML, on the other hand, has benefited from years of open contributions. There’s a lot of accessibility affordances, and they’re well documented and easy to add. In fact, our hope long-term is to use ML to make papers more accessible without (much) effort on the author’s part.
We’re also excited about distributing papers in their HTML form as we think it’ll allow us to greatly improve the UX of reading papers. We think papers should be easy to read regardless of the device you’re on, and want to provide interactive, ML provided enhancements to the reading experience like those provided via the Semantic Reader.
We’re eager to hear what you think, and happy to answer questions.
Our focus right now is on providing a tool folks can run it on whatever papers they have access to. For instance, some researchers might have access to documents that aren't available to the public. We want them to be able to run this against those.
That said as we expand the effort I imagine we'll eventually pre-convert things that are publicly available, like those on ArXiv, etc.
Their job is a little bit easier because arXiv papers have the .tex source available, so you can use one of the various tex2html variants, instead of having to extract the paper's contents from a rendered PDF.
My guess is it's largely for historical reasons. At the time most venues were organized PDF was probably the best (or only) mechanism for sharing documents for print distribution.
But we think it's time to change that :).
Regarding your question, I'd say that it is a natural continuation of centuries long tradition of writing on the actual paper. The invention of TeX actually made it easier to produce more papers, then came PDF, and you could produce virtual papers. Also science journals pretty much have monopoly on scientific knowledge distribution, and they are mostly paper too
PDF is the only widely supported format can guarantee accurate reprint.
HTML for text.
SVGs for diagrams.
Equations can be exported as images if needed.
If all goes well we won't need this software anymore. In a best case scenario the publishers start accepting HTML, and gone are the days of having to convert PDFs to something better...!
Page anchors can be used for navigating between sections. We present a table of contents that makes this easy. For instance:
https://papertohtml.org/paper?id=6f9fc51102cf49bff4f4e2b3367...
It would be great if you could add some basic CSS rules for print? Right now navigation elements are needlessly repeated on each page, obscuring the content.
Also, you forgot to include bold and italics webfonts, so you have faux-styles for all headings and emphasis.
You know what, they are.
I like print format for reading purposes, even if it's on my epaper tablet. The other day when I took a train for 8 hours, I printed out several papers to read on my b&w laserjet. And it's more difficult to read diagrams these days because people make them all in colour, sometimes in ways that are very difficult to read when it's converted to b&w.
I find it a real tragedy that all these efforts to turn papers into dynamic content, which I wholeheartedly applaud, ignore the still very relevant use case of printing. Every preview mechanism for camera-ready papers should include a b&w print-preview mode.
The other advantage of PDF is that "page count" still means something. There's a reason journals limit page count, and it's not because it adds a few kbs to the download. It's because long-winded papers that don't get to the point need editing.
Here's the output of this tool on this PDF - https://arxiv.org/pdf/1909.00031.pdf - content is preserved, but text is readable: https://imgur.com/a/EPCaWxP
This probably evokes some kind of sense of authenticity. Like some physical paper document it has exactly one appearance.
I don't know of any more convenient way to directly attach my thoughts to a specific portion of text. (If there is, I'd genuinely like to know).
And it even works well across multiple devices: I have a folder on my PC that's synced with my phone via syncthing containing mostly PDFs (savend web pages, papers, books, ..) and the annotations I make in those PDFs on my phone are directly available on my PC ... all without using some cloud bullsh*.
Edit https://allenai.org/terms point 5, you own all the uploads! So if by mistake we send a medical PDF for example or something else that is under gdpr, we can't ask you to delete it???? ? Wtfffff
See https://papertohtml.org/about:
> What data do we keep? We cache a copy of the extracted content as well as the extracted images. This allows us to serve the results more quickly when a user uploads the same file again. We do not retain the uploaded files themselves. Cached content is never served to a user who has not provided the exact same document.
Also, we can delete the extracted data on request. Just send a note to accessibility@semanticscholar.org.
Sorry for the confusion!
>Also, we can delete the extracted data on request.
Just to be 100% clear, you are referring to the cached extracted data, right?
Any plans on that?
I think I will stick with pdftohtml, pdftotext, and pdfimages https://en.wikipedia.org/wiki/Poppler_(software). These take seconds not minutes.
From user perspective I dont understand why not release the source code and let people compile a native application. (Did I miss the link to the source code.) Instead it looks like this is just a means of collecting free data (metadata, more training data, data from submitted papers by default) everytime someone submits a paper.
It seems to me the masheen learningz technikz boil down to a generalization of my lightbulb moment here[2].
[1]: https://grobid.readthedocs.io/en/latest/
[2]: https://www.nu42.com/2014/09/scraping-pdf-documents-without-...
Also, we're working hard on a new extraction mechanism that should allow us to replace GROBID [2].
There's a lot of really smart people at AI2 working on this, I'm excited to see the resulting improvements and the cool things (like this) that we build with the results!
Off-topic low-value comment, but I'm now going to be getting a T-shirt made with the caption "i can haz masheen learningz?"
One comment is that the slowest page to load was the Gallery [0] as it loads an ungodly amount of PNG files from what appears to be a single IP (a GCP Compute instance?)
I see 421 requests and 150 Mb loaded. As it seems to be mostly thumbnails, have you considered using jpegs instead of pngs, potentially use lazy loading (i.e. not load images outside of the viewport) and potentially use GCP's (or another provider) CDN offering?
Once I clicked a thumbnail, loading the article itself (for example [1]) was quite breezy.
The gallery is a great showcase of what your site does -- I think that it'd be worth making it snappier :-)
Cheers and congrats again
P.S. Also, the paper linked below [1] seems to have a few conversion problems -- I see "EQUATION (1): Not extracted; please refer to original document", and also some (formula? Greek?) characters that seem out of place after the words "and the next token is generated by sampling"
[0] https://papertohtml.org/gallery
[1] https://papertohtml.org/paper?id=02f033482b8045c687316ef81ba...
Yup. There's no CDN or anything like that right now. We kept things simple to get this out the door. But we definitely intend to make improvements like this as we improve the tool.
The more adoption we see, the more it motivates these types of fixes!
> P.S. Also, the paper linked below [1] seems to have a few conversion problems -- I see "EQUATION (1): Not extracted; please refer to original document", and also some (formula? Greek?) characters that seem out of place after the words "and the next token is generated by sampling"
Thanks for the catch. As you noted there's still a fair number of extraction errors for us to correct!
Very cool project, looking forward to seeing how it develops!
For thumbs of text papers, perhaps a GIF or PNG would be smaller than a JPEG while retaining pixel accurate crispness?
We're not sure if it's something that we can distribute as OSS just yet. It relies on a few internal libraries that would also need be publicly released, so it's not as simple as adjusting a single repository's visibility.
Though OP has his kindle offline all the time, so not a issue for them.
[0] https://www.mobileread.com/forums/showthread.php?t=320564
But clearly it is a nice idea and I can't wait that such tools work better!
Yup, this is a known issue that we're working towards fixing.
> But clearly it is a nice idea and I can't wait that such tools work better!
Glad to hear it!
Demo: https://pdf2htmlex.github.io/pdf2htmlEX/doc/tb108wang.html
> What are the limitations? There are several known limitations. Tables are currently extracted from PDFs as images, which are not accessible. Mathematical content is either extracted with low fidelity or not being extracted at all from PDFs. Processing of LaTeX source and PubMed Central XML may lack some of the features implemented for PDF processing. We are working to improve these components, but please let us know if you would like some of these features prioritized over others.
But we intend to fix this!
Cant wait to see what people do with this . . . .
There's a lot of amazing people here, doing really great work. It's a really inspiring place to be. I feel really lucky to work with such great people on interesting, important problems.
Also, I should mention...we're hiring!
My friend wrote his PHD in Latex, but it all ends up being PDFed anyway for what, eye candy?
It's time to move on. #ditchpdf
As per other recent discussions on HN I think the general accessibility of academic papers is ripe for improvement.
Maybe we'll work on vi bindings next...