Converting my PhD thesis into HTML (2021)
desfontain.es
desfontain.es
It still beats the competition because of rock-solid referencing, both to in-text elements like equations, chapters, etc as well as citing literature with bibtex.
Plus, it’s extremely stable, so someone who learnt LaTeX 20 years ago, like yours truly, can download the newest TeX distribution and feel at home immediately.
Nevertheless, I would prefer a Markdown-based system that can use CSS and MathML, and has a 100% bibtex clone for references.
Yes, pandoc goes quite a long way along this route, but setting up such a pipeline is still too complicated for many.
I see it supports Jupyter notebook. Math support in those isn’t too bad at all, so it might just work for many cases.
(edit: generating PDFs that is)
What people don't acknowledge is that there is a base level of syntax complexity needed to produce fully general documents. If you do, the natural conclusion is that to fix latex, you need a full rewrite of latex with minor changes to fix all the inconsistencies that have crept into it.
https://phd.row1.ca/phd/introduction
Allows web-first, as well as a PDF output (via LaTeX).
While it does not directly address the issues you point at, it does alleviate some issues.
* The syntax is somewhat easier to parse.
* It is a lot easier to write functions to redraw the same components over and over again.
* Doing math calculations to systemically place objects in relation to each other is a lot easier because python's arithmetic syntax is a lot more intuitive than TeX's.
Of course, this does mean that you have to fire up python to draw figures.
Examples code is a must for any software (test cases work as examples btw) and example graphics are a must for any graphics software. I know this isn't your fault, but I'm just venting a bit here. I'll check this out the next time I'm writing a paper but I just don't get how people can put software out there without any examples (even toy examples). If you have any examples you're willing to share I'd find that extremely helpful.
There are some examples in this notebook https://nbviewer.org/github/allefeld/pytikz/blob/master/pyti...
Otherwise, this is just a python interface for tikz. Whatever is possible in tikz is possible in this. For every command in tikz, the corresponding python command is here https://allefeld.github.io/pytikz/tikz/
However, I do agree that more examples would help. In its current form, its mostly useful to people who are already familiar with tikz and need simpler syntax, and can quickly get up to speed with this.
Not yet. But you can easily learn to do them:
\usepackage{animate}
...
\animategraphics[width=10em,loop,autoplay]{4}{a_}{0}{10}
My free, cross-platform desktop Markdown editor, KeenWrite[1], integrates with the ConTeXt typesetting software[2]. I'm working on a branch to make integration containerized[3] because its installation is painful. KeenWrite limits math to plain TeX[4] so that the output can be rendered using any TeX-based typesetter (ConTeXt, LaTeX, MathJax, εχTEX, etc.).
Here's a sample document typeset using ConTeXt (skip to page 40 for the math):
https://pdfhost.io/v/4FeAGGasj_SepiSolar_Highlevel_Software_...
That document theme is called Solare[8].
> that can use CSS and MathML
Adding CSS mixes presentation logic with content, which is something KeenWrite strives to avoid. Instead, KeenWrite implements Pandoc's annotation syntax to keep presentation logic out of the content. I've written about this extensively in my Typesetting Markdown series[5].
You can produce some pretty amazing documents just with annotations, such as the following that I wrote in Markdown and typeset using ConTeXt:
https://impacts.to/downloads/lowres/impacts.pdf
> has a 100% bibtex clone for references.
Markdown fails at references. At some point, I'd like to implement cross-references in KeenWrite. Except there's at least six competing standards for the syntax, which I've also remarked upon[6], making the choice of syntax difficult[7].
> setting up such a pipeline is still too complicated for many
FWIW, my Typesetting Markdown series, which explains how to set up a typesetting pipeline using Pandoc, is one of the reasons I developed KeenWrite: to replace that entire pipeline (R, Markdown, externalized variable interpolation, math, and typesetting) with a single tool.
[1]: https://github.com/DaveJarvis/keenwrite
[2]: https://wiki.contextgarden.net/Installation
[3]: https://github.com/DaveJarvis/keenwrite/blob/1_typeset_using...
[4]: https://github.com/DaveJarvis/keenwrite/blob/main/docs/scree...
[5]: https://dave.autonoma.ca/blog/2020/04/28/typesetting-markdow...
[6]: https://talk.commonmark.org/t/cross-references-and-citations...
[8]: https://github.com/DaveJarvis/keenwrite-themes/tree/main/sol...
https://github.com/Mathpix/mathpix-markdown-it
Disclaimer: I'm the founder of Mathpix.
My own experience was as a physics student, 30 years ago. Students paid a heavy price for being able to print and submit the entire thesis with no manual intervention. The students who chose LaTeX took the longest at it. I didn't have access to a Unix terminal anyway, and banged out my thesis on an MS-DOS machine. Whatever my word processor couldn't support, I added by hand. The readers were OK with this.
My solution to all typographic problems was "take care of it after defense." I spent a few days after my defense getting my copy to be ready for duplication, including sticking all of the page numbers on with glue because I couldn't make inline figures work.
For example, automatic updates of figure numbers in captions and references: Countless times it failed on me and I had to manually recreate the fields, bookmarks, cross-references, and whatnot is needed.
Bibliographies are hardly doable without an external tool that comes with its own headaches.
Typography in MS word is quite decent these days, though. Anyway, the content of a PhD thesis shouldn't be judged by its typography (as long it maintains a readable standard).
There's always Overleaf, Pandoc or LyX to make things even simpler. LyX in particular deserves to be better known.
Complex things, like TikZ, are of course difficult and time consuming. But those are impossible using Word.
IMHO, the biggest advantages of LaTeX are reproducibility and reference management. Big Word documents are quite fragile. And reference management is a mess.
My university actually required that people use MS Word for their thesis, which seemed to work out okay for many, despite such a top down approach not seeming like the best option.
Personally, I used LibreOffice anyways and while it was certainly as clunky as Word (especially once images, diagrams and formulas got involved), it was also passable.
Except that things like bibliography refused to work correctly and completely broke, about which I wrote a bit of a rant: https://blog.kronis.dev/everything%20is%20broken/libreoffice...
Pandoc I think really e a different niche, hard to imagine really a complicated document benefiting from even more pipelining. You're editing a lot...
I find PDF math papers easy to browse, unlike the author. They're much easier and more organized than a website, can be easily searched and have a *proper table of contents* compared to websites. As for poorly browsable on a phone -- well I think that is irrelevant because nobody is going to read a complex technical paper in practise on a phone. They do look decent in tablets, and as for screen readers...well that's a valid point but screen readers don't work well for material with lots of equations anyway.
I applaud the author for the effort but looking at the result, I would not want to read math that way.
So do I. Still, I wish LaTeX produced easily reflowable PDFs, especially when a document is formatted in two columns.
Science needs a culture of open sharing, the same way physics and math has it.
what's needed are narrow margins.
What guarantees of layout do you require?
In related news, MathML is back in Chrome v109
Some people write documents that can only be clearly presented on a 15" or larger display. Maybe a comparison table with a bunch of columns, maybe a detailed chart, maybe a PCB schematic, whatever.
These people, being considerate of their readers, want to ensure if someone with a 13" screen comes along, they'll get scrollbars or small text, rather than a badly reflowed table where the word 'Yes' gets split over 3 lines.
Other people want to read those documents on 5" phone displays.
This is something that we’d like to change. There are many visually impaired students who need to learn mathematics the same as you and I.
My “eyes were opened” when I was working with a blind student in my class. The textbook I’d written in pretext (transpiled to pdf and HTML) could be read on his BrailleNote but some of the equations were wonky, so I rewrote them to work for everyone.
It would be better if we developed tools to make them work for everyone straight away, instead of relying on authors. That’s one of my career goals.
I think MathML (which has gotten much better in browsers, thanks to Igalia[1]) is a much better bet we have to make this possible than LaTex compiling to PDF.
or authors should distribute their PDFs as bundles that include formatted versions for all of paper, large screen, and small screen.
I do, in fact. Or rather, I often would like to but with PDF? No chance. IEEE explore online reading sometimes works, but it would work better if they cleaned up their UI to be compatible with phones.
I have read thousands of pages of fiction on a phone and quite enjoyed it. Phones are great for reading if the content reflows properly.
Now publishers and content creators would need to embrace non-paginated, reflowing output. This would not only facilitate reading on phones, but also on tablets and laptop screens.
O‘Reilly‘s online platform does a good job with their app.
There is zero reason why paginated output should be the default in 2022.
IEEE explore does a good job rendering equations on phone screens. Therefore, it is possible.
There is no technical reason why equations couldn’t be rendered on a screen just as well as on a PDF. Sure, canvas size constraints might interfere, but this problem exists in principle also on paginated output. Plus, horizontal scrolling is a thing.
I‘m not saying a phone is the ideal platform to read a paper containing free energy-like math, but it can go a long way. Much longer than with the artificial restriction to paginated output like PDF.
I would like to be able to read a book also on a phone, but I am not going to compromise on quality for that, given that I can just read it on a large tablet in PDF format.
Does this live up to your maths standards?
To compare, open the accompanying PDF, which is also provided along with the webpage. It is of MUCH higher quality, which is partially due to the fact that layout is static.
Furthermore, the webpage doesn't support pagination. The problem is turning it into a book, and there doesn't seem to be a good standard that supports HTML+KATEX/MATHJAX properly. In theory there is no reason they shouldn't, as epub support javascript, but in practice it just doesn't work properly.
> doesn’t support pagination
Isn’t that exactly the point? Please, no pagination on a screen. A web page will do just fine. In fact, that was what the www was conceived for: publishing science.
> open the accompanying PDF, […] of much higher quality.
The only thing that is higher quality in the PDF is the justified alignment and pagination. The figures have the same (poor) resolution.
The bottom line is: * it is perfectly possible to publish technical papers in a format that is accessible on a phone. * non-paginated, free flowing output also works better on larger screens (e.g) I can resize the window and have my note-taking app open next to the paper. * PDFs are still great for annotating and printing, if required.
As for your bottom line:
* Of course you can make HTML pages with KaTeX/MathJax that look great. I have done it myself, and this very HN post also points to such a web page.
* While a webpage is often fine, for anything I will spend a significant amount of time reading, I definitely want a separate entity, a book or a paper, that I can have in my library. There is no such format currently that has satisfactory quality, except for PDF.
* Non-paginated, free flowing output is the only real option for a phone, but does not work well on a large screen. Your IEEE example looks bad on a large screen, compared to a PDF. I am looking at it on a 27 inch screen, and on an iPad 12.9 inch, and it shows me one column, which separates the images from each other and from their context, and makes it harder for me to scan back and forth between related content. Pages provide a context, even though an admittedly rather arbitrary one, and non-paginated text is missing that context. You could do two columns, but how would that work for non-paginated text that knows no boundaries?
* For me resizing is not an issue, I take notes on paper or on my computer, but I can see how that might be important for somebody who is willing to trade reading quality. For me, PDFs are clearly superior for digital books and papers to anything else currently out there, at least if they have been created with care. PDFs that have been created by printing an epub or webpage, on the other hand, are useless for me and land immediately in the garbage bin whenever I encounter them.
[0]: https://github.com/IDPF/epub3-samples/releases/download/2017...
Research papers are far more complex, and have established standards that aid quick reading and parsing. I absolutely don't want to deal with reflowing equations, reflowing figures, or whatever when publishing papers. Precise margins and column widths.
Responsive webdesign has been around for quite a while. I don’t see a reason, other than lack of effort/investment, why we shouldn’t be able to read technical papers on variable-width screens, in a non-paginated form.
Dealing with the technical challenges should not be the task of the author, but the publisher. And indeed, most publishers are on it.
What‘s missing is a standardised format that can be downloaded, annotated, re-shared like a PDF.
And because the PDF has a fixed layout it's also much easier to prevent "rivers" in paragraphs. Which hence makes it a no-brainer to use justification. To me many print publication using justified text (including LaTeX documents) are a thing of beauty and I do hate how "left align" breaks the flow of reading. I'm taking slightly different spacing between words due to justification every day over horizontal lines of different length, which I find fugly and confusing beyond repair.
More hyphenation controls are coming to CSS and, one can dream, it may be possible one day to programatically detect rivers?
Meanwhile rivers be damned, I override anyway many sites and add "text-align: justify". The nice thing is: because "text-align: left" is the default many sites and minifiers do not bother with text-align at all, so adding "text-align: justify" works for many, many, many sites.
And I only half-buy anyway the justifications (ah!) for left alignment on the Web.
It's basically saying: "We know better than people who've been working in print since decades (or more), left align is easier to read". I don't buy it. Left align breaks my reading flow. And I cannot be the only one.
To me left align is trading potentially ugly looking paragraphs (due to rivers) for certainly ugly looking paragraphs (due to left justification: just look at the right of each paragraph... Such lack of clarity, such chaos cannot be unseen. It's pure fail).
P.S: I've actually typeset books both in LaTeX and QuarkXPress and their were justified, not left-aligned.
I think you're an outlier in your strong preference for justified text but this serves as an example in favor of using HTML to present content. Well made web content is much more malleable by users to make it meet their needs and preferences.
On the other hand, PDFs generated from Latex are completely useless for screen readers.
> Even slight differences in layout can make a complex set of equations difficult to parse.
Such set of equations should normally be represented by a single block, I can't imagine a reason why layout should change inside that block.
The layout of pdf is unnecessarily rigid. When I'm reading it on my screen, there's no reason the text should be split into A4 pages with very specific margin values. Latex also often moves your figures a few pages ahead because they didn't fit on the specific page. There's absolutely no reason for that when you have access to the big continuous canvas of an html page. This works for equations too; if you have a long equation block that happens to be right between two pages, you either have to let one page have a gap, or reorder/rewrite your paragraphs to make the equations fit. None of this has a good excuse when it's read on a screen.
I don't think we need a website, but a js-free webpage with hyperlinks would be a lot better than pdf. Pdfs I find imperfect but ok.
Wasn't this precisely the use case for HTML and the WWW as originally conceived by Berners-Lee and his fellow internet pioneers?
It’s paragraph breaking was state of the art when it was new but other systems break paragraphs now and potentially better. I also think ragged margins aren’t really a problem.
I think if layout mattered as much as you imply, scientists would have to use a tool that offers more control like indesign.
None of this is to say that getting good layout in HTML is easy, of course.
Yes, precisely that. As a scientist I don't even want to have to deal with layout. That's what publishers are paid extremely well for. When I self-publish content I want the process to be as simple as possible. If this means ragged margins, browser-default styles for headings etc., default colors and fonts — so be it.
(but to be fair, optimising the layout is an excellent way to procrastinate on doing hard research)
A pdf is also much easier to archive. The job of sci-hub would but a lot more difficult if every paper came with separate html, css, javascript, and images.
But the author is right, LaTeX is widely used, translating it to HTML is hard, and there are no incentives to make or improve tools. Even if you don't want HTML, it'd be good for the LaTeX tools to automaticalky generate reflowable PDF for accessibility. There should be a process for funding infrastructure to accelerate science, and this would be a good example.
There's an interesting trick you could try. PDF supports embedding other contents. LibreOffice, for example, can slip its original edited file into a generated PDF, producing perfectly editable PDF. Maybe a variant of this idea could be used, e.g., store the LaTeX source of HTML in the PDF, so people can "get the PDF" yet still have options. But that's just a side idea, the real issue is funding infrastructure for science.
Lwarp worked for me to produce an HTML version of the TikZ documentation (https://tikz.dev), and that's probably one of the more complicated tex documents that exists. (Though granted, this was still a major effort.)
The final thesis that I will hand in is of course a regular PDF (well, a print based on that). But the HTML version can contain lots more stuff that doesn't fit (and belong) into the actual paper thesis, e.g. code snippets to generate plots etc. (optional export of Org subsections). By publishing the git repository of the thesis, linking all code and data + a bit of work -> full reproducible thesis.
Source code is always a pain to export for PDF, especially when switching from 1 to 2 column layout depending on the publication.
My blog is written in org too, but I post-process to make it fit in with the rest of my static site. At some point maybe I'll get enough free time to swap out my makefile setup for org-publish, but if it ain't broke...
To anyone who'll listen I advocate for org-mode as a better alternative to Jupyter notebooks, Markdown, and LaTeX. It's in some ways the antithesis to "do one thing well". If you try to do N things well while adhering to the unix philosophy you end up learning N different tools. But org-mode is one tool that does N things well, and some of the things you learn doing thing N transfer to thing N+1, so you get economies of scale.
I'm looking for a way to plot simple numeric data signals in time series, which are pretty trivial in jupyter notebooks.
If I had more time and wanted something more convenient and magical, I would probably write a elisp function that takes X Y (Z) columns and generates a plot from those using a simple Nim program in the back that receives the data, generates the plot and returns it somehow. Haven't given this much thought though.
# first run this code block:
#+begin_src python
# some code to generate a plot at images/plot.pdf
#+end_src
# then put the image where you want it:
#+CAPTION: Some plot or other
#+NAME: fig:asdfhjkl
#+ATTR_LATEX: :width 0.7\textwidth
[[file:./images/plot.pdf]]
Here's a full example from one of my papers. You can see I made quite a few revisions with all the commented-out lines. Note the :results none :exports none arguments to the org-babel code block, which makes the code itself invisible in the resulting paper.
#+begin_src python :results none :exports none
import matplotlib.pyplot as plt
import seaborn as sns
def get_loss(filename):
loss = []
with open(filename) as f:
for line in f:
loss.append(float(line.split(' = ')[-1]))
return loss
data = {}
# data['NLP (P=0.1)'] = get_loss( '../sample-programs/loss-epochs-nlp-0.1-30.txt')
# data['NLP (P=0.2)'] = get_loss( '../sample-programs/loss-epochs-nlp-0.2-30.txt')
# data['NLP (P=0.4)'] = get_loss( '../sample-programs/loss-epochs-nlp-0.4-30.txt')
# data['NLP (P=0.8)'] = get_loss( '../sample-programs/loss-epochs-nlp-0.8-30.txt')
data['NLP (P=0.1)'] = get_loss( '../sample-programs/loss-epochs-both-0.1-30.txt')
data['NLP (P=0.2)'] = get_loss( '../sample-programs/loss-epochs-both-0.2-30.txt')
data['NLP (P=0.4)'] = get_loss( '../sample-programs/loss-epochs-both-0.4-30.txt')
# data['NLP (P=0.8)'] = get_loss( '../sample-programs/loss-epochs-both-0.8-30.txt')
data['Plain'] = get_loss( '../sample-programs/loss-epochs-30.txt')
sns.lineplot(data=data, palette='deep')
# TODO add plots for combined Plain+NLP at other probabilities
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
filename = 'images/loss-plot.pdf'
plt.savefig(filename)
return filename
#+end_src
#+CAPTION: Training loss for data augmentation
#+LABEL: fig:loss
#+ATTR_LATEX: :width 0.5\textwidth
#+RESULTS: fig:loss
[[file:./images/loss-plot.pdf]]But your approach sounds great. Good luck!
Isnt html or a high level abtraction (like Spectacle) the best tool for the job today?
https://formidable.com/open-source/spectacle/docs/#one-html-...
Similar products have been in business since the early 2000's.
I wanted to do the same with mine but I lost the sources (it was 20+ years ago and did not survive some upgrade/technology change). I was mostly concerned with the .eps files that are hardly portable to .png or similar.
This made me think a bit about preservation of ~recent data (1990-2010). a lot falls in the category of "not natively on the web yet" and "stored on stuff that does not work anymore".
I regularly read papers from 10 or 20 years ago and sometimes even ones from 30 years ago. The old PDF documents work without any major issues. I have much less confidence that future browsers will continue displaying old HTML documents laid out using then-obsolete techniques in a sane way on future hardware.
In particular, the points they specifically call out about the difficulties with PDFs are either contrary to my experience or irrelevant to me:
They cite: "... difficult to browse, impossible to read on a phone, uncomfortable to read on a tablet, hostile to screen readers, impractical to search engines ..."
Browsability: I use Calibre [1] as an e-document library and PDFs are a first class citizen. It is a lovely piece of software for which I have few complaints and much praise.
Phone and tablet readability: I've used a mix of GoodReader and more recently Documents on my iPhones and iPads and have never had troubles with reading PDFs on them.
Screen reader access: I'm not much of a screen reader user, so no comment.
Impractical to search: While you may occasionally need to go to an extra step to search PDF collections, it's hardly impractical and there are many performant options. Calibre includes a full-text indexing feature for your whole document library, to name one.
1: https://calibre-ebook.com/The printed page is the optimal format for consuming a thesis or paper, so it absolutely makes sense to optimize for that, rather than dilettante browsing on a phone.
Disclaimer: I'm the founder of Mathpix.
I settled for markdown with KaTeX for math, although I would like to return to the LaTeX->HTML project at some point soon.
Because, why not
Sorry! I should have said our. You are right to call me out on that. Sometimes I don't reread my comments before posting.
> I would currently recommend against any PhD student using it for their PhD -- it can't even generate a PDF with required formatting, which is required by most Universities.
If it doesn't do something someone needs, file an issue or email me and we can probably add what is needed.