Pandoc
pandoc.org
pandoc.org
I use it for turning .md files into .html or .pdf.
I use it for creating slides with it.
I even use it for fixing the hard-wrapped text I write in vim before sending emails. When I write in vim, I prefer the text to be hard-wrapped, but for emails, I like it better when the text is not wrapped. I recommend arp242's essay explaining the problem with hard-wrapping [1], but basically the way I workaround this problem is using a local script which uses pandoc at some point [2].
Overall, pandoc is really good.
[1]: https://www.arp242.net/email-wrapping.html
[2]: https://github.com/kugurerdem/dotfiles/blob/2d68357273e1bc30...
* I use this css file when converting: https://gist.github.com/killercup/5917178
As for pandoc, yes it's amazing, and I have been using it to convert my word documents to markdown so I can publish a technical textbook I'm working on using Quarto. I tried writing directly in Quarto for a while, but as per my point above, it really slowed me down and distracted me from actually writing, so I figured out the pandoc pipeline. My most favoritest feature so far is that it converts tables AND equations to markdown and latex perfectly. It's so seamless that I'd actually recommend Word->pandoc as the best way to write a complicated markdown table.
> writing should happen in "word processing software"
Writing should happen where you are comfortable editing text. I am comfortable in same editor where I write code.
> To people who say "Word sux", I say "That just means you don't know how to use Word properly". Writing any markup or markdown syntax in an IDE is a disaster for the creative process.
To all the people who say "Writing markup in an IDE sux", I say "That just means you don't know how to use it properly". I can write in a flow and apply/change formatting easily. I can jump around and rearrange documents with ease. And it is in a format that can be opened and read by native software on almost any computer.
If you think that Jack Kerouac would prefer MS Word over a much simpler plain text editor, I don't agree.
For that particular context of text.
I write my code, notes, text and emails in vim. Some in markdown. But to this day, I still miss the incredible usability of LyX 1.x when writing pure long-form text.[ß] I could force all writing to occur within central 60% of the editable screen height [doable with vim but not as cleanly]. No whitespace or formatting issues, ever - set the document defaults according to my liking and it would feel "just right".
Proper rendering and visually correct editing of math formulas as part of text. Oh my. Fond memories of being able to type '<raw latex hotkey>\frac' and continue fitting in the values...
If it had had vim's search-powered navigation, it would have been nearly perfect. LyX 2.x was a step up in visual appeal and two steps down in raw usability. I've since picked up writing raw latex where I need good formatting, just because I could not make LyX 2.x bend to my taste anymore.
ß: Back on early 2000's, I wrote a book in LyX. As well as all my university course papers, including the master's thesis.
That's not a prerequisite for the Latex folks, in my experience :D
(I'm also in the "wrote my masters thesis in word, including the fair number of equations" group)
But Word for collaborative work is a nightmare. Sometimes fastest way to work with the science text or documentation in a Word document is to hire a freelancer to retype all the text in LaTeX or something else.
It's a kind of Trolley problem: sometimes Word fanboys, unfortunately, _should_ suffer to let all the team get the job done.
Why? It has comments, tracking etc. Concurrent edition i impossible, though (even when MS says it is possible). For that Google Docs is great (or some self-hosted systems)
It doesn't scale. At all.
I used to work at a university lab group where all 30 of us would need to concurrently write, edit and review 150+-page, heavily technical reports with lots of diagrams and tables spanning pages. To be clear, most of the time all of us were working on the exact same huge document.
Word's version tracking stood no chance. Formatting was regularly off, tables were breaking apart, diagrams misplaced. Syncing was extremely bad, often with entire paragraphs in changes going missing, other times deleted portion were reappearing, all that jazz.
LaTeX on an online collaborative environment (well-known, not naming it -this post isnt an ad) on the other hand, despite its archaic way of working, never showed any of those problems. If a table was placed somewhere, we could be sure it would never get moved to random places, and changes/rewrites would be always synced correctly (as LaTeX source is plain text, merging algorithms/CRDTs have a much easier time).
As I wrote, it does not work at all for concurrent access - I mentioned Google Docs & Co for this.
> LaTeX on an online collaborative environment (well-known, not naming it -this post isnt an ad)
I wrote my MSc and PhD thesis in LaTeX (physics) so I know how fantastic it is. You write content without caring for the container - and since changing anything is black magic you give up and do not try (which is a VERY good thing - it just works).
I never used Overleaf though (I guess that this is the product you refer to). I guess that having a concurrent system (such as etertab or something - or Overleaf if it supports truly concurrent editing) is the graal.
The drawback is that you need to know the language to cooperate. In a university setting this is not complicated, in a company - not so much.
Indeed Google Docs is much better - we also used that - but it's still a WYSIWYG editor, which IMHO it translates to 'extremely hard to enforce style'.
> I never used Overleaf though (I guess that this is the product you refer to). I guess that having a concurrent system (such as etertab or something - or Overleaf if it supports truly concurrent editing) is the graal.
Yep, Overleaf was what we used. Its paid version was very much like Google Docs but on a plaintext editor wrt. to concurrent access. It could even do change tracking, comments, all the jazz, even Git synching (which we used for backups and CI)
> The drawback is that you need to know the language to cooperate. In a university setting this is not complicated, in a company - not so much.
I'm curious as to why. If the company is new and built on LaTeX from the very beginning why not? When I joined, I didn't know the language at all, but that wasn't a problem-one would learn on the job.
It really depends on the company. I worked (and work) in large high-tech companies and whenever I tried to introduce something like Markdown I quickly hit he wall of non-technical people who did not want to try a new system. They new Word, were suffering with Word but did not have the mindset to give a try to something different.
For the ones on Google Docs it was even more difficult because, arguably, Google Docs is a really neat product for collaboration.
My teams use Markdown for all text (either Obsidian or internal wikis) but this is because they are good in what they do and that they fear their management line :) :) (just kidding)
Anyway, Word «collaborative» features are so much worse than git repository and pull requests! And even you need to collaborate with some extremely non-technical folks, in the case of LaTeX they still use add comments to the PDF file — and this workflow is still way more productive than editing the same Word document.
Other reasons to not to use Word in collaborative pipelines are already mentioned in neighbor threads.
We are the ones that have to suffer because people we are forced to collaborate with do not take 5 minutes out of their day to learn the basics of a tool they use professionally.
It is the equivalent of seeing somebody use right-click to copy-paste, except it tangibly makes my day worse.
BTW, when I said tech people, I was thinking mostly about the computer savvy academics who use Latex for everything.
Personally I find LaTeX misses the mark. Too much markup is needed and it detracts. I'm a fan of asciidoc though. I just wish the templating was a little better though.
For those not in the know, Simon Peyton Jones is one of the originators of Haskell, and sort-of-but-not-quite BDFL of Haskell... and he uses Comic Sans for all his presentations because it filters out people who will complain about font choice for a presentation.
https://softwarefoundations.cis.upenn.edu/lf-current/Basics....
[1] https://rhodesmill.org/brandon/2012/one-sentence-per-line/
To be fair, for a significant number of things LaTeX is the better environment, even with it's own issues.
I come from a trades background and am now in the academic field (teaching and admin for my trade) and I would really like to do some professional development in this area. Also excel! I know I’m lacking and it would help my long suffering director haha.
[1]: https://support.microsoft.com/en-us/office/make-a-document-r...
[0]: Not a rhetoric question that implies one answer to be the only "right" one.
I'm either working on a document for myself or with collaborators.
In the first case I'll use markdown for simple things or LaTeX for bigger things since I can work in a familiar environment where I work most efficiently (VSCode).
In the second case, I'll work with collaborators so I will never be able to trust that a document I have sent off for reading is still consistently formatted when I receive it back. This means that any benefits of the collaboration tools (eg. review history or suggesting changes) are wiped off the table. I will have to integrate any suggested changes into my own authoritative version of the document by hand anyways. At that point I may as well work where I fell comfortable and use markdown / latex, and send off Pandoc converted word files for comments by others when it is relevant.
This is of course for serious pieces of writing, not throwaway stuff like eg. meeting notes, but for those Google docs is plenty.
Word (and similar text processors) is a toolset. All it gives you are tools. Word lets you define formatting rulesets, lets you apply formatting rules and create rulesets from applied rules.
Some evangelists may say that "safe" text processor would only allow application of rulesets, because direct application of rules leads to "spaghetti formatting". However that is one of the powers of WYSIWYG text processors: you apply the rules and extract those to rulesets once you are satisfied with results, in an explorative way. Direct application of rules is a feature that makes Word what it is.
Now, if a user takes a document with predefined rulesets and still applies their own rules inconsistently that's simply misuse of the tool.
I don't think discussion around fault is meaningful in such scenario altogether.
What would you prefer, instead? CTRL-C, CTRL-V? SHIFT-INS, CTRL-INS? some vim incantation?
As a select-to-copy/middle-click-to-paste guy, seeing people use these inferior alternatives looks extremely annoying to me.
I think they're equivalent and certainly both are better than using the right-click menu.
Using CTRL+C/V requires an unholy synchronization of mouse and keyboard. But unix-style middle-click paste is entirely mouse-controlled and very elegant. Of course, if you are inside a text file, you can use vim keyboard tricks that are even faster because you don't need to select the text.
You can select text with the keyboard by holding down the shift-key, and use the cursor-keys or use ctrl to efficiently jump by word-boundaries.
No mouse required.
From my experience Word from Office360 breaks complex formatting of desktop Word document even worse, than LibreOffice/Google Docs does.
Closest thing that I use is Google docs, sometimes.... The reason I use vim for everything now is - I use vim for everything.
I’m talking about the Microsoft product that is now sold under the Microsoft 365 Online brand.
It isn’t a perfect replacement but for the times I’ve used it, it’s been pretty good.
I've used Word professionally since the mid-1990s. I do know how to use it properly, and it still sucks.
"Writing" isn't meant to be done in a word processor, which was developed as a business tool, not a creative tool. Writing should be done in whatever tool one wants to write in.
Word is, be design, both a desktop publishing app and a secretarial tool. For book-length writing, it works poorly with long files, the file format is subject to corruption. The docx format is also proprietary and subject to Microsoft's whim; any conversion scheme is a hack (though Pandoc and many others do work adequately). Unless you learn the ins and outs of Word's style scheme (and sometimes even if you do) and follow it slavishly, formatting is often inconsistent and there's no certainty that the styles you apply to make your document to make it look a certain way ensure it looks that way on someone else's machine.
There's no doubt, though, that a Word-compatible word processor needs to be in every writers' toolkit, since it is the standard in the publishing world.
[0] https://www.sfwriter.com/wordstar.htm, scroll down or search for THE LONG-HAND PAGE METAPHOR.
Sort of.
It's a ZIP containing a collection of XML files. The actual content is a single file, but you need separate ancillary XML files for things like styles, links, headers/footers, numbering schemes and so on. Each with complicated namespacing and nesting rules and each file referencing items in the others.
The area where it is most at risk (for me personally in practice) is the lax handling of the file by Word itself. As a specific example there are nested XML elements where sibling child elements define properties on the main element, such as properties that define paragraph styles.
Being siblings they should be supported in any order, but in practice generating DOCX files and importing them into Word will fail for no obvious reason, until you reorder them in the raw XML (even though they are still at the same level in the hierarchy). Then they work.
In other words, it's less the 'spec' and more the MS implementation that makes it fragile. And different versions of Word can have different behaviour in that regard.
Kerouac used a typewriter, which is about as minimal as it gets before you drop down to pen and paper. Something like Word is full of distractions: fonts, section headings, various formatting options, etc.
If you really want to get into the flow of writing, do it like Kerouac: plain text editor that wraps words at whatever width is reasonable to you.
After you're done, then copy it into a word processor and apply your formatting rules. Or just stick with the plain text editor and use anything from markdown to tex.
Ultimately, though, use what you're most comfortable with! That's going to be different, sometimes, for different people. The idea that it's a "fact" that everyone should be writing in any particular way using any particular software is just nonsense.
The fact? Should? In "word processing software"? Why shouldn't it happen in "text editing software"? Writing produces text, after all.
When it's time to collaborate, I use Pandoc to turn it into docx and then I send it around and the final formatting happens in Word because that's the easiest for everyone to work with, but the "get the ideas down" phase works best for me in a more "minimal" editor with little formatting.
I love the idea of Quarto, and if I had that when I was in grad school, it would have made my life so much easier. The workflow I see for Quarto is that you can write your paper while you're doing the experimentation because the code is embedded with your thoughts. But in that case, you're mostly slowed down by the research process so it can be a little more clunky to get the writing done because you have time and you're iterating over ideas more than words in that phase. I'd use it now for work in the R&D phase, but I know I won't have a critical mass of collaborators to make it worth while.
There's also Typst which is the new kid on the block but seemingly very good.
It's a bit of a matter of perspective.
The distinction between looking at a document in markdown versus Word is a bit analogous to the distinction between looking at a movie in its textual form as a screenplay versus looking at a movie as a piece of video: Text is capable of abstraction in a way that video is not.
In the screenplay, it might say "table", but when the director translates it to video, the director will have to decide: What kind of table? What design? What period? What texture? Is there anything on the table?
None of these decisions matter to the construction of the story, so, for a screenwriter, it would be very distracting if they had to make all of those decisions just to be able to get "table" committed to the medium.
In Markdown you worry about text and nothing but text. But Word shoves a particular font in your face as soon as you're laying down the first letter, so, if you don't like Word's choice of fonts, you can either let it annoy you throughout the project, or you can start worrying about fonts right then and there, which will be a distraction. If you write Markdown in a code editor, then, presumably, you've already set up the code editor in a way that doesn't annoy you. And then your future self (or someone else entirely) can worry about the font.
It puts the header notation and style (like italic, etc) in-line. So does Typst or LaTeX, and I can't think of any typical stand-off examples for headers and such, but it does muddy the text in-line in that sense. It typically doesn't really slow down the writing, but if you're writing \# as comments for code, then don't have those wrapped in \`\`\`, you can get some problems.
pandoc and some other tools turn those scenes into a full screenplay.
Anyway, to work within your analogy, I would say that Word lets the write do a bit of a 'mockup' of the set with nearly 0 effort. Like "I want a table here", so 2-3 clicks and you have it. Then you can "let the director" take your mock up and flesh it out properly later. As a writing, it helps me to see the mock up of the product as a go, but I want that mock up to be effortless. And as I said above, I do some work up front to make sure that Word's mock up looks good (or good enough).
It's true that many document elements, such as tables, are easy to create in Word. This puts it over the edge into desktop publishing territory. In itself, that's not especially a bad thing, especially if your target is a printed file or a PDF. That still makes it a publishing tool, not especially a writing tool.
Have you tried Libre Office? I'd love to hear your opinion.
Now granted, if you're just hammering out words without any editing this doesn't matter, but I think almost every piece of good writing has had 3+ revisions.
Of course this is a personal preference, but my friends who do a lot of writing (and never used vim at all) still seem to prefer a distraction-free editor with a lot less features and a much less noisy UI for the actual writing.
In Excel, you click on the cell and see either e.g. the number or formula used to get the result you see.
In Word, well, it's just difficult to figure out exactly why a thing looks the way it does.
Can also use control+space to strip off any direct formatting.
2. Add formatting using preferred markup language
3. ???
4. Profit!
Seriously, though, writing prose in a simple text editor and worrying about formatting later is far less distracting than writing prose in a WYSIWYG word processor. Also, adding formatting using a markup language ends up looking far nicer far faster than using a WYSIWYG word processor.
However, I'm not sure that even with greater knowledge of Word's features, whether developers in general would come to like it.
Or maybe those people just have eyes which can spot the differences in spacing between documents created with different Word versions, the mediocre kerning, and a multitude of other typographical annoyances.
Even a website is easier to replicate exactly in another browser than a Word document to be replicated from scratch in a new version of Word. And if by some miracle you manage that, the end result will always look meh.
I used to teach how to write serial letters with Word to secretaries when I was in high school. I used to write VBA macros that call in and out SAP systems when I was a junior software engineer - because "I was young and I needed the money."™
Now I either write in Emacs or Sublime on a Linux box with 2 TB RAM, or in Overleaf (LaTeX collab Web application), and I say: "Word sucks", I shall be suffering no more.
Word does not exist in my operating system (except for QEMU), and nobody notices.
LaTeX creates beautifully typeset publications, and most day to day writing requires nothing more than plain text, which is the most durable format.
I think of this as another version of enshittification -- the acceptance of poor performance as the "standard".
And don't get me started on math equations in Word ...
"Word sux" because it takes forever to start up and I forget what I went there to type. It's like opening a jetbrains ide to take a quick note. The only time that's worth it is when I'm sitting down to write for several minutes at a time, which happens (maybe) once a quarter, vs 17 times a day for quick notes.
Now, you could argue that I just don't want what word offers, but that's what everyone means when they say something sucks. It doesn't do the thing they want (usually because the implementers made trade-offs for other things).
MS Word or markup in an IDE are not the only options.
I used to use Lyx quite a lot. The best of both worlds in many ways.
There are text editors which give you a simple UI than an IDE without the distractions and complexity of a word processor. I find Kate quite pleasant to write in.
There are distraction free writers editors (which will often save in wordprocessor formats) that exist only because a lot of people find wordprocessors are a "disaster for the creative process".
So, as it must be me not knowing, can you tell me why Word won't let me change table column size ~50% of the time (for the same table from the same source). Half the time autofit works (yay!), half the time it won't let me resize columns, neither by typing the number, nor dragging the invisible off-page divider (Microsoft should just make the transparency of off-page content 50%??), nor changing to draft view and dragging the actual dividers ... wtf is going on there? If I paste it into OneNote first then it fits ... sometimes, if I drag a column wider first (which it already rendered 5x the width OneNote did) then it will let me narrow it afterwards, what's that feature called AutoNoNarrowColumnRenderedIncorrectlyFiveTimes ExpectedWidth?? I'll just search the settings to turn that off ... oh wait!
These are the times I long for markdown, or 'reveal codes' ... MS Word has a lot of problems they could probably have fixed if they hadn't put so much effort into preventing interoperability. These sorts of issues were around 20 years ago when I stopped using Word, and 5 years ago when I restarted. Same asinine poorly implemented numbering and styles that are unintuitive, opaque, and ungainly ... and don't get me started on search! Multi-highlights? Sorry best I can do is "find next" with no find previous, no regex, ... you can do find in AutoText though, right, right? ... and all the AutoText and AutoCorrect gets saved in a single sensible format that's easily modified? ... Word changes the format of all windows when you open a new one too, just in case you thought the suck was restricted to within the window chrome ... and doesn't have always-on-top, and doesn't open windows in their last position, and ...
Doesn't suck ...???!
Whilst you're here, any ideas why OneNote eliminates footnotes so you can't cut-paste between Word and OneNote? I'm sure it's not flawed and I'm just holding it wrong, right ...?
Pandoc was a game changer for me. I picked up LaTeX equation editor pretty quick, and being able to write markdown was so much more pleasant in my mind.
It’s not perfect; tables are a pain still, but I have no desire to go back to Word.
I think part of it was that I had basically memorized all the keystrokes for the MathType editor, and most of them didn't work in the MS Equation editor, which annoyed me. Also, I had issues with parentheses formatting correctly but I suspect that's been fixed in the last 15 years.
Still, I really do prefer to work with Markdown in general. The Markdown -> Pandoc -> LaTeX rendering just ends up looking prettier in my opinion, and at this point I'm pretty useless in any editor that doesn't have Vim keystrokes. Pandoc irons out the parts of LaTeX that I really hate (the `` vs " being the thing that's given me the most headaches), while letting me drop into raw LaTeX when I need it; not even getting into the fact that there's just math stuff that (as far as I know) doesn't work in Word or MathType's equation editor (e.g. bussproof trees).
I do get pretty annoyed when people try and tell me that regular LaTeX is "just as easy" as Word, because even as someone who has a reasonably good handle on LaTeX I can say that is just not true. TeX is arcane and weird and annoying and inconsistent, and I don't blame people for using Word compared to it.
Uh, that's just like, your opinion, man.
You might be interested in Texmacs [1]. It is has a wysiwyg interface, and it handles nicely tables and mathematical equations. Also you can export documents to the latex format.
Or Scrivener. No Zotero plugin, though.
It depends on the person. The best way for me to write is on paper, on a desk with lots of empty space, with paper versions of all reference material.
I would like to quote this on a MonsterWriter landing page
markdown -> pandoc -> docx -> upload to google doc -> share
Gdocs would then really only need to support the same semantics with underscores, asterisks, octothorpe heading levels and title sizes.
# Hello *world*
## Math
$$x = 2 \cdot y^3$$
- foo
- bar
- baz
1. Yes
2. No
3. Maybe
---
Right Left Center Default
------- ------ ---------- -------
12 12 12 12
123 123 123 123
1 1 1 1
Table: Demonstration of simple table syntax.
pandoc -t html input.md | xclip -selection clipboard -t 'text/html'
Other than that, I strongly agree. Word is also de facto standard when you want others to open and edit your document.
Emacs has `fill-paragraph` built in and `unfill-paragraph` is a short function definition [1]. Both work across multiple paragraphs.
You are absolutely right and I understand what you mean though. I am open to trying other alternatives and I should try to come up with a better method to workaround this problem.
Never tried Emacs, just went with Vim so far. Did not know that Emacs had already an elegant solution for this problem. Nice! :)
to record a macro that joins a single paragraph together
<N>@f
to then replay that macro <N> times
I am certainly not gonna be learning and switching to emac just to perform this task for unwrapping a .docx file
Never used Shellcheck?
I also very much like Pandoc-markdown's ‘simple table’ syntax, because it's actually human-readable and human-writable without confusion and pain.
Back in the day there were a couple of utilities (I think they were "catdoc" and "wordview") which could take MS Word input and generate text-only output. Good for reading MS Word attachments in mutt or from a terminal / console / SSH session.
<https://wvware.sourceforge.net/>
A primer on lessfilter:
<https://www.miskatonic.org/2020/06/24/lessfilter/>
And lesspipe:
Currently, I just open vim in the terminal, write what I need, then copy the text. I open dmenu using a shortcut, type something like "unwr", which is sufficient for selecting the "unwrap-clipboard" script of mine, press enter, which unwraps the text on the clipboard. Finally, I paste it in the email client.
I know it might seem a bit tricky, but it's better than what I did before. I used to set the textwidth to a really large number like 9999, highlight the text I wanted to unwrap, and then type 'gqq' for formatting. And don't forget, you also have to wrap the text back if you don't want to change the original format of the text file you wrote.
You can use soft-wrapping in vim, but I don't prefer it. Soft-wrapping is not as convenient to me as hard-wrapped text when using vim shortcuts. For instance, if you are using soft-wrapping in vim and you press 'o,' the insertion mode will start at the end of the paragraph because vim will consider the entire paragraph as a single line. However, there are many situations where I only want to insert text in the middle of a paragraph. You will also most likely set j to act like gj and k to act like gk, to make the cursor move between lines that are visually separated but actually form a single line. I don't like this either.
Someday I may try that again, if email is still relevant by then. Vim and email would be my perfect set up.
Wow, this is very interesting, and I might even try it at some point. It might have been a bit challenging to sync and read the emails though, but the sending part seems nice.
Was the reason for why you gave up related to syninc emails?
qlmanage worked great for HTML messages and attachments, though.
I've got it installed on MacOS and Linux, occasionally use it for longer HN comments.
The extension automatically invokes a vim session with the contents of your current browser edit window, and reads back in the output of your vim session when you're done with it.
<https://addons.mozilla.org/en-US/firefox/addon/edit-with-vim...>
:set tw=0 linebreak nolist
There are recipies which will reflow paragraphs to the end of a document (or a given mark), but here one of the issues is text which shouldn't be reflowed, say, Markdown syntax for tables, lists, possibly blockquotes, code blocks, and the like.
My preferred solution is to write either single-sentence-per-line (rarely) or flowed text in vim using:
:set tw=0 linebreak nolist wrap
While I am very fond of both tools, Pandoc's command-line interface seamlessly integrates with my understanding and intuition, unlike FFmpeg's byzantine command line options and concepts. With FFmpeg, I frequently find myself documenting specific incantations and recipes in my notes, lest I should forget how to solve certain conversion problems. Pandoc, on the other hand, has spared me from such cognitive overload.
This should not be taken as criticism of FFmpeg though. FFmpeg is solving very complex problems too. Both Pandoc and FFmpeg are excellent tools. Both tools have saved me hundreds of hours of research, experimentation, trial and error, etc.
See: <https://www.etymonline.com/columns/post/bio>
(Bottom of page.)
I also went ahead and set up my root "notes" folder to be served with mkdocs so I can easily browse them, and just render a PDF when I must submit some file to a third party
https://en.m.wikipedia.org/wiki/John_MacFarlane_(philosopher...
> If you need to convert files from one markup format into another, pandoc is your swiss-army knife
It also sits at the heart of Quarto[0], which adds Jupyter-like code execution (in R, Python and others) into document production. Combined with RStudio as an IDE, it's my new favourite way to write anything - from static documents to full on code notebooks.
No affiliation with Posit, the company behind Quarto & RStudio. Just a happy user.
--
[0]: https://quarto.org/
--
There's a long legacy here, it does nobody any good to disregard it. Maybe Knuth is well-acknowledged for his invention, but I think for instance Yihui Xie is a little under-recognized.
When talking about Quarto within the R community we usually frame it this way, but obviously it’s not a very useful description if you’ve never heard of RMarkdown.
Though I guess we can stop there in this particular case, unless someone knows of an example of literate programming that precedes Knuth. I'd be interested if there are such examples...
I want to write a small book that I want to generate in 3 formats: HTML pages, EPUB and PDF. What is the best input format (source format) for the book? Pandoc Markdown? CommonMark? GFM?
I'm a little hesitant to committing myself to Pandoc Markdown or any Markdown because they all have tiny differences with each other. Each is like its own standard.
I considered Org-mode for some time but there are so many edge cases in which Pandoc does not parse Org-mode properly. I mean sometimes simple things like internal links are not rendered properly by Pandoc in the generated output.
So what's the best format to write the input in? Any ideas? Opinions?
For works which consist principally of standard sections (e.g., Book / Part / Chapter / Section / Subsection / ...), fairly standard font styles (normal/roman, italic, bold, code blocks / pre / poetry), footnotes/endnotes, and perhaps a few tables, illustrations or images, Pandoc-flavour Markdown is far more than sufficient. Writing or formatting is virtually seamless.
If you're writing something with more complex internal formatting, then I'd lean more strongly into LaTeX. You can do most of your initial authoring in Markdown and generate LaTeX from that, for further finishing work, or simply start with LaTeX. The key discriminator here would be either mathematical formulae or complex image placement. Note that creating the output you want in HTML or ePub (itself effectively a specialised HTML format) might still be challenging.
The next step up would be a specific layout tool (Krita or Adobe Illustrator, say).
But start with Markdown + Pandoc and see if you like the results. It should be Good Enough, and if not offers a smooth path to more powerful tools.
Pandoc Markdown-as-input is probably the best-supported input format for Pandoc, as far as "reasonable defaults for outputs in other formats" is concerned, and it's broadly compatible with the norms of other markdown styles.
You can always drop into latex, include custom CSS headers, etc. It's also a format where the "formatting" won't generally get in the way of actually-writing, unlike HTML or LaTeX (speaking on behalf of mere-mortals, here).
I'm writing a small book. I shared my experiences with Pandoc and Asciidoctor in case it helps you or anyone:
https://adammonsen.com/post/2122/
Your use case may differ from mine (I didn't see you mention printing), but my anecdote above might help suss out tooling differences between Pandoc and Asciidoctor.
Here's an example printable book generator using Asciidoctor PDF:
Won't work if you want to keep everything local, though.
If you want to look down the path of implementing book layout yourself, here are two breadcrumbs from my bookmarks:
https://journal.stuffwithstuff.com/2014/11/03/bringing-my-we...
https://iangmcdowell.com/blog/posts/laying-out-a-book-with-c...
That's one thing I learned, having written a couple of books. Even though writing is tough, it is easier in many wyas than the marketing.
However markdown - if you stick with the subset that everyone supports is the most widely supported alternative. If you go with a specific markdown you lose support for something else you might want. If you go for non-markdown you will lose support for most of the world.
I personally selected restructured text which is really powerful for the complex documentation I'm trying to create. However I keep running into nothing else supports it problems (I can extract doxygen from C++ - but only with tools that don't support the latest. I haven't figured out what to do about Rust documentation)
Then again, content is king. Write it on napkins if you must. When complete, you can spend two days to transcribe to whatever format the publisher requires.
However, the open source tooling doesn't support what I need for a physical book, so I chose to use pandoc and markdown instead mostly for to the market size of markdown.
(I previously wrote my own tool chain for rst to latex and epub, so I'm week seats of what features are needed to make digital and physical books. I was sick of using a format that had limited tooling while the world had moved on to markdown.)
I mean I did not try AsciiDoc until now because there are so many choices of input formats and the ones I've tried so far have been disappointing one way or the other.
I talked about Org-mode rendering broken in edge cases. Same with Latex too. I see Pandoc has first-class support for its own Pandoc Markdown format. But the support for all other input formats seem patchy.
If you think Pandoc has good support for AsciiDoc without any edge case issues, I'll be most certainly trying it out.
The "internal links" thing is a pain, admittedly. I have an idea for a workaround:
* sprinkle hidden, unique <a id="ch1.2"></a> around
* on GitHub, use links like chapter1#ch1.2
* for Pandoc, preprocess to remove the filename before the #
I'm working with a big enough book that it's an undertaking, so I haven't done this yet.
For simple things, you can easily write Markdown that is compatible with all dialects. Mainly, remember to indent your lists 4 spaces if you soft wrap.
But Pandoc Markdown has the most extensive support for other extensions, like footnotes, figures, etc. That's useful because it minimizes how much HTML or Latex you need to write, which in turn makes your documents more portable.
There are other formats that support more features, but in my experience the communities are smaller and the syntax is not as pretty. Ultimately you're betting on that format continuing to exist longer than Pandoc, which I think is not a great bet in most cases. The only format which I think might have better long-term support and compatibility is CommonMark, but it comes at the tradeoff of substantially fewer features. Which again means sacrificing portability because everything you can't do in the base language you need to do in HTML or Latex.
I use markdown for the 90% of my writing that is blurbs (and parse it into something like a graph/knowledge kb, and often render via pandoc to pdf, word, Anki, and html).
In both cases, I restrict myself to using features that can be parsed.
I've written multiple books with it and have also used proprietary tooling of publishers. I have a few plugins I use to customize my books, but have yet to find a tool that I wouldn't need to customize.
Does Markdown have any way to specify eg "begin a chapter on a new page" ? I don't think this is really a thing in Markdown or HTML but I'm admittedly a casual Pandoc user.
So, then you convert each chapter file to pdf, and then join the pdfs.
Google has many examples; one's here using a header file: https://medium.com/@sydasif78/book-creation-with-pandoc-and-...
More are here, including an example using the header-includes YAML metadata param: https://github.com/Wandmalfarbe/pandoc-latex-template/issues...
Pandoc handles these fairly seamlessly, and with many options for PDF engines, though I'd say it has a preference for LaTeX and HTML in the backend and Markdown in the frontend, based on my experiences with the edge cases (sometimes entirely solvable with a little Haskell or Lua).
Since LaTeX is the default for PDFs, it pays to keep that in mind and help LaTeX help you (you can use it inline with Markdown or included as preamble in configuration), but sometimes I've just had better luck converting via HTML to PDF ("-t html output.pdf" or directly chaining on from output.html) for what I'm writing in the moment, though other times I'm not stressing LaTeX as much and can just go straight from Markdown to PDF (for example, just writing up something with inline maths). I prefer to avoid LaTeX or HTML's escaped character encoding and often need far more than a single Latin font can provide, so I've ended up dealing with LaTeX's limitations here (even in lualatex and xelatex) more than what I'd suspect is typical. Meanwhile, the standard HTML to PDF backend uses Qt, and I've found it works for everything else I've needed when LaTeX isn't the right backend (and it does come up). On one occasion, I did have to switch that to weasyprint, and that was everything sorted. Alternative backends is an unsung power that few have, while pandoc not only has many built-in (or it is at least internally aware of) but will also integrate with any CLI needed.
Output to all three with HTML, EPUB, and PDF can just need a bit of fiddling before it comes out right, depending on how much you're willing to mess with specific metadata for each versus accepting the limits of what Pandoc can handle universally in its AST. Invariably, some compromise is required, but the core semantics of Markdown (including extensions) almost always translate without an issue. The dialect problem of Markdown is really just in the confluence of said semantics with things that have not been separately included, such as the lack of an actual header in Markdown (Pandoc here allows YAML for some, or you just fall back to HTML).
So, tldr; there's no "best" input format, except the one that you find most comfortable to just write the book in, but I find Pandoc is usually best approached from Markdown with the LaTeX or HTML backends. It's powerful and oh so very handy, but it's not going to do all the thinking for you, just a lot of the grunt work, same as any other tool. When in doubt, the user manual is quite readable, and I've found it answered almost every question I had. When it doesn't, other people do, and when they don't, it means I'm either going about it the wrong way or I get to solve an actual problem (but usually the former). But, as always, the most important thing is actually writing it, distribution comes later, so focus your efforts on that and the tools you need to do that effectively.
I find Org-mode the easiest but like I said in my comment, the conversion quality is not great. Pandoc breaks a lot of stuff in Org-mode in edge cases. One example I shared in my comment was Pandoc breaking internal links.
So by selecting something I find the easiest I have burned many hours of troubleshooting figuring out why the output does not look right.
That's why I want to draw upon the wisdom of the community here to find out which input format works best and by best I mean flawlessly. No edge case issues. No rendering flaws. If I get the specific recommendations, I'll try them out for sometime and then commit myself to it instead of burning more time trialling all of the different input formats.
Again, I find that Markdown (with inline LaTeX or HTML) seems to be Pandoc's preferred starting point, and that the HTML backends are quite useful (particularly when not needing full LaTeX), so perhaps there's some luck to be had there, since HTML may preserve Org's linking and such a bit better, though I don't use Org myself so can't attest to it. And if there's really a problem, then perhaps Pandoc needs some help sorting Org-mode out!
Org mode can also export html and markdown, so that's three potential pandoc inputs, with potentially different properties. All of which might be massaged before input. And in extremity, an org-mode parser permits emitting customized input. Then pandoc's parsing and filters permit altering the pandoc ast in flight. And the ast isn't hard (assuming comfort with ASTs), so if some other tool has templates and output one likes, one might skip the pandoc backend and emit it oneself from pandoc ast json. Rather than hoping to persuade that other tool to both accept and generate what's needed.
So for instance, last year I had a project written in a project-specific markdown dialect, kludged to pandoc-flavored markdown, parsed with `pandoc -t json`, and html emitted custom from the pandoc ast. With embedded directives from dialect to emitter. And html templates copied from non-pandoc tools. In a language with nice pattern matching (julia's Match), the emitter was a short page of code.
"Avoid reinventing wheels, but sometimes it's easier to assemble a satisficing custom vehicle, than to find and adapt a previously-built one."
He's inspired a small number of long-time, serious contributors.
Together they maintain a super-high-traffic utility that simplifies the very arbitrary complexity of document formats for untold millions of users.
It's really a stunning example of social good.
I learned LaTeX late in life for a few reasons: - I publish in journals that all have different layout requirements, so reformatting to submit to more than one is a big pain. - Using a non-WSYIWYG editor enforces clarity. It's a lot easier to see each sentence as a whole. If a sentence is longer than a single line in VSCode it should probably be more than one sentence. - The features of an IDE allow you to see each line (sentence) in one place, and move things around more flexibly with the IDE shortcuts. You can easily rearrange the flow of a text. And of course you can make inline comments without having to clean them up before sending the doc somewhere. - I don't have to care about things like image placement and anchoring in the text until output. This is an area that WYSIWYG editors like Word are particularly painful.
There are other reasons like equations and notations, but that's enough for now. All that said here's my workflow:
I write LaTeX in VSCode (soon switching to vim). Then, I can use Pandoc to convert to Word if I need to (it's still where most of the templates come from in the discipline). This is also helpful in working with collaborators in my area since they typically won't know LaTeX.
Here's where it gets fun - I write LaTex, then use Pandoc to output directly to .icml (InDesign Markup). These link directly in my InDesign document, and so I can edit text where it's better and more clear to edit (IDE), then seamlessly get it into the environment where I have maximum layout control. I haven't gotten to do this so many times that I need to write a script to automate conversion as part of my tex build, but I probably will soon for fun.
Pandoc just works so well, allowing me to concentrate on making good content and not having to sweat all the annoying file conversion details. Thank you to the developers and maintainers.
1. https://www.jillesvangurp.com
You can turn any file into anything. PDF to rtf, latex to .doc, etc. It does a great job. Written in Haskell, too!
In a way, I hope I never have to use Pandoc again, but I'd hate for it to not exist.
But I was unable to align the following 2nd table with taxes: cells are all over the place and it does whatever it wants. And there is no information online to be found about it.
(I eventually gave up long time ago and still to this day manually do them in LibreOffice Writer adding taxes with a calculator)
Except this, it's a really neat piece of FOSS software!
https://hledger.org/invoicing.html
https://plaintextaccounting.org/
One previous discussion: https://news.ycombinator.com/item?id=20012499
Might have to use a headless chromium wrapper (maybe pandoc has this anyway) to then get to pdf but that may not be too bad
In fact I quite often go .md -> .html with pandoc, but write the .md in such a way that, when translated, it is the kind of html that weasyprint will be able to turn into the PDF that I want.
It’s an ugly script that’s been working quite well for more than a decade, but I wouldn’t recommend it to anyone other than myself.
For example, in my code, if a table has the class “total” it sums all <td>s which contains a dollar sign, and so on.
When you need them you need them, and nothing else quite works, but I have to re-learn them every time.
I wonder how ChatGPT would do with focussed pandoc requests.
Can you think of any command-line tool you might not use for 6 or 12mo but when you crack it open after a long time it is intuitive how to do what you need to?
I hypothesize muscle memory is required for efficiency at the command line.
Most of them that I do use; sed, awk, even perl. I don't bust them out often, but when I do most of what I need is pretty "front of mind".
I'm not asserting it's necessarily a tool issue vs a "me" issue, but jq, parallel, and pandoc (although pandoc the least of these) just don't "click" with me, and even if I use them multiple times a month I have to go back to previous commands, --help, or man pages routinely.
Then, fzf your history.
I also use Anki SRS flashcards and put a lot of tool usage exemplars in there; it not only helps me remember a bit of what I need to do, but if nothing else I remember THAT I put it there so I can use that to go look it up again. And this is coming from someone who grew up with and is comfortable with `--help` screens and man pages.
However, often I'll open up a GUI I haven't used in a while and feel like I'm just as lost as I am with a command-line tool I haven't used in a while. I rely on notes I've taken and try to stick with stable software.
It made university more accessible, as I get frequent hand cramps while writing notes. So I started to take them with Pandoc and added custom macros.
The best feeling was when professors would ask for a copy of my notes at the end of the term because they were formatted so well!
I recently learned you can use LUA to write custom plugins and change some of the converting behavior. I'm using it for example to create slides similar to the "sent" program.
It helps me bootstrap new presentations and talks very quickly: https://github.com/KarimJedda/justslides
Statically compiled cross-platform program for convertng markdown to self-contained portable HTML with a nice styling embedded by default. Simple, small, fast, hackable, written in Nim.
I end up using it for random doc generation too in my day to day. It's just a damn fine piece of software that always works and has every feature I could dream up for my use cases.
I used to write papers and slides in LaTeX (using vim, because who needs render previews), then eventually switched to Pandoc (also vim). I eventually discovered RMarkdown+RStudio. I was looking for a nice way to format a simple table and discovered that rmarkdown had nice extensions of basic markdown (this was many years ago so maybe that is incorporated into vanilla markdown/pandoc).
The RMarkdown page claims:
> R Markdown supports dozens of static and dynamic output formats including HTML, PDF, MS Word, Beamer, HTML5 slides, Tufte-style handouts, books, dashboards, shiny applications, scientific articles, websites, and more.
...which I think is largely due to using pandoc as the core generator.
RStudio shows you the pandoc command it runs to generate your document, which I've used to figure out the pandoc command I want to run when I've switched to using pandoc directly.
This is a bit of a "lazy" way to interact with pandoc. Maybe the "laziest" aspect: when I get a new computer, I can install the entire stack by installing Rstudio, then opening a new rmarkdown document. Rstudio asks whether I'd like to install all the necessary libraries -- click "yes" and that's it. Maybe that sounds silly but it used to be a lot of work to manage your LaTeX install. These days I greatly favor things that save me time, which seems to get more precious every year.
Maybe one day I’ll write a blazing fast Markdown parser that does exactly what I want, full control and maximum simplicity and all that. In all probability though, I won’t. I really like what little I have seen from Pandoc.
Recently rewrote a content stack to use Markdown (among other formats) for the source, the file system as the database, generating outputs (including HTML with embedded Mathjax LATEX) via pandoc, and it works absolutely brilliantly. Fully recommend.
The other comment nailed it pretty well, though they hedged it by citing habit (presumably to counter the weird anti-docker trend that has arisen). Dockerizing (or simply containerizing) most vendored products is a choice that is often beneficial, and the marginal overhead is a rounding error.
For single-file binaries this is less of an issue, but even those sometimes require dependencies that you already have something else that needs a different version of that same dependency causing conflict.
But even with packages in a manager, there have been times where different packages conflict on different dependencies that when packaged up in a well written docker image (which not all are) would have obviated.
As a developer I see this more with programming languages than other things, and this very thing is what led to language "package management systems" like rbenv, sdkman, asdf, and the like.
I use mdBook [1] for web versions though. I found the default setup much easier to use. And it came with themes (light/dark/etc) that readers can choose.
https://m.mediawiki.org/wiki/Extension:PandocUltimateConvert...
https://chat.openai.com/g/g-YX1CmSAA9-franz-enzenhofer-pando...
installs pandoc and then you can interact with it via chat interface.
sad that to PDF conversion does not work, as it would need pdflatex and I can't find a simple downloadable version (amd 64 linux) anywhere on the internet. If you have one on hand, please link it. I willl upload it, too. then PDF conversion will work.
[1]: https://github.com/furiousteabag/asmirnov.xyz/blob/master/bu...
I haven't had time to look into it a lot but I think that would be amazing.
Has anyone tried integrating the Pandoc command line tool into a desktop product on Windows or Mac?
How big is it?
What are the licensing implications of shipping it with my (closed source) software?
The static site generator for my personal site is just it with a thin bash script wrapper: https://git.sr.ht/~kb/open-notes/tree/main/item/.build.sh.
So much simpler than pulling a universe of node modules.
I previously used restructured text and had to write custom tooling, but now I can write markdown on Jupyter.
Just check pandoc installation and get started guide. Overwhelmingly detailed, like I never use command line before. Maybe it assumes many users of pandoc are writers or Information Developer, without much tech skills.
In theory I like its design. as a CLI guy
In the late 1990s / early aughts, I'd written a toolchain to generate multiple document formats from a source based on HTML fragments. With Pandoc (and usually Markdown, occasionally LaTeX), I've discarded all of that, and have greater utility (more outputs, and FWIW, more input formats as well if I choose those).
though I've used this program too in the past to seamlessly convert html to text to easily import into a database...