Writing a book in the age of open source
blog.incrementalforgetting.tech
blog.incrementalforgetting.tech
Yes, it's worth optimising for your productivity. It's not the be all and end all. I've written at my desk with the comfiest chair (A Mirra) I have, and the most ergonomic keyboard for my needs (Ergodox EZ). I write at cafes with just the laptop. I write on the couch at odd but comfortable angles. I write on public transport squished against strangers.
I love using AsciiDoc as the tooling (asciidoctor + friends) give me output that looks decent, and the way I _input_ into that is not mind-breaking like Docbook is. Asciidoctor gives me a PDF which I then style how I like with CSS and then can put on leanpub.com and sell for real dollars.
The way I would put the writing section for tech books is this:
Start with the _topics_ you want to cover. Make these the chapters. Then dive into each topic and figure out what you want to say about the topic. Usually 3-4 main points per chapter. These come out to be your subheadings. Order the chapters from beginner-to-advanced concepts or in a way that makes sense for the book you're writing. For the books I've written it's usually start with a simple base app and then incrementally build things on top of that.
I actually only wanted to focus on the tooling, but that lacked a little body, so I added a little bit on writing.
It actually deserves its own article
Apologies for being Mr. Negative Guy. I hate folks that do that.
But as somebody who is extremely technical and loves tools and coding, moving to long-form fiction writing, I find that every minute I spend on tooling is _not_ writing. It might be good and necessary, and tools can do some amazing things, but it's not writing.
Fiction writing, at least for me, is a bit of organized struggle with my subconscious. My brain would much rather be coding, writing build scripts, creating a huge cross-layered tagging system, commenting on HN, and so forth. That's because "struggling with your subconscious" is not something we teach or emphasize.
There's a really strong allure of tools, process, and tooling that calls to us when our minds don't want to do the work. Buy the product, learn the tagging, follow the recipe, etc -- it's all about hey, don't do the work, have the tool do the work. Don't be a dummy!
But since I don't know the work myself until I create it, there's no tool or process that's going to do anything but distract. Love my tools, spent a lot of time with them. I spent all day yesterday re-creating and re-factoring my build pipeline from markdown to about a dozen different formats. But that wasn't work, not really. It was avoiding work by doing (hopefully) very useful tangential things.
I'd love to write a tooling/process essay or book. I will not. This is why.
https://gist.github.com/dcode/0cfbf2699a1fe9b46ff04c41721dda...
To others reading along, admonitions or callouts, noted here as key motivations for asciidoc, are supported in quite a few flavors of Markdown, but specifically GitHub Flavored Markdown:
https://github.com/orgs/community/discussions/16925
This handles Note, Tip, Important, Warning, Caution.
(After years of incremental work, a suggestion last week complains it's not using mkdocs admonition syntax -- it must be a pain to make custom extensions for a global userbase.)
As easily as we would google an event in history, he could bring up the tagged items associated to a chapter, world event, or character in his book.
At the late stages, when he was modifying a lot with the help of his publisher/editor, this was worth its weight in gold.
Amazing level of discipline and wisdom choosing to set it up in that way for himself.
I would recommend similar to anyone considering writing a book.
His book was recently published and is sold in stores everywhere under the title “Gods-Forged”.
I’ve in the past, suggested Obsidian. He is interested in trying it out.
He’s going to look into Obsidian for his next books.
I know a developmental editor, and occasionally we bond over the the overlap in our fields of resolving rafts of conflicts between versions of big text datasets.
I remember I once showed off Mediawiki as a "just so you know this kind of cross-referencing tool exists", but I think they have their own note-taking strategies.
With fantasy settings in particular it's crucial to keep the setting consistent, since the book is the only "source of truth" on the matter. This sounds like the way to go. Excited to check it out.
In my (very minuscule) experience in writing 99% of the time is spent switching between browser, scratchpad, notes, and standing up to take a walk and think about it.
Start with the minimum possible format you can (plaintext) and step up a standard when necessary. The simpler the standard, the more software there is that just works with it.
I’d say ergonomics, organization of information, and the “feeling” of using the tools are the driving force behind all these efforts of optimizations.
In fact, I have a very short book about my step-by-step process on how to format and publish your own book with LibreOffice.
I recently wrote a blog post giving the cliff notes version of the book in a couple pages but I seem to have accidentally deleted it. I need to go dig it up.
Reminds me of how my son watches youtube at 2x all the time and probably would crank it up faster if there was a higher option.
https://chromewebstore.google.com/detail/enhancer-for-youtub...
https://addons.mozilla.org/en-US/firefox/addon/enhancer-for-...
And I totally agree with your son, YouTube and Podcasts are sometimes just slow or waste time with irrelevant/boring information, and changing the playback speed is a highly underrated way to deal with it. You do get used to 2x after a while.
[0] https://addons.mozilla.org/en-US/firefox/addon/videospeed/
[1] https://chromewebstore.google.com/detail/video-speed-control...
Asciidoctor was a godsend in this regard, able to output in PDF, EPUB, MS Word, whatever you want.
With Asciidoctor, ImageMagick, FFMpeg, LibreTranslate, and about 800 lines of Python code I'm now able to generate any part of the book draft in any language (this is mainly to build excerpts for publishers).
I also do literate programming (write code as if it's a book written for your future self) with this exact same approach. Just write a two line script that removes everything that's not between ```python ``` and you can code inside Obsidian. (E.g. you can use a markdown parser like `marko` for python) I personally don't need syntax highlighting, but if you do, you can open it in your IDE/Emacs/etc and IDE should highlight most languages in Markdown files. You obviously don't get all the IDE bells and whistles, for those you can have separate files and import them in, or enable Python mode in markdown files (somehow). (Obsidian should support basic syntax highlighting for most common languages anyway)
Since Obsidian is so awesome, this approach has endless possibilities. E.g. you can do project management inside Obsidian via Kanban board, you can use Excalidraw tool in Obsidian to draw architecture diagrams, write music inside Obsidian with its Lilypond plugin, write math with LaTeX plugin etc. It just automatically works out of the box.
As an example, let's say you tagged every time a character is mentioned (will be done automatically as well with 'untagged references' for proper names). Or you have all of your backstory and timelines in sections that are linked to your writing, but you can output the finished writing without all of those links in there. So as an author you get a full timeline with metadata. You can also use additional tools for research with Obsidian plugins, including Zotero for academic references, Markdownload Browser Extension for shooting MD and images straight into your vault, git plugins inside Obsidian, etc.
Sorry I'm rambling, but absolutely go look at it again!
I sometimes use Git branches to explore different story ideas but I mostly just use the system for basic versioning and history. I also use Git for written roleplays to store my logs and thoughts, and since MD is basically text I can use Unix tools like grep and wc.
https://dave.autonoma.ca/blog/2019/05/22/typesetting-markdow...
Or the editor I wrote for this very purpose:
https://keenwrite.com/screenshots.html
The last screenshot shows a single source file having multiple themes applied, one being a manuscript theme.
The editor integrates variables in an external file so that if I want to change a character's name, I can do so in one place. Even diagrams (e.g., family tree) are updated.
[0] https://oxide-and-friends.transistor.fm/episodes/rfds-the-ba...
I only wish that it had better integration tools with other wiki tools (confluence, notion, etc.)
That way I could use asciidoc for all my documentation and sync it to whatever wiki tool the company I work for wants to have
Repo:
https://github.com/bigskysoftware/hypermedia-systems-book
Blog on the port:
Though I use Typst on its own for my CV since I primarily needed it in PDF. Almost every other project I create starts with `quarto create project`. https://github.com/mintyfrankie/brilliant-CV
I assume this means the internal versioning features that Word offers, and not something like Git. I sympathize, I imagine the editing process was rough.
Saving complete drafts, then receiving marked-up copies, worked fine for fiction, at least in my experience. They've been using Word for decades at this point so it's quite formalized. You just follow their instructions (even with file naming conventions). Trying to get a publisher up to speed on Git while your editing is usually just not worth the hassle.
Technical writing, though, is a different story. Managing code blocks is pretty essential in git, in my experience. But then you have specialized publishers for those.
- first loads some packages and defines all the customization commands to do nothing
- second defines the customization commands and applies the formatting as specified by the publisher
Edit the LaTeX text, add customization commands for shortening/lengthening paragraphs, &c. for balancing pages and tweaking float placement --- make PDF, send to printer, then at the end of the project, comment out the second package line and return the updated source to the author for their next edition.
That said, my current books are being done using:
- GitBook: https://willadams.gitbook.io/design-into-3d/2d-drawing
- using Literate Programming completely in lualatex for coding up the OpenPythonSCAD module: https://github.com/WillAdams/gcodepreview
- in normal LaTeX (re-setting of a translation of an Old English poem)
and I look forward to a future project working in LyX, since the new 2.4 looks _amazing_.
I'm not a Microsoft Word fan, but why does their tool need to be free and open-source?
They went to great lengths to explain how the performance of the tools is so important; only to exclude anything you might have to pay for or is closed source.
Who cares if you have to shell out a few bucks for something if it works well and provides great value (i.e. saves you more time and hassle than the money you paid for it)?
- encourage development of libre software, to reduce entry barriers for newcomers, allow experimentation, sharing knowledge and reducing trillion-dollar companies dominance
- commute with bicycle or electric vehicle, so the planet lasts longer, and anti-liberal governments controlling the resources enjoy our money little less
- don't plan holidays in countries which are stuck in teocracy, autocracy or exploiting their citizens (N Korea, SA, China...)
- don't buy products originating from such countries, or produced with use of child labor or animal suffering
"My life amounts to no more than one drop in a limitless ocean. Yet what is any ocean, but a multitude of drops?"
Many people can appreciate Org-mode the format, but aren't fond of Emacs the editor, which can be a hindrance to collaboration. The way I get around this in a professional setting is that while I'm working on documents and gathering feedback on them, I take pains to ensure that I can export snapshots to OpenDocument Text format--which can look nearly as good as LaTeX. Google Drive can import documents in ODT format and subsequently Google Docs renders them quite nicely. I accept feedback via the "suggestions" mode of Google Docs, and only after I incorporate the changes in the Org mode source do I accept the changes in the ODT/Docs version.
However several comments here mention the convenience of a workflow based on Markdown + GIT. Exactly that workflow is the heart of Leanpub[0]. That site facilitates writing into text files organized and stored on Github (or Dropbox), compiling at any point to a finished-looking e-text, which you can then sell via their storefront[1], or take the finished ebook or pdf to a different platform. Leanpub supports their own Markdown variant (sigh).
[0] https://leanpub.com/authors
It just felt like going to a knife fight with a Bazooka.
Finally, one thing that I really, really wanted, and YMMV. There's no reason why you should care about it, but I did: drop caps (the first letter of a section is 3 or 4 lines tall). I just think it's cool. You could do it with a lot of machinations in Scrivener, but it's trivial with Vellum.
Vellum knows all the things about publishing conventions. What I prefer about it is, you pay money for it. That gives them a stable business model.
When writing an essay, I often start with pen and paper. Mostly, an outline, and sometimes also the introductory couple of paragraphs. But I switch to a word processor when the bulk of the words start flowing.
https://digitalsuperpowers.com/blog/2019-02-16-publishing-eb...
Did you get the chance to try it out? Does anyone know if its worth checking out?
Felt like a step backward to me; asciidoc is so powerful, even if some of the syntax was weird.
I can't imagine someone calling themselves a developer finding writing asciidoc difficult. It isn't any harder than markdown. It only has a slightly different syntax[0], and more features.
I've also faced a lot of resistance from people when asking them to migrate from markdown. My, unfavorable, opinion is that it's simply the usual reluctance to change and unwillingness to learn something new, deal with the short term pain of learning, despite longer term advantages.
[0]: https://docs.asciidoctor.org/asciidoc/latest/asciidoc-vs-mar...
* The ordered list isn't the best way to write numbered lists. Numbered lists can be: 1. 1. 1., etc. The computer will auto-increment.
* Typographic quotes are an extension. I wrote KeenQuotes[0] to solve the quote curling problem and integrated it into KeenWrite[1].
* Document header. I fundamentally disagree with putting formatting instructions into plain text documents, in either AsciiDoc or Markdown. I wrote KeenWrite to completely separate the two. Documents are typeset using ConTeXt[2] and a theme[3].
* Admonitions. Pandoc (and KeenWrite) supports annotations in Markdown. I used them on page 5 of my Impacts Project[4] to insert the spectra. I'd say that annotations are more flexible than admonitions because admonitions are often canned (TIP, WARNING, etc.); whereas, annotations are user-defined.
* Sidebars and block titles imply presentation. These are also handled by annotations.
* Includes. You can use R Markdown to get includes. Or write an extension. Fair point that it isn't bundled, though.
* Custom CSS. Again, avoid mixing presentation and content. Specific presentation logic can be applied by annotating the content, rather than trying to format plain text as though it was HTML.
* Definition lists. Supported by Markdown, I use them for a glossary in a novel I'm writing.
* Tables. While perhaps not CommonMark, basic tables are widely supported by almost all Markdown implementations.
There's a fair amount of incorrect, biased, or outdated information on that page.
Here's an example page written in Markdown and made into a PDF:
That's a blockquote (story within a story) with nested annotations (the four simultaneous calls).
The bigger picture is this: Why have plain text format wars? Here's an architecture I developed for my text editor:
https://gitlab.com/DaveJarvis/KeenWrite/-/raw/main/docs/imag...
With that architecture, the source document format doesn't matter. Take any input document, transform it into a structured document format (such as XML), and then typeset it. Pandoc has a similar architecture.
[0]: https://whitemagicsoftware.com/keenquotes/
[2]: https://wiki.contextgarden.net/
For writing I really enjoy using WriteMonkey
These tools are a distraction to the process. I just posted another book I'm writing on github and was asked to share a link to it. Now that an audience is actually watching, I spent countless days building tools around it rather than writing the content. It's nice and all, but I better get back to writing the book or I'll have a beautiful pile of code that does nothing. It supports markdown though...