Literate Devops with Emacs Org-Mode (2014) | Hacker News Reader

Literate Devops with Emacs Org-Mode (2014)

howardism.org

213 pointstmstieff80 comments

Org-mode & Magit really are the secret superpowers of modern emacs. And notmuch (full-text search of one's email, at least one order of magnitude faster than Gmail's!). And elfeed. And gnus. And dired. And pretty much everything else in emacs …

"ag" and "ag.el" have been of great utility to me. Together they replace "find-grep," and exclude files in a relevant .gitignore from the search.

https://github.com/Wilfred/ag.el

Read find-grep exclusion patterns from a .findignore file: https://emacs.stackexchange.com/a/19038/6682

Seconded. ag is amazing on its own, but really shines in emacs since all the results have hyperlinks to their files/line numbers.

Org-mode has been nice the few times I've tried it, but without decent mobile support or multi-master syncing, it's hard to adopt it as my main productivity tool.

I've tried a few mobile org-mode clients and been universally disappointed. I'm increasingly convinced that the right path forward is to give Emacs an Android port (it's not that hard) and let the mobile org just be the mainline org, perhaps with a layer of android.view.View widgets (orchestrated from lisp) as an interface.

I've been pretty happy with Orgzly on Android, but that's probably because I don't want that much from it. I don't really like typing more than a sentence or two at a time on a phone, so I never expected to do any serious editing of Org mode files on a phone. What I do want, and feel Orgzly adequately provides, is the ability to read my Org files, mark things as DONE or put check marks in checklists, and capture one sentence notes that I later fill out and refile, once I'm sitting in front of a real keyboard.

Check out orgzly: http://www.orgzly.com/

As a user of Orgzly it's fine for basic notes and task management but you hit the limits of what it can do fairly early. It's a small partial Java implementation of the Org file format.

I think you're being slightly unfair. Of course it doesn't compare to full Emacs Org. Nothing does. But if you (like me) keep your life in Org and would forget to eat without it but sometimes have only your Android device, Orgzly has picked a highly efficient 1% subset. The dev is responsive and moves very fast too -- if you tried it more than a few months ago, you should give it another try.

I would not be where I am without Orgzly. Yes, I wish it was more complete. But given the very limited manpower behind it, it is excellent.

It's also better than any other similar app I've tried, even non-Org ones. So it's the best note-taking/todo/planner app for Android, and then the fact that it uses Org files behind the scenes is just a great bonus.

I don't think phones are good for note-taking. There's even evidence that they may harm cognitive functioning.[1] It seems like it would be more productive to carry around a small pad of paper and transfer the paper notes to Org Mode later.

[1] https://news.utexas.edu/2017/06/26/the-mere-presence-of-your...

For iOS (which I realize isn't answering the parent but just putting this out there), there's now beorg, which is pretty good.

https://itunes.apple.com/app/id1238649962

> without decent mobile support or multi-master syncing, it's hard to adopt it as my main productivity tool.

I find that for a true productivity tool, I'm fine relying on my desktop & laptop. I'm actually dialling back what I use my phone for, simply because I can't trust either of the major mobile OSes to respect me & my privacy. It would be awesome to have all of org available on a phone though; you're not wrong at all there.

I would kill for an emacs which makes sense on mobile, but I've not given enough thought to it to imagine what would make sense. Emacs without the key chords would be still powerful but also odd.

On Android, Termux [https://termux.com/] gives you access to Emacs; and it enables using the volume buttons a bit like modifier keys.

Termux is, hands down, the best thing to happen to Android. It's still pretty problematic and incomplete as a productive shell tool.

Android will kill applications pretty indiscriminately, so your Termux session persistence isn't good.

Terminal performance is poor, especially on scrollback, for some reason.

I've bumped into numerous SSH issues.

My general preference would be for a hybrid tablet/laptop system, running Linux, rather than Android (or iOS). We may eventually get there.

The other thing tablet formats are good for is reading electronic document formats, with an 8" or better tablet size. Though a large eInk device might be better there (again, ideally with shell capabilities).

Termux of course is not the same as using a regular terminal on desktop Linux, but it's pretty damn good.

I've had pretty good luck with SSH on Termux - it's great to be able to transfer things from my computer to my phone via rsync (I have some apps which do SAMBA, but they tend to be flaky; likewise physically plugging the phone in and transferring with adb isn't very convenient).

I don't disagree with anything you're saying except that Termux is, of and by itself, a sufficient replacement for a full desktop system.

I'd tried to make that happen over the course of 2.5 years, a full year of which was with Termux. And as mentioned: the best damned thing to happen to me on Android.

Termux is really nice, and for the 700 - 800 or so packages it has, there's a tremendous amount of utility. But you're still stuck with limitations.

1. The Termux-dies-randomly problem is, AFAICT, inherent to Android and its memory management model. Unless Termux can somehow indicate that it is not a terminable process, that's not going to go away.

2. The terminal-performance issue appears to be specific to Termux, as other full-screen console tools (ConnectBot SSH client) do not have this problem. Since it manifests on both local and remote connections w/ Termux (e.g., scrolling backwards within a manpage on a remote system), it seems to be the local terminal code on Termux itself. Of all my complaints, this seems the one most readily fixed. But it has not been. (I have reported it. Several other issues I've noted have been rapidly addressed.)

3. The SSH lag and hang issues I saw seemed to cover all Android SSH clients, and were either Android, its networking stack, or quite possibly my Very Shitty WiFi Network (since somewhat improved). But in all Not Particularly Heartening. Wireless sessions from a laptop didn't have the same lag issues.

4. Package selection. Unless you switch to a full Android/Debian system or something like it, you're limited in what's available. Commendable effort, and a Hard Problem. Just ... well, it leaves me short.

5. The Android storage model. Termux is Just Another Android app, and has Very Limited Access to storage. Which fucking sucks as there's a 128 GB MicroSD card on the device, but almost completely unusable. Rooting your Android might get around this, but if you're stuck with an unrooted (or unrootable) device, Thou Remainst Fucked.

6. Shell only. There's no X.org or other graphical interaction. (Other than Android native, which leaves you with that platform's bastardised and pitiful apps selection, on which I've written elsewhere.)

Mind: I really wanted to like Termux. I kind of wanted to like Android. I have a Samsung Tab A and Linksys Bluetooth folio case/keyboard. In terms of a portable form factor, it's really fairly close to my ideal. But in terms of capabilities, performance, behaviour, applications, surveillance, and vendor shitfuckingheadedness, from Samsung, Google, and Linksys, and absolute and complete clusterfuck. I want to run as far as I possibly can from any of them. I absolutely recommend AGAINST any of these vendors.

The simple truth is that I want and need something that they are unable, or far more likely, unwilling to provide. Neither my financial nor time budget for exploring alternatives is large, and the profound level of frustration I've had in this (and earlier) Android excursions has left some keep and I'm sure long-lasting scars.

I've tried and used many other systems, few of which strike me as desireable. I don't care for Microsoft's products (though the MS Surface tablet, again, strikes me as well-considered hardware). Apple's tablets and laptops are shiny, but ultimately annoying. I'm typing this on a retina iMac, which has a luscious display but an annoying OS. I've been partial to Thinkpads for nearly two decades, but find Lenovo's recent shenanigans massively off-putting. I'm also no longer generally satisfied with the traditional clamshell laptop design, yet don't see what might yet supplant it. Portrait reading for documents is a very compelling feature, but what I see are either tablets (with no or poorly-considered keyboard and mouse options), or hybrid devices with hinging mechanisms that leave me in a cold sweat. (Maybe they Really Do Just Work and Will Prove Rugged in Actual Use, but I Have My Doubts.)

In no way could my Android phone with Termux serve as substitute for my Linux desktop/laptop, but it is handy.

Presumably the only real solution is using desktop Linux on a device, rather than Android.

Is the GDP Pocket [ https://medium.com/@tomac/qpd-pocket-7-the-return-of-the-hac... ] included under your category of sweat-inducing devices? It looks handy, and even seems to have some sort of pointing/TrackPoint-like nub. I haven't used or even seen one in real life, but I've heard good things.

I'm not convinced there's anything that will replace an X-series ThinkPad. Non-phone devices that fit in our pocket and run Linux are rather appealling, and I'd love to have one, but I still don't imagine it could really oust my ThinkPad for most purposes. I need at least the X-series sized screen for most of what of I do.

I'm leaning much that way myself.

One major correction to the above: Linksys are an absolutely innocent party in the above, result of a mental failing on my part.

The keyboard, a complete piece of crap, and with abysmal customer and warranty non-support, was from Logitech, who deserve absolutely all opprobrium that can be directed their way.

(Linksys make my old router -- a (very) tried-and-trued WRT54G model. Since retired, after well over a decade of excellent service.)

I use dropbox for multi-master syncing. Seems to work ok. Emacs even has some file locking stuff that seems to help prevent me from clobbering changes I've made on another machine

You mean you're modifying files stored in Dropbox from your phone? What sort of editor setup do you use?

Sorry no - I mean between desktop machines.

There are a few different android apps for the phone (I think they are Dropbox based) though I don't use any.

Is there a good side-by-side video of someone using magit vs the command line? I keep reading the documentation for magit, and I can't really imagine it giving me a huge advantage over GitX plus good command line completion settings.

I don't have side by side comparison, but I'm adept at command line and I still use Magit even when I have command line or other IDE available.

Magit is my go to tool, I only use command line to quickly switch branches sometimes or for non-standard things.

Things I love about Magit:

- CHUNKING .. staging chunks or unstaging chunks is so easy (my favorite feature)

- General overview of information

- diffs, rebases, etc

- It's really fast when you learn the key chords

Other thing I use for Git:

- Command line (if I'm in command line and need to change a branch, might as well do it there)

- Tig (Log is fine in Magit too, just old habbits)

It’s mainly that things happen in a couple of key presses instead of a few typed words.

For example to stage things you can just tap ‘s’ while running your cursor down a list of changed lines/files. To commit, it’s just ‘c c’, and you’re in an editor, in a markdown-like mode, with hard wrapping lines at a good default.

Amending previous commits is also very easy, and there are some quality of life abstractions involving branches which I’ve found very useful.

But, ultimately, it’s all the same stuff. If you have a good system with Gitx at the moment (and not much of an interest in Emacs), I wouldn’t worry about it.

I'll second the chunking feature. Magit is very good at staging and unstaging files and PARTS of a file. I don't know any good way of doing this on the command line.

Is there some way to tell org exactly how I want it to spit out HTML?

I was going to do my blog entirely with org-mode+org-publish, but it spits out a mess of HTML and also a small bit of js for some reason, so I'm just writing html directly myself now.

I don't really find writing html myself all that annoying or anything, but it'd be nice if I could get org to generate really clean/simple HTML for me that I could then write a small script to parse and wrap in divs I've defined for styling.

I doubt there's anything actually wrong with the HTML that Org produces that would keep you from putting it in a blog, but you can certainly configure things to your liking. At the very least you can easily modify the HTML preamble and postamble, the javascript script tags and the CSS in style tags that Org's HTML export produces. All of those are just strings stored in variables that you can customize: org-html-preamble, org-html-postamble, org-html-scripts and org-html-style-default. There's more stuff you can configure by just reassigning some variable, see the file ox-html.el.

If tweaking those is not enough for you, you can write your own HTML export. You wouldn't have to parse Org syntax: you'd receive the Org markup already parsed into a nested list structure. If you don't use many Org features, writing an Org backend that supports only those probably isn't that much work. And you'd have ox-html.el to base your own backend on.

Are you kidding!? This is Emacs! You can tell it exactly how to do everything. Just advise/redefine the relevant functions. ;) The documentation of ox and publish does not cover everything, but when in doubt, just ask Emacs what code it runs when you do the thing you want to modify.

On my site (https://two-wrongs.com/) there is very little left of the original HTML. I'll see tomorrow if I can dig up the Lisp code I wrote for it.

You website design is awesome! Do you have the site source public? If allowed, I'd be interested in converting that to a theme for Hugo (https://gohugo.io), and possibly demoing that on how to use that with my package ox-hugo.

Custom css is supported for html export, from the docs at (https://orgmode.org/manual/CSS-support.html) --

The HTML export back-end includes a compact default style in each exported HTML file. To override the default style with another style, use these keywords in the Org file. They will replace the global defaults the HTML exporter uses.

#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="style1.css" /> #+HTML_HEAD_EXTRA: <link rel="alternate stylesheet" type="text/css" href="style2.css" />

To just turn off the default style, customize org-html-head-include-default-style variable, or use this option line in the Org file.

#+OPTIONS: html-style:nil

I've been looking for a similar solution - albeit not for HTML - but haven't found anything convincing so far. My train of thought was: It'd be great to have a a canonical XML format for org files to export to, since after that, you could essentially write XSLT transforms for almost any target format (even back to org-files to go full circle). In my case, that e.g. would be odt-files with branding for the organization I'm taking minutes for.

After a bit of googling, I also found an email[1] on the mailing list asking for more or less exactly that, but things don't seem to have gotten far.

I've been thinking about adapting the existing HTML export for such an XML format, but I really haven't had the time for it. :-/

[1]: https://lists.gnu.org/archive/html/emacs-orgmode/2009-08/msg...

Pandoc supports org-mode, so you should be able to use it to convert to DocBook, which you can then use with XLST (pandoc also supports HTML directly, by the way).

If I remember correctly, I had a look at the pandoc option, but got to the conclusion that this way, I would throw away useful 'syntactical' information (for example, I use todo-flags, tags and timestamps to track open tasks).

If you use org-mode without the need to extract such specific information (and are only interested in generating a document), pandoc might of course be a good way to go.

Edit: ...or maybe there even is a hack to identify and extract things like timestamps anyway. I didn't look into it that much.

I doubt XSLT is any easier to write than a new Org export backend, but that's probably a matter of personal preference.

Tooting my own horn, if you love living in Org mode, and are willing to try out Hugo (https://gohugo.io), I have an Org exporter package ox-hugo that exports to Markdown + required front-matter for Hugo. The package is even more magic if you choose to use a single Org file to export to multiple posts/pages.

Few examples:

1. ox-hugo documentation site from one file[1], source[2]

2. My blog[3]

3. ox-hugo test site (300+ test pages and counting)[4]

[1]: https://ox-hugo.scripter.co/ [2]: https://github.com/kaushalmodi/ox-hugo/raw/master/doc/ox-hug... [3]: https://scripter.co/ [4]: https://ox-hugo.scripter.co/test/

I've been blogging with org-publish for a while, but I switched to a Ruby script instead [1]. I'd not recommend using org-publish for this, because it's slow, and it's hard to add new posts. It's nice for static projects (say like a bunch of PDFs or some documentation maybe), but it does not lend itself to a more dynamic thing like a blog (dynamic in the sense that you want to add new pages rather frequently). Applying a custom HTML template is difficult, it means you need to either implement many org-export functions or use HTML_HEAD settings and a stylesheet to tame org's output. I say this as a happy everyday user of Org mode, both for word processing and any sort of notes, and my entire agenda and todo-lists and stuff.

The JS is for interacting with tables, if I'm not wrong.

[1] https://www.gkayaalp.com/blog/20180221_siteremake.html

I am assuming your site remake is still in progress? The fonts on your web site show up in super-huge size at the moment. You can see the font size on other sites like this one for comparison: [1].

[1]: https://i.imgur.com/um3yTdU.png

Thanks a lot! I haven't been able to reproduce that in any computers and browsers I have access to. Is that a Retina screen maybe?

I don't use Mac, so not sure what resolution qualifies as "Retina", but my Dell monitor resolution is 2560 x 1440.

There are, but I never explored them. I use Pelican, and although it doesn't support org files, it's easy to write a plugin to add your own formats. Behind the scene, I think I use pandoc to convert from org to rst, which Pelican does support.

Yes, this came up on /r/emacs recently, see[1] (you can control if you want the HTML preamble and/or postamble).

[1]: https://www.reddit.com/r/emacs/comments/809xwx/how_to_get_pl...

For the most part, just styling what it does isn't that tough. Similarly, you can tell it to spit out raw html for some fun. http://taeric.github.io/cube-permutations-1.html shows some silliness I did recently that includes styling up portions of the page.

Do you want to do something much more complicated/simple than this?

Not exactly what you're asking for but I got great results with org-page[0]. It might be worth taking a look.

[0]: https://github.com/kelvinh/org-page

Not the ideal solution, but you can always use pandoc to convert org to html, or even markdown and configure the html generation from there.

Both html and markdown miss various org-specific features, like code fragments, drawers (imagine that one could want them to be able top open and close on click), task states, etc.

A general-purpose markup language like XML would be an ideal intermediate export format; XSLT can basically emit anything.

For drawers you could use https://developer.mozilla.org/en-US/docs/Web/HTML/Element/de... but yeah in general I agree

I know you can define CSS for how you want your export to look.

Weird... it's as if text was really important in computing and having a tool at the center that excelled at processing text put you in more fluid control of your computing environment.

I know it's being worked on but every time I see posts about how great magit is, it pains be because my experience is that it continues to be ungodly slow on MacOS. Other MacOS/Emacs folks, is my experience considered atypical at this point?

That won't be much of a problem anymore once Emacs 26.1 is released. Meanwhile you can use the latest release candidate - those usually are very solid. For more information see https://magit.vc/manual/magit/MacOS-Performance.html.

M-x profiler-start

M-x magit

M-x profiler-report

You can accomplish a surprising amount of optimisation after seeing what it is that takes time.

How slow is ungodly slow to you? It's pretty slow on Firefox's code base but that repo is huge. For most things it's not lightning fast but definitely not ungodly slow.

Oh, yeah, gecko is not a fair comparison at all for any git utility. Even on my beefy linux box, m-c takes an age to load under magit. Even doing CLI-based interactive rebases takes a while. Yay 300k+ commits.

Is there a tool for git that only downloads the last say 1000 or 2000 commits or is a full history tree fundamental to git working correctly?

It's the --depth option for git clone/fetch: https://www.git-scm.com/docs/git-clone#git-clone---depthltde...

Are you using stock magit? I think a few of the GitHub integration plugins for example have performance issues, so it may not be Magit's fault but some of the other extension you're using.

Yes it is atypical. Have you tried looking through the github issues on the subject? There seem to be a number of possibilities, depending on your environment.

Ive tried so many times to use gnus over the years, without luck. Any suggestions? Or links?

I'd recommend mu4e [https://www.djcbsoftware.nl/code/mu/mu4e.html] for email. I used it for a number of years; switched to gnus for a year, and then ended up back on mu4e.

IMHO Notmuch is easier and much simpler for most email workflows.

I agree. I have used gnus for email in the past, but when I discovered notmuch I moved to it immediately. Gnus can work if you want to treat your email like a News Group. Not a bad approach for email lists for example, where you are mostly a reader and occasionally a contributor, and you rarely need to find an old message that you've already read. But it's not quite the right model for the way most people deal with the rest of their email which is one-to-one or small group threads.

Notmuch is like the best of the newsgroup and email models. It has a great threaded view, tagging and filtering are easy, but it still makes everything very easy to find when you need it.

Also in combination with w3m it does a really good job at rendering HTML emails in plain text. Even those with tables and other layout tend to be rendered very accurately within the constraints of a fixed font plain text view.

Org-mode and babel sound like good ideas, but in practice I've found it very finicky to use.

My main problem with it is that, unlike with executing commands in the shell, with org you don't get any progress, and emacs seems to hang while running the command.

For example, running apt update tends to freeze emacs without any progress reporting for a few minutes.

I'm hoping that I'm doing something wrong and that someone can chime in with a suggestion, because I really like org-mode, and I really like the idea of doing devops using it.

It's not 'freezing' Emacs so to speak, source block execution is synchronous.

Check out: https://github.com/astahlman/ob-async if you want to change that.

Thanks. I've tried that in the past. The problem is still that it doesn't give any feedback on progress.

I still struggle to see how people actually use this with devops on a day to day basis.

I think that some type of integration with tmux would work well - have a running tmux session, and then just queue commands from emacs to that session, instead of running directly through emacs.

If you want to see progress, it should be easy to add a process filter somewhere (I bet there already is a hook for that, but not that hard to do even if there isn't one) and make it display sth. on a secondary buffer. That's the thing with Emacs: if you don't have it, it's very easy to add it.

This is a cool use of emacs, but containers have made the process of iterative development/system creation so much easier. If you have the deployment story down, it’s even easier. And the configs are much more portable.

The process demonstrated can be adapted to a wide variety of environments, containers or not.

It's mostly about making a literate program, executable on the spot, sort of like a Jupyter notebook, only for shell commands (or nearly anything, frankly).

FWIW, Jupyter notebooks also have a kernel to support shell commands.

So "Literate DevOps" is writing comments in your shell scripts and running the commands line by line?

I agree it's a good idea. I do similar things when writing PowerShell scripts (can run them line by line using PowerShell ISE).

EDIT: If you're going to downvote me, it'd be better to explain why.

I didn't downvote but I'll try to explain.

Is the fact that you got the idea exactly backwards: is not about putting comments on your scripts. Is about writing an article, intended to be read by humans, that is also executable.

Imagine it this way: Suppose you are reading a blog post on how to install, configure vagrant and install drupal on it. The article is pretty clear an well written and you enjoy it. And you also know that is correct because every single one of the instructions shown was executed while you were reading the article and every output was given back to you.

That is literate programming, is closer to a jupyter notebook than to 'comments on scripts'.

I’ve started to use Jupyter notebooks, with devop recipes as different sections, for this.

You can keep the notebook in the repo if you do two (or three) things:

1. The scripts read passwords from a key store instead of the notebook source. I wrote a Python package for doing this easily in Python cells; for bash, I just use subcommands.

2. A git pre-commit hook clears the notebook output cells.

3. (Optional) A git pre-commit or pre-push hook searches for text that looks like private credentials.

> "That is literate programming, is closer to a jupyter notebook than to 'comments on scripts'."

Surely that's a matter of perspective? Imagine you have a text document containing one command and one description of that command. Is this a literate programming document or not?

Furthermore, does the mode change depending on which tool is used to open the source file? For example, if I open a Bash script in a text editor then it's a document, and if I open it in Bash then it's a script, but if I could open it in a tool that allows me to both read and execute the script at my leisure does it then become a literate programming document?

Aside from this, to use your Jupyter notebook example, is it necessary for the results of execution to be captured in-line in order to follow the literate programming methodology?

Answering questions one doesn't like is better than voting them down.

much of whats covered in org mode can be handled by vim's folding option. if you write ansible playbooks with coworkers who handle automation as a stream of consciousness out of a single file, folding can save you some sanity.

http://vim.wikia.com/wiki/Folding

Do vim's folds give you a full literate programming environment with support for executing code, saving results in vars, WEB syntax, etc? From what I'm familiar with, vim's folding is a lot closer to outline-mode[1], which org relies on for folding. The folding is a very small part of org-mode's functionality.

[1] https://www.gnu.org/software/emacs/manual/html_node/emacs/Ou...

I think you can run python inline in vim. But I obviously vim doesn't really have something as full-featured as Org mode. I mean, there's vimwiki - and you could probably get a majority of the rest of the features through messing around with plugins. But at that point, you may as well just switch to spacemacs.

The ability to fold is defintively important to org-mode, but that's not really the point. Folding is a core-feature of Emacs (hs-minor-mode: https://www.gnu.org/software/emacs/manual/html_node/emacs/Hi...) and not inherent to org-mode.

org-mode is a major-mode and covers a little more than that^^

folding is a convenience feature of org-mode, and really not what it's all about.

I'm sure most of the individual features are available in vim and other editors, but it's tying everything together in a neat package that's the main draw of org-mode.

Since we're off-topic anyway: have you tried Spacemacs? It's all the ingrained vim keybindings on top of a... more powerful system.

Personally, if I was going to give Vim a try again I'd use Oni:

https://github.com/onivim/oni

Reply on news.ycombinator.com