Use Vim Inside a Unix Pipe Like Sed or AWK | Hacker News Reader

Use Vim Inside a Unix Pipe Like Sed or AWK

blog.robertelder.org

92 pointssigna1146 comments

This post hints toward vim's history: the ed family of editors.

http://blog.sanctum.geek.nz/actually-using-ed/ http://blog.sanctum.geek.nz/using-more-of-ex/

A big use case for them was using them as scriptable editors in pipelines, and a lot of vim's commands are inherited from them. The diff(1) utility actually has an option to emit ed script, allowing files to be patched in ed pipelines: http://www.gnu.org/software/diffutils/manual/html_node/ed-Sc...

It's common for people today to think of ancient Greeks as being very unsophisticated and primitive, but then when you start to read what ancient philosophers wrote you think "Wow, these guys really had things figured out, maybe better than we do now." I kind of feel the same way comparing modern GUI applications to ancient editors like ed.

Which is why I have been learning and using emacs daily for a few years now, and it's amazing how much more powerful it is than I ever seem to fully understand. I'll think "now I have it all like I want it", and then a few months later I learn about a way to do X.

Especially as a sysadmin, I spend so much time in a ssh cli, that part of my reasoning was "I want to be able to do all my normal tasks without the gui." Gmail, news, and rss in gnus, irc in erc, org-mode (loving export to latex for reports), eww for browsing, and who knows whats next. Elixir and go modes are improving too, and my general productivity due to staying mostly inside a single ecosystem has really improved.

Another reason I have done this though, is I feel like it's less about gui vs text, and much more about FOSS vs proprietary. In a few years when everyone has an iBrain with Apple (NSA) inside, I intend to have a MEmacs brain that I have control over.

I think RMS will be vindicated in history as a man far ahead of his time.

Which philosophers? The only one I've read in any depth is Aristotle, and while he is clearly very smart, I agreed with essentially none of it.

https://en.wikipedia.org/wiki/Lorem_ipsum

I do not understand your reply; Lorem ipsum text is tangentially related to Cicero, but he's Roman, not Greek.

I find learning this lesson often enough made me realize that it's difficult to remember that people in the past were every bit as intelligent as people are now. They simply had fewer, older tools.

There's another related lesson there that progress, like evolution, is progress in a certain direction. No one said that direction is whatever you call "good" at the moment.

> people in the past were every bit as intelligent as people are now.

... and some evidence supports they were smarter...

http://www.huffingtonpost.com/2013/05/22/people-getting-dumb...

Good point.This argument that domestication lowers intelligence (among other powerful abilities) and that we are unquestionably domesticating ourselves is pretty damn solid. The only argument against it seems to be "... but my ego!".

Obviously there is some interplay with the fact that we develop new mental models and thinking tools to augment intelligence.

Also immersion as children in highly abstract ways of thinking further augments/multiplies raw intelligence (most convincing explanation to the Flynn effect imo).

I've lost the link, but there was an excellent article I read related to the amazing Otzi discovery (https://en.wikipedia.org/wiki/%C3%96tzi) describing how adults of that era (modern humans, primitive societies) would likely have been terrifying to us now in just how much they outclassed us in raw strength, intelligence and stamina. We would be relying a lot on the benefits of childhood nutrition and education to feel superior. This isn't completely convincing, there are a lot of factors in play, but those levels of brutal competition and danger would have a profound effect, especially epigenetically.

I haven't used ed, but I'm inclined to agree. People still make fun of me for living in the terminal.

Very true. I read Plato's Timmeus for a music theory class once and noticed that along the way of explaining his version of music theory, he also--from first principles only--deduced the existence of fundamental particles. He didn't call then protons, neutrons, and electrons, obviously. Be he got the basic idea correct: that all the things that exist are fundamentally built from the same things.

My mind was constantly being blown in that class.

But then you remember that Ed was designed to be used on what essentially was a typewriter, because dialup was so slow, that you couldn't send an entire screen worth of text at once. So you would basically be remembering the text in your head, while inserting and regexing lines.

And the great error handling when it doesn't know what you just told it to do. ?

It's not that they had dialup, or that they couldn't send an entire screen at once (though I'm sure these factors later helped keep ed around). The devices they were using were much more like a typewriter than you seem to think, they literally printed the text out on paper! The programmer didn't need to remember the text in their head, they had the paper right there to look at!

I get a tiny ping of irritation every time I use an application to do some mundane, repeatable, well-defined task and it doesn't have a CLI.

Sort of. While ed was scriptable, it was by sending the commands to stdin, which doesn't lend itself to being composed in a pipeline, where you'd more intuitively expect the text it's modifying to be on stdin. We can do it without problem now, because shells have process substitution, but ed predates that shell feature by a couple of decades. Instead, to use ed in pipelines, we ended up with a modified ed to work on streams: `sed`, the Stream ED.

Vim is actually a relatively bad filter (some reasons are mentioned in the caveats section of the article). I explicitly designed my vis editor[1] in such a way that it can be used as an interactive filter. The following works as expected, use :wq to write to stdout:

    printf "foo\nbar\nbaz\n"  | vis - | sort

dvtm uses this mechanism to implement its copy mode.

I also recently integrated sam's structural regular expression based command language into the editor. This might be useful for people who want the power of stream editors but with instant visual feedback.

However keep in mind that this feature is relatively new thus likely still contains some bugs ...

[1] https://github.com/martanne/vis

Thanks for your efforts with the Vis editor...

You can do this with any other editor using "vipe" from "moreutils" package. It just opens $EDITOR with stdin input (put to a tempfile) and then reads the tempfile and dumps it in stdout.

    printf "foo\nbar\nbaz\n"  | vipe - | sort  # invokes $EDITOR

Yes vipe is indeed useful for editors which do not support it by themselves. Personally I find a solution solely based on pipes without temporary files more elegant.

I should probably let dvtm fall back to vipe if it is installed. Also I will have to check whether there is an easy fix to make vipe+vis work.

Hi, I'm the author. While I've got your attention, I would be interested in hearing feedback about how I can improve the reading experience (text colour/fonts/sizes/wording etc.) of my blog posts.

I used to have blinking text for the email signups at the bottom, but I got rid of it since it didn't have a huge effect (I think it only doubled the conversion rate). I have learned that some people really hate blinking text.

Hey Robert! I've had your virtual memory article open in a tab since it made the HN homepage recently - but I had to close it because every day it would cause a massive memory leak in Safari for some really odd reason. I'd repeatedly kill the process hosting the webpage (it'll respawn 3 times then surrender) but every time I visited that tab by accident it would happen again.

Just an FYI :)

Yes that's not surprising, the code for that implementation is quite scary and in order to make the page not lag terribly I realized at the end that I would effectively have to re-invent some of what react.js does (because it would have taken too long to completely re-write). I've considered re-writing an open source version with react that people could fork and incorporate into course material, but all that stuff takes time.

I can think of a few places where there would probably be circular object references, so that's probably where the memory leaks come from.

Thanks for the feedback :)

The color, layout, and font sizes are quite fine the way they are. The containers for your paragraphs expand to the full width of the screen, no matter the width, which is awesome!

Everything in the linked article is easy to read on my crummy laptop screen. (Its screen makes seeing too-low-contrast color changes [0] rather difficult. The fact that all of your site is legible is a strong indicator that your contrast and color scheme choices are perfectly adequate ones.)

In short, don't change anything about the layout, unless you are really motivated to do so. You've made good choices!

[0] Such as -2 or -3 and below comments on HN.

Actually, pretty good, though I'd recommend letting body font being solid black or perhaps #330 (and no lighter).

I can't view elements from mobile, but the results look good. Layout is comfortable, good margins, well-differentiated bits, and all without being distracting.

I've got a general set of tips most of which you can likely ignore. Check that you're using em/rem rather than px or pt for font sizes: https://www.reddit.com/r/dredmorbius/comments/29eqrk/web_des...

Yikes, we can shorten that a bit. :-)

    function vimify() {
      (vim - -esbnN -c $@ -c 'w!/dev/fd/3|q!' >/dev/null) 3>&1
    }

    $ echo '<h1>Change Me</h1>' | vimify 'norm vitxiOK I Will'

(Don't forget '-c' if you have more than one transform.)

And further shorten that to

echo '<h1>Change Me</h1>' | vimify 'norm vitcOK I Will'

And further shorten that to

echo '<h1>Change Me</h1>' | vimify 'norm citOK I Will'

For robustness sake, you may change $@ with "$*".

Why is this article talking about idempotence? I don't see why that matters in the slightest for a unix pipe. Idempotence only matters if you're going to take the output of the pipe and run it back through the pipe again.

There's a bit later on about idempotence and newlines, but I don't see what that has to do with idempotence. Vim will append a trailing newline if the file doesn't have one, sure, but if I pass the output back through Vim again, it won't append another trailing newline, it'll just preserve the existing one, so if the vim command you run is idempotent then the whole thing will still be idempotent.

I'm guessing here that the author is simply confused about what idempotence means. Because of the newline bit, I'm going to guess that what the author means by "idempotence" is "any text that the vim filter isn't supposed to modify will be left exactly as found in the input", but that's not what idempotence is, and I'd also say that that property is pretty much a given for any kind of text transformation used in a pipe, because if it doesn't hold then why are you even using that tool?

That bugged me too, although it was such a fun article I didn't want to complain. :-)

What he really means is that his first snippet is the identity function. Like x*1=x or x+0=x. Whatever it gets, it passes along unmodified. I think it's actually kind of nice that he started with that. Sometimes it is not easy to figure out how to do, for instance with XLST.....

Yeah, an identity function is probably what I should be calling it instead.

I suppose you're correct about the terminating newline case, in that the newline is only added once. I'm not sure what you'd call it, but I'm talking about it as if it were a kind of idempotent 'base case'. For example:

If you do

    echo -en "" | ( (vim - -esbnN -c 'w!/dev/stderr|q!' >/dev/null) 2>&1) | xxd

versus

    echo -en "" | ( (vim - -esbnN -c 'w!/dev/stderr|q!' >/dev/null) 2>&1) | ( (vim - -esbnN -c 'w!/dev/stderr|q!' >/dev/null) 2>&1) | xxd

These will the same output, so it is idempotent. However, if you build upon that and do

    echo -en "" | ( (vim - -esbnN -c '%!xxd' -c 'w!/dev/stderr|q!' >/dev/null) 2>&1) | xxd

versus

    echo -en "" | ( (vim - -esbnN -c '%!xxd' -c 'w!/dev/stderr|q!' >/dev/null) 2>&1) | ( (vim - -esbnN -c '%!xxd' -c 'w!/dev/stderr|q!' >/dev/null) 2>&1) | xxd

The result is not the same, and therefore not idempotent if you plan to add extra -c commands.

Why type vitxi (visual inner tag delete insert), when cit (change inner tag) will do?

Indeed, why use visual anything in a pipe :D?

That's a good point. For some reason, I never use 'change' in vim, and I was just following muscle memory.

I have to say the only time I ever pipe directly to vim in a real world scenario is when I want to see a diff in color:

    diff file1 file2 | vim -

Gives you the diff but colorized with Vim. Handy little trick.

Why not run vimdiff directly?

Or vimcat: https://github.com/ofavre/vimcat Or pygments: pygmentize -l diff

Or just:

    vim -d file1 file2.

This brings up a side by side comparison of the documents. This is different than the pure diff.

Sure, for that there is also colordiff.

Yeah, there are many other external tools that can accomplish this. Piping directly to Vim just allows us install one less.

Why did I read this.

This is far from practical to use on a day to day basis. It'd make my eyes bleed if some coworker used this anywhere.

Why not just pickup a scripting language like perk or Python. Use awk or sed. They're available everywhere.

Would non-ASCII data and escape codes have the effect of vim-injection[1]?

[1]: Think SQL injection but with vim commands.

How does this compare to piping through Elisp?

This isn't supported at all in emacs by default - or, if it is, I never figured out how.

You can try my pmacs script to get something sort-of-like: https://github.com/tom-seddon/bin/blob/master/pmacs.py (not productized, and some self-assembly may be required - should work out of the box on Linux, though, and probably OS X too)

Works pretty well, though only at the end of a pipeline. (It could probably be extended to work mid-pipeline though; you can get emacsclient to block until the file's buffer is closed in emacs - this is what the "-n" switch inhibits - giving the script a chance to wait, load the (presumably modified) file and print it to stdout.)

The title is missing the word "inside" from the original title, i.e. "Use Vim _Inside_ A Unix Pipe Like Sed Or AWK". Made me wonder for a bit...

We put it back.

Reply on news.ycombinator.com