Introduction to sed | Hacker News Reader

Introduction to sed

catonmat.net

125 pointspkrumins31 comments

Flagged for excessive self-promotion. Take out "World's best" or make the article provide strong evidence that it's the world's best beyond the fact that its author thinks it is, and I'll unflag it.

If this didn't have "World's best" in the blog title and someone had added that to the HN title, a moderator would be deleting those words. The fact that you've inserted the same peacock words in your own blog title as well as your HN submission does not make this any better.

(I don't like being so blunt with a highly regarded veteran HN user, but abuse is abuse whoever does it. Sorry, Peteris.)

[EDITED to add: The HN guidelines say not to comment saying "I flagged this". I think this is an unusual case. I appreciate that others may disagree.]

Thanks for letting me know. I just removed the "World's best." I just thought I'd name it "World's best" cause I was feeling fantastic today. :)

Unflagged. Thanks.

But the word "World's Best" is still there in the URL, even in the Page Title and the article title. boo !!

The introduction was alright, but I'm not sure that the hold buffer needs to be introduced straight away. Using the hold buffer can get complicated pretty quickly, and it isn't something that you'll frequently have to do. The example the author gives of getting lines matching a regex with the previous line is more easily done using grep -B1, which doesn't require the mental overhead of sed scripting.

Perhaps more useful for an introduction would be retaining portions of the input back-references (\1 to \9) and `&'. Also useful is is the `g' flag for the `s' command, for global replacement, and the `i' flag (a GNU extension) for case insensitivity. Another note is that you can actually use any character to delimit `s' commands you want, which is useful if you have /'s in your pattern or replacement strings. The last main part of sed which I use regularly is the -r option, for extended regular expressions (basically the same as grep), which lets you use regular expression tokens like + and | without escaping. On that note, the author misses this point in his introduction, and the example of "sed -n '/a+b+/p'" should actually be either "sed -rn '/a+b+/p'" or "sed -n '/a\+b\+/p'".

For a serious introduction to all the extra functionality in sed, there's a great reference at the top google result for sed[1]. In addition, GNU sed adds functionality that you might be useful at some point (e.g. s///e), all of which are described in the GNU sed user's manual[2].

[1] http://www.grymoire.com/Unix/Sed.html [2] http://www.gnu.org/software/sed/manual/index.html

Thank you, this is a really great comment! I'll improve my introduction based on your feedback to make it better.

Are there any HN reviews of this guys eBooks? I think he has two on SED and I'm a little leery of self promotion although the preview was pretty good.

Hi there!

I have two e-books, one on Awk, and the other on Sed. Here are the threads of the e-books on HN:

Awk: http://news.ycombinator.com/item?id=2674284

Sed: http://news.ycombinator.com/item?id=2995711

And here are reviews that people have sent me:

Awk e-book:

http://www.johndcook.com/blog/2011/08/31/awk-one-liners/

https://plus.google.com/102155999275405024037/posts/W5ANwQgV...

http://benoithamelin.tumblr.com/post/10559496046/review-awk-...

http://twitter.com/#!/barryezl/status/121334247893114882

Sed e-book:

http://www.johndcook.com/blog/2011/09/27/sed-one-liners/

I've got both one-liners books (Sed and Awk). Peteris takes a very practical approach in the book. I almost think of it like a field guide. Think of the times you've google searched for something like 'number lines in file sed'. These books not only provide solutions, but give detailed explanations of what's happening. Pick any given example and there's a good chance you'll gain some insight about how sed/awk work. They're very well written, and very accessible.

Having said that, if you're a long beard who has long been acquainted with these tools, you could maybe skip them.

As the author also explains in the introduction, the compilation is based on the earlier compilation by Eric Pement, available as .txt files on his website.

http://www.pement.org/sed/sed1line.txt

http://www.pement.org/awk/awk1line.txt

To quote "The Awful Truth about sed" section of the Grymoire, "It is not your fault you don't understand sed."

I do, however, understand Perl, so that's what I use. E.g.: `echo 'fubar' | perl -lpe 's/fu/foo/'`

It might be a few milliseconds slower-- for example, commenting every line of my 352-line .zshrc (via s/^/#/) takes 0.007s total with Perl (so does `s/^/#/ unless /^\s*#/`) and 0.005s with sed. Commenting out every line of /usr/share/dict/american-english (98569 lines) takes 1.124s with Perl and 0.813s with sed.

Since I already know how to do more complicated things with Perl (like conditionals, named backreferences, etc.) it doesn't seem worth it to take the time to learn how to use sed effectively. I can wait the extra second since I'm not on any kind of deadline or under any efficiency constraints.

I am not trying to say that Perl is better than sed or any other text processing tool. I also don't mean to imply that speed is sed's only advantage-- it's just one example. I think that for someone who already knows some Perl, learning another similar tool doesn't make sense. I'm sure there are exceptions. This is only my humble, personal opinion.

For people who do need/want to learn sed, the article did a pretty good job of showing you how to get a lot done without a whole lot of reading.

On the speed question, I actually find Perl considerably faster than sed in a lot of use-cases, if there's enough processing to dominate the slightly higher startup costs of Perl.

For example, at one point I had reason to take a gigantic single-line textfile, and break it into lines based on a specific 3-letter pattern that didn't occur anywhere else:

   s/ABC/A\nC/g

In whatever sed comes with Debian, this took about 10 minutes, CPU-bound, for a 2-gigabyte file. With Perl: 1.5 minutes, IO-bound. Not too sure why. Maybe sed runs everything through the regex engine, while Perl special-cases constant strings? Perhaps Perl has better buffer management for processing gigabytes of text? I haven't done any real testing.

The copyright section of sed(1) suggests that the Debian package sed (I'm running wheezy) is Gnu sed 4.2.1[1]. Plan9 sed is also available in the 9base package. I'm running perl 5.12.4 (also from Debian testing). I don't see any significant difference in speed with vanilla 5.14.1 and the commands I mentioned earlier.

[1] http://www.gnu.org/software/sed/

Turn on locale UTF-8 for GNU software before benchmarking it to compare with Plan9 software.

GNU stuff gets a speed improvement from assuming single byte characters.

I know how to do exactly one thing in sed, and that's sed 's/blah/blahprime/' somefile.

If sed has capabilities other than that, I don't particularly care... but that's one thing that I need to do frequently which is more painful in awk or python.

I would guess that that's all a significant number of unix-like OS users do with sed, and in those circumstances, it does make more sense to use sed than a more extensive language. For simple substitutions like that, I'd just use an alias: alias ped='perl -lpe' to save the extra keystrokes.

however, perl has -i, which not all seds do, so those 0.002 seconds you lost to perl will be more than gained by being able to type

      perl -pi -e 's/foo/bar/' somefile

instead of

      sed 's/foo/bar/' somefile > somefile.tmp && mv somefile.tmp somefile

Sed has -i, too! Well, at least GNU sed.

yes, that's why i said "not all seds".

missed it. sorry!

because GNU is Not Unix

The utility 'sponge' is useful for doing this in the general case.

Part of the 'moreutils' package, for the curious. The description for sponge on the project homepage reads "soak up standard input and write to a file."

Rather than -i alone, I always use -i.bak (or whatever you prefer instead of '.bak'). That way, you get (1) in-place editing and (2) a backup. The files you work on are backed up to "file.bak", so that if you make some simple regex mistake with huge negative consequences, you have a ready fix. (And if you do things perfectly, it's easy to run "rm *bak" after you confirm that.)

Apparently the Debian kernel team agrees. I just noticed this while building a kernel package:

test -n "$k" || perl -pli~ -e 's/\$\{shlibs:Depends\}\,?//g' debian/control

I dont know if this off topic but is there an utlity that can parse C files and print all the function names or local variable names for example.

I know it can be done through sed,but a perfect regex for parsing functions and nothing else can be complex and I am wondering if someone has already done it.

perhaps https://en.wikipedia.org/wiki/Ctags

In particular, `ctags -x` generates human-readable tabular output (instead of the normal output designed for parsing by text editors) and annotates each symbol to indicate whether it's a variable, function, macro, etc.

If you want to get a list of all function names from file "foo.c", you could do something like:

    ctags -x foo.c | awk '$2=="function"{print $1}'

Look at what the perl script at http://www.gson.org/egypt/ does -- if you are using gcc, you can compile with certain options to generate info from the source code into an intermediate file and then parse this intermediate file with a script to get the info you need. A bit complex, but useful if you are trying to read through code you are maintaining. A compiler is better than a regex for this, I think, for C.

Also ctags/etags like another comment mentioned as also the cscope utility.

I checked out Egypt.It is pretty cool but unfortunately it doesnt seem to have support for Objective C.

I'll just leave this here...

    s/&amp;/&/g

Sorry! :D

Reply on news.ycombinator.com