Using sed to make indexes for books (1997) | Hacker News Reader

Using sed to make indexes for books (1997)

pement.org

69 pointsTMWNN15 comments

Previously:

Using sed to make indexes for books (1997) - https://news.ycombinator.com/item?id=16080505 - Jan 2018 (17 comments)

I used awk and eqn and t/roff to make a university phone book in the late 80s and early 90s. I was set the task of mimicing the hand-set phototypsetter version from a printout listing. I needed EQN to make giant ellipsis across groups of names who shared a phone, and had to tune macros to use to get n column output to work with the gutter needs of the new bitstream bromide making machine.

I used sed to compile YAML. Bit of AWK in there too but mostly sed.

More articles from the 90s please.

Two thought:

1. I feel the use of AWK, SED, GREP and Bash (to name a few) is general waning. I rarely see it outside of HN posts. I wish I were better at using it. Sometimes for a special need I’ll ask a LLM to write me a script in bash just to keep this spirit alive.

2. On the other hand, I’ve written a book and added an index by using Microsoft Word, and it was basically pretty easy to add. I don’t know why I should want to go back to a world of intricate command line tools to do a job that should just be easy enough?

> I feel the use of AWK, SED, GREP and Bash (to name a few) is general waning. I rarely see it outside of HN posts. I wish I were better at using

You can find plenty of examples with a quick github search[0]. The query in the url is

  path:bin dotfiles language:Shell

A shell script is my first option for any automation. sed is for quick transformations and awk is for more procedural ones.

[0] https://github.com/search?q=path%3Abin+dotfiles+language%3AS...

When this post was written (1997 updated in 2001), awk wasn't very portable because you still had the awk,nawk,gawk flavors. With sed if you stuck with BRE and avoided ERE you were going to be more portable.

But this email is a there because it is a bit of a feat of strength, thus why it was sent to the seders list. I wouldn't have used sed for all of this at the time, and I am wondering what they were using for publishing in anyway. In 1997/2001 I would have just taken text inputs and used LaTeX, and even indexes with troff was easier.

QuarkXPress and Pagemaker were very popular at that time, with QuarkXPress having almost all the commercial market in the 90s, so they must have been using something very specific for this religious market.

But also note that VS Code/ides with Vim keybindings is extremely popular among modern software developers, so while they may not call sed directly they still use it with `:`

That is the same reason we used it in the 80-90's, you had sed, vi, ed, etc... that were at least usable without much pain across all the drift of the UNIX wars.

But I am really curious on why this email even existed at all, it doesn't make sense outside of someone playing with UNIX/Linux as a side quest, unless they were *troff/TeX based?

> awk wasn't very portable because you still had the awk,nawk,gawk flavors

Debian and Ubuntu both ship with mawk and Alpine has the busybox flavor of awk (nawk like). MacOS still ships with a default nawk. I don't know the scene has changed much, still many flavors out there.

The issues from the 80s/90s were mostly fixed by POSIX in the 2000's. IIRC all major Unix-like systems ship with awk implementations that are POSIX-compliant by default today.

With a few warts of course, but not like back then.

Perl was in 1997 and is today a better awk+sed.

Better is to use the typesetting language

Even on MacOS, save this as `doc.tr`

     .\" Index Macro
     .de IX
     .tm \\$1^\\$2^\\n%
     ..
     .\" Content
     .sp 2
     This is the first page.
     We are talking about the history of Unix.
     .IX "Operating Systems" "Unix"
     .bp
     This is the second page.
     Here we discuss the C language.
     .IX "Languages" "C"
     .bp
     This is the third page.
     We go back to talking about Unix.
     .IX "Operating Systems" "Unix"

The index will be output on stderr with:

     % groff doc.tr > output.ps 2> raw.idx

Which will output the following that you could use bash/sed/awk/perl/... to make into another troff file.

     % sort -t '^' -k2,2 -k1,1 raw.idx                    
     Languages^C^2
     Operating Systems^Unix^1
     Operating Systems^Unix^3

(Note: `^` as a delimiter is an old Unix thing, just going really retro)

Even inserting the `.IX ...` lines on a string match in the troff file is fairly easy. But somehow they were parsing the typeset output, not sure what kind. But no need to try and track page numbers when typesetting languages do that for you.

TL;DR What I was saying is if you are using a typesetting language, use the tools in the typesetting language when possible. If you can do it in troff you can do it in LaTeX with makeindex etc...

They should really be considered obsolete beyond very basic tasks. They have been obsoleted by short Python scripts.

Hopefully someone will make a linux distribution with Python as the shell instead of bash. For Python fans. /s

Python is obsolete too. It's only there for those not wanting to get performance and typiy

I use those tools nearly every day. Instead of heavy LSPs and IDEs using the RAM that could be otherwise used, I grep and ctags through my codebase for instant navigation. For deterministic changes if I haven’t already done it in vim, I awk and sed, though usually I do buffer-wide changes in vim for such things. When troubleshooting CLI outputs, especially for binutils and other tools I didn’t write, I still use combinations of awk, sed, and grep.

Not everything is thrown together code churn. Some tools have earned their keep and don’t need to be in the constant spotlight because they do their job silently and effectively for people such as myself on a daily basis.

Reply on news.ycombinator.com