GNU ed ate my homework | Hacker News Reader

GNU ed ate my homework

edoverflow.com

113 pointsatan265 comments

If this "save a backup file" functionality was built into the OS, instead of having to be redone by every application, this probably would never have happened – it is much harder to do the wrong thing when you are just calling an OS API than when you have to implement it all yourself. And if the OS API is implemented wrong, you can just change the OS API, instead of having to change every application.

I like how Vim uses swap files to warn you if you are editing a file you already have opened (or have autosaved data from a previous crashed execution) – but it only works for Vim, if you open the file in another editor you don't get the warning. If that functionality was in the OS instead, it could easily work cross-application.

While not quite as good, there is NILFS: https://en.wikipedia.org/wiki/NILFS

I wasn’t aware of it. Fascinating

Unfortunately it's deprecated, I have used it for around an year for my home as a test, the idea is good, the implementation not much:

- the cleaner daemon is damn slow, so on write-intensive (as a modern WebVM profile dir, improperly named browser for legacy reasons profile dir) you'll easily end up in a filesystem full, and recovery is not that easy;

- navigating through the write history is manual, a manual mount of a snapshot that might disappear while you actually go through a bunch of them, so you need to mark them to stop garbage collection, mount them, diff/see the change of something you are interested in etc.

No shiny meld-like UI is there, there were some project in the past but so long abandoned that I've just found references to them, not even their code. Nowadays the sole Linux log-based fs is Samsung f2fs, witch is far less developed than nilfs2, to a point you can't even go through the write history. The sole good log-based fs I know is DragonFlyBSD Hammer, witch is "a better zfs" but being a DragonFly-only show with very little devs it's not really useful if you are outside the HPC world.

Personally these days I have zfs with auto snapshot service well backed by ZnapZend to sync data to a homeserver backup, it's not write()-triggered but time triggered so going through history is less useful, but as a raw protection against accidental disasters it's generally enough, zfs diff is not much detailed (just tell about changed files) but might be of help, snapshots are auto-mounted typically under the volume root in a .zfs/snapshots/$snapName dir so reasonabily easy to traverse with meld, a file manager etc to directly grab some files.

Unfortunately since the '80s no one except for SUN (zfs) have done much on generic data storage/file storage side...

You don't really need that much functionality, a copy-on-write filesystem with fine-grained access to historical data. Once you have that, you can easily access the old version of a file and that is probably 90% of the functionality that you need without many changes to the software (regular automatic saves and ability to access old versions from the program itself).

While there are a number of copy-on-write filesystems around (ZFS, btrfs), most of them don't have fine-grained access to historical data, but only through snapshots. One exception is them HAMMER fs from DragonFlyBSD [1], though I don't if this is still true (HAMMER2 seems to be quite a different beast)

[1] https://www.dragonflybsd.org/docs/docs/howtos/howtorecoverda...

Sadly that’s not what copy on write means. When you have file a and you copy it to file b. COW means the copy b will only be truly materialized once you write something to it/edit it. Before that, file b will be aliased to file a.

If you have file A and a make a snapshot of it, let's call it A@2022-04-01-12:00, you have now two different files pointing at the same blocks.

If you write to one of the blocks of A you will not overwrite any data of A@2022-04-01-12:00 because the new data will be saved to a different block. I would call that copy on write.

I admit that the description is not perfect because it doesn't always apply (like in the example you make), but I couldn't come up with a more appropriate description and for the discussion at hand I thought it was quite apt.

Also, for what is worth, Wikipedia seems to agree with me [1]

[1] https://en.wikipedia.org/wiki/Copy-on-write#In_computer_stor...

VMS!

The main feature of VMS which seems relevant here is file version numbers–it is a great idea, and it is sad more mainstream operating systems haven't copied it (if only Dave Cutler had taken that VMS idea to Windows NT... if only RMS' plan, announced in the GNU Manifesto, to include a file versioning filesystem in the GNU system had panned out...)–but I don't think it fully addresses the issue of editor autosave/etc:

(1) File version numbers are great to make a backup of the old file on saving, not so relevant to creating an autosave of a modified editor buffer / open document / etc. You generally want autosave to not create a new version of the file you are editing, because that would make your changes visible to other applications/users, and you haven't decided yet whether you want to "commit" your in-progress changes to the actual file.

(2) Doesn't really address the case of warning you that you (or somebody else) has the file open in another editor, or that you have recovered changes from a prior editing session. I know VMS has some file locking features that might help with the "warning" part, but other platforms do too. As far as I know, VMS still leaves it up to each app to build a sensible UI around those capabilities, rather than providing something built in to the OS.

Files-11, specifically.

https://en.wikipedia.org/wiki/Files-11

We already have tools for this. Important work should be in version control and application authors who want more out of open have sqlite. People need open/read/write to work with the file directly and I don't want shell tools scattering backup files everywhere.

Leaving aside the correct advice to avoid such hard coded backup files, I wish people would recommend at least considering "sudo -e" instead of "sudo $EDITOR" when editing something important with root permissions.

`sudoedit` should automatically invoke `sudo -e $EDITOR`.

"sudoedit" is equivalent to "sudo -e".

Why is this?

sudo $EDITOR launches the editor itself as root, sudo -e launches the editor as a regular user on a temporary copy of the original file, and copies the contents back over to the original file when you are done. The less done as root, the less opportunity to mess things up.

Couldn't a malicious actor edit the temporary file without the user who invoked sudo -e noticing?

Perhaps, but you will still overwrite it when you exit the editor and let sudo copy the contents back (that being said, there might be a race condition between closing the editor and sudo noticing that the process has terminated). Not sure if this would work, but a possible workaround could be to use /dev/shm (https://www.kernel.org/doc/gorman/html/understand/understand...).

The temporary file should be created in such a way that other users cannot modify it. If that does not happen, if other users can modify it, I would regard that as a bug.

Malicious processes running as the same user could potentially modify the file, but if you have malicious processes running as a user with sudo privileges you have probably already lost.

Yes, I think she can. I think this is better than running the whole editor as root.

The vi editor is, or was just ed. All ed commands are vi commands.

Likewise Emacs: originally it was just a bunch of TECO macros plus the “^R mode” (visual) addition to TECO. I frequently used to casually type raw TECO into a mini buffer to perform certain edits (meta-altmode — my hand just made the gesture automatically when I thought about it).

I suspect Bill Joy knew about this when he added visual mode to ed about a decade later.

PS if you haven’t checked out the MIT TECO language you should — it will blow your mind.

Technically I think vi was an interface for ex, Joy's ex-tended version of ed.

In 1999, after installing Linux on a mainframe for the first time, I needed to edit a few files. Being in the Linux console, ISPF wasn't an option. But the copy of Linux we were working with (Marist College's release) didn't yet have vi available. So I had to go with ed. That, as they used to say, was "a real trip".

I was once stuck on a box that didn’t even have ed, and I needed to edit a file. This is maybe 1995 and it’s an HP/UX machine that was used as nothing more than an X console, but its fstab or automount config was messed up so it couldn’t boot anything but single user mode.

I managed to cobble together an “editor” with just the shell that as I recall read each line of the existing file and echoed it to stdout. I could press return to keep it as is or retype the entire line with whatever changes I needed. The output went to a new file. When I was done I copied the new file into place.

Another time I needed to transfer an executable to a server I could access only over serial console. On the receiving end, I think uudecode was all I had to safely get the binary over serial console.

Fun times.

Damn. That's the equivalent of having to walk to school uphill both ways during a snow storm. It makes me feel really grateful for the hundreds of thousands of man-hours that have been spent on giving us good and reliable tools.

Clever with the shells "read line", I was in a similar situation, and used dd as an "editor" :-)

I use cat file|gzip -9c|base64 -w $COLUMNS To copy and paste files, I've even had to use rdiff/zstd with a dictionary.

Fun indeed.

Just another reason I don't mess with unix utils outside of the most trivial scripts one can imagine.

I really don't care if the software I use is "high quality" or not, anywhere near as much as how many edge cases they've covered.

If your app is small and has very few dependencies, as a heuristic I generally expect it to somehow be trouble one way or another.

The only time I've had major trouble with VS Code is when I installed a buggy extension that helps add licenses to things, without noticing that it modifies every file in the entire workspace, even if they have a license already, and sometimes gets the syntax wrong.

It wasn't a bit popular extension, so I probably should have known better.

> If your app is small and has very few dependencies, as a heuristic I generally expect it to somehow be trouble one way or another.

Wow, that’s extraordinary! I have precisely the opposite view, especially the “few dependencies.” The more dependencies the higher the probability of failure in my experience.

Oh you're probably totally right. The smaller apps are less likely in and of themselves to fail.

The problems I expect to see are not really the app itself failing, its the inability to deal with some other failure, or some changed requirements that make some ugliness, or maybe it doesn't have hardware acceleration and the performance is a little worse.

Maybe it doesn't detect malformed packets right. Maybe there's a UI issue that makes it easy to delete files, as in all the rm and dd problems. Maybe it only supports bitmap and wav files and now your app needs 5x the storage when you start adding media.

Very small things rarely outright fail. It's the rest of the whole system from users to hardware that does wrong by those very small things, and in their perfect bug free innocent perfection they're all "OK I guess the user knows what he's doing, one deleted home folder, coming right up!"

Is it really pronounced "ee-dee"? Here I've been pronouncing it all the time like it was some zany protagonist in a 90's cartoon show!

Yes, Brian Kernighan said it's pronounced "ee-dee": https://www.youtube.com/watch?v=NTfOnGZUZDk

...and that never made any sense to me. Because you don't pronounce "sed" as "ess-ee-dee". So if you pronounce "sed" as "said", you must surely pronounce "ed" as "ed".

The reason is that it's a descendant of QED — https://en.wikipedia.org/wiki/QED_(text_editor)

there is also that talking horse which even predates the editor

This is completely off topic, so please downvote it.

It's a children's story, personifying ed, along with several other UNIX commands.

https://github.com/peterburk/peterburk.github.io/blob/master...

It's not finished, but it did help a couple of young girls from the homestay family where I used to live to fall asleep.

This is even more off topic:

I couldn't resist the urge to actually upvote you when you said downvote it.

So please downvote this one for doing exactly opposite of what you said and being even more off topic.

It's not possible for a parent commenter to downvote their children.

https://github.com/minimaxir/hacker-news-undocumented#downvo...

"You cannot downvote comments which are direct replies to your own comment, and you cannot downvote 24 hours after the original comment was made."

So therefore I upvoted you. Other members of this community may downvote you, though, (and probably me too), because we're really far off topic now.

Found first half good but cannot say the same about second half. (Though a kid's opinion may differ.) Wonder how a Unix-y version of Aesop fables would look like.

the story is clearly about the editor and it is therefore more on topic than say a certain talking horse

The talking horse is especially offtopic for those of us who are case-sensitive.

The code fragment that the author complains about has an even more serious fault: it doesn’t check for the existence of an `ed.hup` file. If you happen to have one, because you’re unaware of `ed`’s behavior and you like to give weird names to your files, `ed` will wipe it out when trying to help you.

Vim does similar irresponsible things with its backup files¹ in its default configuration. It not only destroys existing files without checking, but prevents many auto-build systems from working correctly. All these behaviors are reckless, especially as there are obvious alternatives that don’t make assumptions about the users' files.

[1] https://lee-phillips.org/badvim/

ed was a lifesaver if your Unix system didn't fully boot because unlike ex/vi, it was statically compiled and had no dependencies on other filesystems. Plus it was cool to learn just as a bit of history and in a very small way follow in the footsteps of some amazing people.

I wasn’t alive then but I’m very nostalgic about computing and the people pushing it forward in the 70s, Bell Labs, AT&T, Xerox.

It’s probably less amazing to the people who were there but I look on those times with a sense of great wonder.

Same here. I'm listening to & reading a lot of computer history lately. From Knuth, Kernighan, Thompson, Ritchie, Hopper to more 'modern' pioneers like James Gosling, Chris Lattner, David Patterson, Rich Hickey and Rob Pike.

I'm just amazed by how much of the foundation of the software world was laid by these brilliant minds.

npm install imposter-syndrome

We need a series of books which would chart out all of the software, innovations and the people involved during that era. I would definitely buy and read it.

The history of that era is really interesting.

Two I can recommend are: Where the Wizards Stay Up Late, and The Idea Factory, which cover the history of the Internet and Bell Labs respectively.

There’s also Dealers of Lightning which is about Xerox but I haven’t read it yet.

that's the ONE AND ONLY time i've ever used ed.

I live upgraded a remote server from a much older redhat release to gentoo. This was a system that didn't have a nice console interface, so no fake local terminal. I messed up the mount paths, so nothing that required shared libraries worked. I was able to fix it with ed!

ed is also the only command line terminal on plan9, which allows you to edit files in case Rio didnt start

The right solution would be to make a log file at startup, and append a copy of the original text and then every last bit of input, exactly as it is typed, interspersed with some metadata. Then, if anything happens, you just exit on the spot, and absolutely nothing is lost.

The type of stuff edited doesn't matter: could be audio, video, 3D design, vector drawing.

All editors should operate this way. It is disgraceful none do.

> It is disgraceful none do.

Sublime Text does. When you start it, all unsaved modifications and files from your last session will conveniently be there, just as you left them. Over years of usage, I've never had Sublime lose any data, even after sudden loss of power.

ed is the way that it is in part because many users in the late 70s had a really slow modem and a printing terminal, and memory was minimal, so just ? was seen as a good error message.

Early 70s, and it wasn't with a modem. I think the typical Teletype baud rates were 45.45 baud, 110 baud, and 300 baud. Having to wait half a second for your Teletype to type out a 15-character error message that tells you what you already know is the opposite of user-friendly. (Later, yes, people were using 300-baud modems, but that wasn't how Unix was originally developed.)

May be. However, my memory is of 2 factors that made line editors like ed necessary.

1. The keyboard mechanics and having paper as the output medium made typing very slow. That's probably why all the commands were short (ls, mv, rm, etc). Coding was not the stream of consciousness you might experience today. You would have to hold a load of stuff in your head (and ignore interruptions) as you slowly typed it out.

2. The paper only ever advances upwards. If you make a mistake there is no up arrow. Your easiest option may be to use the delete command and type the whole line again. Hopefully someone will give you one of those paper cheat sheet summarizing all the editor commands soon. I wonder how many potential computer geniuses dropped out of CS at this point.

Imagine you are at the command line and you take a few minutes to quickly type:

  $ cp inupt_KJuusiwbHSgh.txt output_HswbXVfgawYuq.txt
  cp: cannot stat 'inupt_KJuusiwbHSgh.txt': No such file or directory

Rats! You didn't notice that you mistyped "input". What now? You can't do copy and paste from paper! Well you could use the line editor:

  $ !!:s/up/pu
  cp input_KJuusiwbHSgh.txt output_HswbXVfgawYuq.txt

Job done.

When I started programming as part of my job in 1967, there were no screen editors. I used a teletype terminal (ASR33) with a paper tape to load and save my work.

Line editors like ed were the way to go. You’d keep the state of the program in your head and type out sections of it once in awhile to confirm your conception. Maybe print the whole thing once an hour.

I recently found an example shell session in the documentation for mysqlimport that uses ed: https://dev.mysql.com/doc/refman/8.0/en/mysqlimport.html

Who doesn't reflexively save when pausing to think/etc.? To someone who does save early and often, this seems to be a non-issue --- if the original file wasn't writable for whatever reason, you'd also know pretty quickly.

If you want to do that in ed or ex, you have to exit insert mode, save, go back into insert mode, and then possibly exit insert mode again, find the extra line break you inserted to save, and delete that line break. So I could be wrong but I think probably no ed or ex users save by reflex when pausing to think. Also, saving a file could take a significant amount of time in the 01970s.

The 03660s. ;)

A lot of people. Google docs has save millions of essays from being permanently lost because one forgot to save.

Ed as a Clojure IDE

https://lambdaisland.com/episodes/ultimate-dev-setup

The proposed fix is wrong.

> This is a lesson on why not to use static backup filenames.

Yes, that would address this bug, and this bug only.

What if there's another bug? Or the file system is full? Or `ed` gets kill -9'd? Or the PC loses power?

Assuming it's possible to predict every single failure mode is foolish. It's better to make a system safe by default. In the case of `ed`, simply periodically store the text buffer in a file and restore from it on startup.

I am glad vim has the option to place the swap files where you want them, and make them unique (by setting 'directory' to end with "//").

For vim and ed and every other console program, I would run them inside screen(1) to avoid losing work due to network troubles, and network disconnects is usually where I get my SIGHUP.

> I am glad vim has the option to place the swap files where you want them, and make them unique (by setting 'directory' to end with "//").

i am glad as well. not every program supports --exclude/--except.

i used to keep swapfiles in a single global directory, but i've instead started to try to stash my swapfiles "near" the code:

    alias vi='vim --cmd "$(x="$(git rev-parse --show-toplevel 2>/dev/null)"; [ "x$x" != x/ ] && mkdir -p "$x/.git/_vim" 2>/dev/null && echo "set dir=$x/.git/_vim//")"'

this seems to have some advantages: for example, i can do a find / -name '*.swp' to figure out what i was working on when most of the files are readme.md

(i should note, real ed doesn't create ed.hup files; that's a weird gnuism)

> For vim and ed and every other console program, I would run them inside screen(1) to avoid losing work due to network troubles, and network disconnects is usually where I get my SIGHUP.

i don't like screen very much, even though i do continue to use it (owing more to its ubiquity and utility); it's full of all kinds of questionable ui decisions, and so many features i use so infrequently i need to constantly be re-reading the manual. i think if you have learned it and liked it okay, but all i ever want is terminal mobility, and i get a much nicer experience out of two tools:

- mosh: https://mosh.org/

- reptyr: https://github.com/nelhage/reptyr

using mosh means i don't get SIGHUP, and if i lose my local display, i can open another mosh, reptr my old session (it's still there!) and keep on hacking.

That's brilliant but I have a question: why are you doing your homework in ed?

Reply on news.ycombinator.com