Hacker News Reader

> Are there editors that actually overwrite the original file, instead of creating a new file?

Are there ones that do the reverse? :o.

Seriously though, it's worth pointing out that what you describe is not obvious, and one can easily miss that there are two ways of writing a file to disk. Case in point, it took me 17 years of programming to discover that "writing new files under the same filename" is a thing, and that was only by accident, beacuse I started wondering why #'open in Common Lisp has both :overwrite and :supersede as different options for what to do if the file you want to write to exists...

> it's worth pointing out that what you describe is not obvious, and one can easily miss that there are two ways of writing a file to disk.

Yeah, there's a dichotomy in how files are handled as presented to the user and how it's actually done. For example, despite most UIs presenting files as being opened, they're almost always closed and only opened when the program needs to do something to them at any given instant.

> Case in point, it took me 17 years of programming to discover that "writing new files under the same filename" is a thing

It took me many years (though not 17) to discover the opposite, that all forms of overwriting without changing the inode was possible. For many years, I was under the impression that one could modify files to the same length or a longer one, but that the core syscall interface for files (the calls that one is typically introduced to when learning about working with files) didn't allow for the shrinking of files. I made the wrong justifying assumption that this API design was to help against disk fragmentation. I also thought file-clobbering meant replacing files. I later learned that truncate(2) exists and allows for the arbitrary shrinking of files, but even now, I'm not sure that it's possible to shrink files arbitrarily on all systems with a unix-like file syscall interface. For example, Plan9 doesn't seem to have a truncate syscall.[1]

It seemed to me that the only way file editors could modify a file to be shorter was to write a new one and replace the old one with it. I suppose they could have also clobbered files, but it didn't seem to me that it held any advantage over writing elsewhere and renaming, at least not while I wrongfully assumed that clobbering caused the inode to change. If anything, resorting to clobbering increases the risk of data-loss if there's any error while writing.

Anyway, my point is that what you call non-obvious was what I intuitively thought was the only way file editors could work portably, and that this was due to my late introduction to the perhaps-non-portable truncate(2).

[1] http://aiju.de/plan_9/plan9-syscalls

Come to think of it, I started programming on Windows, and my first introduction to the filesystem operations was iostream in C++. Being a kid then, I haven't learned about syscalls or inodes until a lot, lot later. I always imagined the filesystem as a more sophisticated version of an array of pairs <startAddress, length>. Thus I assumed shortening a file is trivial (just decrement the length), but making a file longer may require copying it over to somewhere else, if there isn't $newLength bytes available after the $startAddress. I believed that defragmentation meant removing gaps caused by shortening or deleting files.

(N.b. this is how simple memory allocators work.)

So in the end, for a long while, I hadn't had an issue that would make me double-check those assumptions.

The traditional way to truncate an existing file is using O_TRUNC in the open call. From your link it looks like Plan 9 has that flag.

> In addition, there are [...] values that can be ORed with the omode: OTRUNC says to truncate the file to zero length before opening it

That's what I meant by clobbering. The thing is that you can't use that flag to say "shorten it by half" to keep the first half, but you can with truncate(2).

My point was that if you can't shorten it by an arbitrary amount to avoid the redundancy of writing the same beginning of the file, then there's really no advantage to it over writing a new file to replace the old one. In fact, I thought clobbering (i.e. O_TRUNC) was nearly equivalent, only with the additional risk of data loss if there's an error while writing.

Creating a new file also replaces all file metadata (creation time, permissions, ACLs, links, ...) while O_TRUNC only erases the contents.

I can reproduce this in Emacs 28 and non 4.9.2 I have no idea how nano handles saving files but emacs seems to handle it safely as described and still hit the dangerous case.

from the depths of basic-save-buffer-2

   ;; Write temp name, then rename it.
   ;; This requires write access to the containing dir,
   ;; which is why we don't try it if we don't have that access.

Thanks, indeed with nano I can reproduce this.

One way to check how the editor behaves is to run

    ls -li run.sh

before and after editing the script. The first number in the output is the inode of the file; if it stays the same then the file was overwritten.

it changes under both emacs and vim but not under nano. Neither vim nor emacs is overwriting it in place but both show the bad behavior on my end.

There are situations in which even vim will write directly to the file. When you do not have write permissions on the directory, vim will not be able to create a new file and will therefore "reuse" the existing file.

Edit: Here is the list in the vim source code when it will not use rename: https://github.com/vim/vim/blob/95f0b6e5a5e5861da34cc064c601...

> E.g. using vim as editor, the modified file is actually written as a new file under the same filename.

I don't know much about the internal workings of vim, but I do know two things:

1) I use vim exclusively (on linux anyway).

2) I have been bitten by this exact problem.

I guess I can understand why bash works this way.. it means that it can handle arbitrarily long scripts without having to read the entire script into memory at once. But purely from an everyday usability perspective, this is a really bad design decision. Anyone know if there's a way to force it to read the whole script into memory on startup?

When writing a long script you should bundle everything into functions and then call one that kicks everything off at the end. This prevents the problem mentioned but also is helpful for keeping it organized.

I'm surprised that helps. Instead of the seeking behavior demonstrated in the `strace`, does it switch to another way of keeping state after a backward jump?

It would also have to buffer the whole file even though it's painstakingly executing it in a way that doesn't require a buffer.

I would imagine it's the same thing. It's going by top-level statement. The backward jump is just to keep track of which top-level statement it has read so far. It's not about being positioned on what it's currently running.

> painstakingly executing it in a way that doesn't require a buffer.

I don't understand why people keep saying that. There's a buffer of 64 bytes. You mean in a way that doesn't require slurping the file. It avoids slurping precisely by buffering.

The parser copies the body of functions and aliases (and heredocs, duh) into memory and interprets it from there. This is necessary because otherwise a function that is called frequently would translate into a lot of system calls for file I/O. Also bash has to work with streams that are not seekable, like piped command output, and so it allows declarations that can be parsed and then later referred to in one pass.

I tried running it in Vim (I have version 8.1.1401), with a regular bash script, and it didn't run the dangerous command. However, when I made a hard link to the same file, then it did run the dangerous command. My guess is that Vim defaults to writing a new file with the same name, but, if there is a hard-link pointing at the same file, then it has to actually modify the file in place. That is probably what bit you in the past (although I'm sure this behaviour is configurable somewhere).

If you want to force bash to read the whole script into memory on startup, you can put the whole script inside a function, and then, on the last line of the file, call the function you just defined.

> I guess I can understand why bash works this way.. it means that it can handle arbitrarily long scripts without having to read the entire script into memory at once.

The normal way of reading a file (using a buffer) solves this. The only reason for doing this dance instead would be that it's considered part of the interface. I was going to look for it in the POSIX standard, but it looks like that would cost me $894.00, and that's $7 more than my entire verifying-comments-on-the-internet budget for this fortnight.

> Are there editors that actually overwrite the original file, instead of creating a new file?

Yes, there are. This bit me at one point, since I created a Make target depending on directories rather than individual files in the directories (for performance). A directory gets updated when a new file is created, so it worked for all developers except one - who had an editor that saved files in place. I think that developer was using Sublime, but I might remember wrong.

I know `nano` does this.

iirc Sublime Text overwrites if you set

  "atomic_save": false

Thank you! I code with Sublime on files mounted with sshfs, and could not figure out why %autoreload (automatically reload imported modules in iPython) did not work. Apparently %autoreload watches directory updates, which are only triggered with atomic saves when using sshfs with cache.

i dont see the behavior you describe after i edit file in vim the inode does not change (in linux)

  → ls  -i /tmp/foo2.txt 
  4063325 /tmp/foo2.txt
  → vim !$
  vim /tmp/foo2.txt #edit file
  → ls  -i /tmp/foo2.txt 
  4063325 /tmp/foo2.txt

Thanks. Very curious. Indeed if the file is in /tmp, the inode doesn't change. But if it's in my home directory, it does change (for me). Both file systems are ext4, so there's something else at play.

in /tmp you dont have write permissions (so cannot create a new file, and falls back to overwrite), in home directory you have.

It depends on vim's 'writebackup' option. By default, it's on, but I guess you have it off. You can check in vim with ":set writebackup?" If it returns "writebackup", it's on; if it returns "nowritebackup", it's off.

EDIT: Then again, like the other comment mentioned, it seems it also depends on whether the file is in /tmp/. This seems to be because it matches the default pattern in the option "backupskip". For unix, the default is "/tmp/,$TMPDIR/,$TMP/,$TEMP/".

What does TFA mean in this context? I know what it usually meant on Slashdot, but it seems out of place here and other comments I see often on HN.

Same: The Fine Article - but vary the second word according to context.

The featured article

> Are there editors that actually overwrite the original file, instead of creating a new file?

Yeah, I've been bitten by this before. I use vim though! Does ":w" have different behaviour to ":wq"?

I'm pretty certain I've encountered this using fairly featureful IDEs too, as I primarily use the JetBrains suite for dev now.

This could all be compounded by me editing stuff on NFS mounts though...

> Does ":w" have different behaviour to ":wq"?

Or does ":w" have a different behaviour than ":up"(save file if changed) in this context? My guess would be that both just reuse the same functionality(why be inconsistent for no reason?).

But regardless it's good to know this edge case can occur - trying to find the cause otherwise and reproducing the problem would be a nightmare, even if the behaviour was consistent across editors.

EDIT: I tested using 'ls -li' to look at the file's inode, and while ":w", ":wq" as well as ":up" always result in a new file it seems that multiple changes can cause the inode to go back to a previously used number. So maybe this issue can occur with every editor if the file is saved multiple times.

> it seems that multiple changes can cause the inode to go >back to a previously used number. So maybe this issue can >occur with every editor if the file is saved multiple times.

I doubt that a inode reuse can make this happen, since the inode will not be reused while bash keeps the file open.

> the modified file is actually written as a new file under the same filename.

That can't be true, because it would break links.

Only hardlinks, which are rare for scripts in my experience, and not symbolic links.

I have used the same overwrite-by-rename technique in very long-running embedded processes because renaming a file is atomic on common local Linux filesystems.

In vim, it depends on the value of backupcopy. If set to "auto" (a default-default on linux, but maybe not for a specific distribution's default vimrc), the behavior is to check to see if the number of hardlinks to a file is greater than one, and then write the modified version in place, otherwise use the write and rename strategy.

See: https://github.com/vim/vim/blob/95f0b6e5a5e5861da34cc064c601...