What I find weird is that there's no analogous normal mode operator command for dropping into a command line without the ! prefix. It's easy enough to write your own (good excuse to learn about "operator pending mode"), but I often find myself scratching my head at what made it into the builtin commands and what did not.
Is the work of programmers less valuable than, say, the work of Google Docs users (where there is an undo operation)?
Mac has Time Machine.
Personally I’d be interested in a shell having undo capability, but not a file system.
I'm not sure how these sentences are connected. Are you implying that allowing undo would make those problems significantly worse? I'm not sure of that. If you have a CoW filesystem, which you probably want for other reasons, then having a continuous snapshot mode for recent activity would not need much overhead.
If you're saying there's too much activity to allow an undo, well, I assume the undo would be scoped to specific files or directories!
I agree it would be nice to have this in the filesystem; some filesystems support this (e.g. NILFS[1]), but none of the current "mainstream" ones do AFAIK. In the meanwhile, git works well enough.
Sure there could be some situations where it could be handy, like in some auomated scenarios, but most of the time it is not a big deal to write
foo data1 data2
bar data2In my case I could recreate the original file but it took 90 minutes to scrape the remote API to do it again...
Use a temporary file always. Sponge process may be interrupted, and you end up with a half-complete /etc/passwd in return.
rename is an atomic operation from any modern filesystem's perspective, you're not writing new data, you're simply changing the name of the existing file, it either succeeds or fails.
Keep in mind that if you're doing this, mv (the command line tool) as opposed to the `rename` system call, falls back to copying if the source and destination files are on different filesystems since you can not really mv a file across filesystems!
In order to have truly atomic writes you need to:
open a new file on the same filesystem as your destination file
write contents
call fsync
call rename
call sync (if you care about the file rename itself never being reverted).
This is some very naive golang code (from when I barely knew golang) for doing this which has been running in production since I wrote it without a single issue: https://github.com/AdamJacobMuller/atomicxt/blob/master/file...
Are those for networked like NTFS or just as security against crashes.
Logically on a single system there would be no effect assuming error free filesystem operation. Unless I'm missing something.
ext4 on Linux (since 2009) special-cases rename() when overwriting an existing file so that it works safely even without fsync() (https://lwn.net/Articles/322823/), but that is not guaranteed by all other implementations and filesystems.
The sync() at the end is indeed not needed for the atomicity, it just allows you to know that after its completion the rename will not "roll back" anymore on a crash. IIRC you can also use fsync() on the parent directory to achieve this, avoiding sync() / syncfs().
This is interesting.
The linked git entry (https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.g...) from the LWN article says "Notice: this object is not reachable from any branch."
Did this never get merged because I definitely saw this issue in production well after 2009.
I guess it either got changed, or, a different patch applied but perhaps this https://github.com/torvalds/linux/blob/master/fs/ext4/namei.... does it?
And yes, the code you highlighted is exactly this special-case in its current form. The mount option "noauto_da_alloc" can be used to disable these software-not-calling-fsync safety features.
The final sync is just there for durability, not atomicity, like you say.
Correct.
The rename can succeed while the write of the file you just renamed gets rolled back.
1. Write to a temporary file 2. Do the equivalent of mv tmpfile originalfile
so it will either succeed or do nothing
No, you couldn't, because a sponge is intentionally using more resources: It soaks up as much want as it can. And the program is intended to soak up all of the output. Otherwise it would be `cat`.
git init .
git add .
git commit -m "wip"
EDIT: Doesn't work if you have huuuuge files, obviously... but you should perhaps be using different tools for that anyway.
cmd < somefile | somefile.tmp && mv somefile.tmp somefile
Mind that this may still bite in interesting ways. But less frequently.
You can also invoke tempfile(1) which is strongly recommended in scripts.
> tempfile is deprecated; you should use mktemp(1) instead.