Because of this Linux "feature" I experienced data loss - I zipped a file and deleted it, but the process which created it was still writing to it. But all the data that it wrote now ended up in the void since the file didn't exist anymore. And there was no error/warning that the process was writing to a non-linked file, so I discovered the data loss much later.
And Microsoft's been trying to manage that insanity since 1995.
So, you experienced data loss because you deleted a file you didn't mean to.
...and?
In the example I gave, the other process was continuously writing to this file while I was zipping it.
If the process creating the zip file runs faster, and finishes, and closes the file, your attempt to delete it would have worked anyway. Because the program that created it didn't have it open any more.
The zip creation and me deleting the file were in parallel with another process writing to the file.
And I didn't delete the zip file, I deleted the original file that I zipped.
This is why you should always check the contents of your archive before deleting the originals. There are myriad reasons why the archive may be incomplete or incorrect. You were hit by one of them that happens to not happen on Windows.
OK, I think that's where a lot of the confusion comes from.
When you originally said "I zipped a file and deleted it, but the process which created it was still writing to it.", I thought the "it"s referred to the zip file, not the original file, and "the process that created it" was the zip process.
Going by your actual meaning - are you saying you zipped a file on disk that was still being written to? That's... weird.
Given that reading from disk is faster than writing to it, and the written data may well have been in the disk cache anyway, it's likely that the zip program could have "caught up" with the program that was writing the data, tried another read and been told "this is the end of the file", and then finished creating the zip file. That would have created an incomplete zip.
But also, as others have pointed out, just because you delete a file and it gets removed from the directory, the actual file and its contents stay around until all open handles to it are gone. So if one process has the file open and is writing to it, and a second process has the file open and is reading from it, if you delete the file then both processes should still be able to access it through their existing file handles, even though it doesn't have a directory entry any more. So you shouldn't have experienced any data loss from that series of events.
(And that's why having the "it"s refer to the zip file is a logical interpretation of what you said, and why a few repliers - including me! - seem confused)
That's how Linux works. Some programs, like rsync, will give you a warning at the end, similar to "the file was modified during the rsync operation". Most others, like zip will give no warning at all.
More precisely, what data was lost, and how did it get lost?
Process P, the data producer, writes to file foo.
Process Z, the zip-process, reads file foo and produces foo.zip
Process rm, removes file foo.
I imagine it went a bit like this:
P --output foo &
Z < foo > foo.zip
rm foo
# time passes
# process P terminates
User 323 assumed that either rm would fail, letting them know that foo cannot be deleted, or that P would throw an error when the file was deleted.
(I don't think this is a reasonable expectation, and it's a failure on 323's hand of not learning the UNIX file model, but there's the situation).
I think I can see the loss:
1. Start zip. 2. In parallel, copy the intermediate zip file. 3. Delete the zip 4. Delete originals
The data was deleted in step 4 and is the loss. It notably would also had occurred had the zip terminated early or not zipped _all_ the files for some reason. The process was not resilient.
I think it's debatable if it's reasonable to continue allowing writing to an unlinked file.
There was discussion some time ago about Postgres data loss because of something similar, where the linux kernel returned successfully from fsync despite the movable media not being present anymore.
> How is it possible that PostgreSQL used fsync incorrectly for 20 years, and what we'll do about it
https://archive.fosdem.org/2019/schedule/event/postgresql_fs...
In this case, you can have your cake (delete the directory entry) and eat it too (get the data back through /proc).
When the last directory entry for a file is passed to the unlink() system call, the file is removed from the directory hierarchy.
However, the data on media will be deleted only after all open file descriptors are closed.
One exploitation of this feature that I have seen is the SQLite temporary tablespace, which is created and immediately unlinked, ensuring that it will not persist after the program terminates.
Yes, it does. Windows is POSIX/FIPS-151
In any case, I have never seen any way to coerce POSIX unlink() behavior on Windows.
'Broad software compatibility was initially achieved with support for several API "personalities", including Windows API, POSIX, and OS/2 APIs – the latter two were phased out starting with Windows XP.'
I can update glibc or other core system libraries on Linux without rebooting. Processes using the old library will show "(deleted)" in their /proc/self/maps files, but they will continue to use the old code. New processes will use the new library, and downtime was not required.
This is not possible on Windows because of mandatory locking on core system libraries. As a consequence, Windows must reboot for every Patch Tuesday.
There are only a few places that Linux has mandatory locking, and this is a good thing.
Here is a script that will show you all the deleted libraries that are still in use on Linux:
# cat stale_libraries.sh
#!/bin/sh
awk '$NF=="(deleted)" && $(NF-1) ~ /[.]so/ {
sub(/;.*$/,"",$(NF-1))
print $(NF-1), $NF}' /proc/*/maps | sort -u
# cat stale_programs.sh
#!/bin/sh
grep '[.]so.*deleted)' /proc/*/maps |
sed 's/[:][^/]*//' |
sort -u |
while read -r line
do pid=${line#/} pid=${pid#*/} pid=${pid%%/*}
xargs -0 printf '%s ' < "/proc/$pid/cmdline" | sed 's/[ ]$//'
printf :%s\\n "$line"
done
Linux doesn't need reboots only in theory. I have an Ubuntu server box and not a week passes in which I don't see a "system reboot is required" prompt when I SSH into it.
Windows does not allow this, which is why we have Patch Tuesday outages.
Also, Oracle KSplice was the first free tool to apply updates to a running Linux kernel without rebooting.
KSplice is free on Ubuntu. I think other free services have become available since.
There are mechanisms for updating the kernel in-place, and I believe canonical is one of the leaders in that domain, but if you're choosing not to use it and you don't want to reboot once/week, you can still keep your system level libraries up to date without rebooting.
The above poster didn't say you could update absolutely everything in linux without a reboot, just that the locking mechanisms in windows means you have to reboot to update things that don't require a reboot in linux.
My point was that in theory Linux doesn't need reboots and Windows does, but in practice my Ubuntu box needs a weekly reboot, while my Windows box just once a month.
What's true is that you can update everything BUT the kernel without a reboot.
Any understanding of theory that involved a belief that Linux never needed a reboot to update the kernel is a flaw in your understanding rather than a problem with the theory.
And in fact, the reason people have been working on being able to update the kernel without a reboot is finally put to rest that final mile. Personally I don't think it will ever come to full fruition, but the list of things that does require a reboot will become smaller and smaller over time.
https://cloud7.news/article/how-to-update-linux-kernel-witho...
> Rebootless Kernel updates are not a replacement for full kernel upgrades but it allows you to patch critical security vulnerabilities and bug fixes. With these methods, you can keep your servers safe and running without outage for years.
> Several Linux vendors offer rebootless kernel updates. Your solution mostly depends on the distribution you are running.
And finally, a bit more info about canonical's solution
https://cloud7.news/article/how-to-update-linux-kernel-witho...
Yet people keep saying that on HN, that unlike Windows, you don't need to reboot Linux after updating, and then maybe mention how they have months of uptime.
By default Ubuntu, the most popular distro, updates the kernel too. And frequently the kernel update is a security update.
It's possible your cloud provider has chosen to have its own package repository where they insist on doing this, but this is a decision by your cloud provider and not Canonical.
Linux has what's called Kernel Modules that can be unloaded and loaded at runtime without the need to update the kernel itself. I myself run ubuntu in vmware, and have done so for probably over 10 years, and your description of Ubuntu's default behavior is inaccurate.
---
But more than that, it's been explained to you, stop arguing.
The amount of harm "security" people have wrought to everyone else around them is astronomical.
I don't doubt your cloud provider made that decision, it's yet another example of security people forcing major inconveniences on everyone else in the name of protecting them. It might even be warranted here (it probably is), but I would almost put money on 85-90% of terrible situations in software dev having a root in a security person somewhere. That sounds outlandish, yet it completely matches my experience.
Linux assumes that long running processes just randomly exploding because they are living in a pre-hotfix ghost state is fine (and usually free of security concerns). Windows assumes users notice and get frustrated when long running apps start acting "weird" and don't know why and that there are too many security implications if you just leave unpatched apps running alongside patched ones.
It's another one of those things on the list that the NT Kernel has the raw design to do a lot of smart, deliberate things, but at the end of the day can't trust user space to play nice with them happening and is maybe a little too safety conscious of that.
(My understanding too is that Ubuntu is an interesting example here because Canonical now explicitly uses a more Microsoft-influenced security model in user-focused distributions for when to require reboots. Under the same general concerns: avoid user-noticeable "weirdness" and avoid unpatched CVEs remaining long running in user space. While it could just kill unpatched processes, that creates "user noticeable weirdness", and for better and worse a full "restart" is much less shocking to the average user than long running processes getting killed "weirdly".)
Someone with more knowledge please correct me if I got the above wrong.
That being said, the issue with Windows is that it prevents me from deleting a file if it's open in a text editor or any program, which should not be the default in my opinion. A directory gets locked simply because you have explorer open in that path, or a terminal cd'd into that path, which is annoying and counter productive most of the time.
The standard Linux compression programs, zip, gzip, tar, don't lock by default. I don't even think they have such an option.
File locking on Linux is advisory, it excludes other processes which try to lock the same region of the same file, but not operations other than locking. Mandatory locks were optional (required a mount option, see https://lwn.net/Articles/667210/ for details), and were removed in 5.15 (https://lwn.net/Articles/874493/).
Of course that's assuming the compression program used does attempt to apply file lock, which may not be the case according to another comment.