Comment by james412 | Hacker News Reader

Comment by james412 | Hacker News Reader

Parent Full thread

Horrendous advice on many levels

- reference counting is an implementation detail, it is not part of the spec

- that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython)

- even if it were part of the spec, the schedule for object destruction also is not part of the spec. Your file object was noticed as part of an unused graph at T+0, even mainline CPython can perpetually defer its destruction to T+infinity due to its handling of reference cycles

- the random system resource will eventually be garbage collected, but perhaps not if your code throws the wrong exception and/or one of those cute syntax sugar libraries takes a reference to its frame object you know nothing about. Or how about pretty much any debugging support library like Sentry? Best to have audited them all..

- the random system resource was garbage a few tens of opcodes ago, but you left it locked, and some future change caused your code to immediately attempt to reopen and relock it. surprise, deadlock in production

The comparison with 'dict.clear()' makes no sense whatsoever because explicit external resource control is precisely required because those external resources aren't automatically managed like normal in-memory objects. They have material external state beyond whether they are allocated or deallocated, and require explicit initialization and de-initialization in order to avoid leaving them in unspecified states. Python's memory allocator provides none of these behaviours as guarantees. It's not even guaranteed to always call __del__, this is most commonly observable during interpreter shutdown.

Perhaps a better way of looking at it is this: try/finally and with: exist for you to demand stronger guarantees from the runtime than it normally offers you. "Hey Mr. Python, I know you'll /probably/ clean up this object eventually, but I also know there are 100 edge cases where you won't bother. For this object, you are on the hook for every case, including the cases where I wrote buggy code"

Meanwhile, feel free to follow this advice, and good luck at 4am debugging your prod system that for some inexplicable reason recently began crashing due to running out of file descriptors..

> - reference counting is an implementation detail, it is not part of the spec

> - that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython)

Exactly. That was a major if not the major drive for context managers & properly closing files: the PFS and the language team were getting much more supportive of third-party implementations, and much more aware of issues in "normal" Python code e.g. though it didn't work out well in the end, ctypes also got merged into the stdlib in 2.5 with the express purpose of limiting the need for writing cpython extensions to bind to native libraries.

> - the random system resource will eventually be garbage collected

Not necessarily! In fact this is one of the major problems for non-memory resources in GC'd languages: the GC is triggered by memory pressure and that's it. Because it knows nothing about fd pressure, if fd consumption is not matched by enough memory consumption (or a properly matching pattern thereof) it's rather possible to keep leaking fds without the GC ever running (or ever running a collection which reclaims your file objects).

For instance, back then trying to install some packages would crash under pypy due to file exhaustion, because setuptools leaked fds all over the place: https://bitbucket.org/pypy/pypy/issues/878

The stdlib also did: https://bugs.python.org/issue13133

>the GC is triggered by memory pressure and that's it. Because it knows nothing about fd pressure,

So why not teach it about fd pressure?

There's no reason not to teach a GC about fd pressure - the reason that's rarely done is that language implementers generally don't care that much about the system interface - they just link against libc and call it done.

> You can't even define a custom hashing function

All objects subject to GC have a memory impact, very few of them have an fd impact. You either have to separately track those that impact fds, or do a lot (up to 100%) unnecessary work in response to fd pressure. Even if user code can disable the fd pressure response, you've got to carry the baggage of a GC with extra code to handle it around. It's very easy to see “teaching a GC about fd pressure” as not worth the time and effort for the main runtime GC for a general purpose language. If you had a facility for swapping in special purpose GCs for particular applications, building a specialty GC that added fd awareness might be a net win for some apps, but...

>It's very easy to see “teaching a GC about fd pressure” as not worth the time and effort for the main runtime GC for a general purpose language.

Yes, that's what I said:

>language implementers generally don't care that much about the system interface - they just link against libc and call it done.

There's no research done in this area (at least I don't know about any). I'm sure if people cared about it, they would find good solutions. But they mostly don't care about the system interface - which leads to the problems that this article is talking about.

(Not that Python is on the fore-front of CS research - it's downstream of more innovative languages - but if those languages cared and had done research about the system interface, they would have come up with good ideas that might flow down to Python)

Another problem: Windows support. Windows' file semantics are different and knowing when a file is closed becomes critical.

Elaborate?

Unclosed file handles are incredibly painful on Windows. The user can't edit or delete the file while it's in use and gets cryptic errors about it. There is no UI showing which process is holding the file handle.

The standard practice on Windows when faced with a file that's got a stuck handle is to reboot.

> There is no UI showing which process is holding the file handle

What about process explorer?

https://docs.microsoft.com/en-us/sysinternals/downloads/proc...

Or Unlocker

omg I had never heard of this tool. Thank you so much!

Wow, thank you for putting into writing one of my greatest pet peeves about Windows. Never thought about how often I encounter this issue until now, and how bad the UX is.

Honestly, everything about the Explorer feels 2 decades behind he rest of Windows 10. Like, the rest of MS is has moved on, but their file browser is like an IE6-esque albatross. Everything makes it lock up.

Explorer is the only Windows 10 app that doesn't make me hate life. I'll take 2010 functionality over "metro" UI apps running in UWP anyday.

I just wish it didn't lock the window at the slightest provocation.

Connecting to a slow network share? Rclicked and one context extension is acting up? Just feeling cranky today? Well (not responding).

And if you want to close the (not responding) window, I hope you didn't want any of your other windows, including you in-progress file copies.

Wacko conspiracy theory: Microsoft really wants the end user completely separated and unaware of the file system so the user can't tell the difference between cloud and local files. This allows Microsoft to create dependency on its own cloud services, the government to have access to everyone's files, and laptop makers to cut down local storage significantly.

Therefore Explorer is no longer going to get any love from Microsoft.

It’s because the file system doesn’t have inodes. It’s nothing to do with Explorer

I thought MFT indexes were the same as inodes.

In Unix files are pointers to pointers so you can have multiple files pointing the the same thing (hardline).

You can’t do this in Windows and that’s also why you can’t delete in-use files or directories with files inside

Relevant xkcd: https://xkcd.com/1888/

For a time Adobe Acrobat would automatically open all PDF files on all USB sticks on my computer. Since at the time moving PDFs around was my main reason to use USB sticks it didn't took long until I simply decided to remove Adobe Acrobat from my computer.

What are you all doing where deleting a file that’s in use is such a primary concern, or even at all desirable?

I’ve had as much annoyance with files that have been “deleted” but are still using disk space (and still being used!) on Linux, as files that can’t be deleted because they’re in use on Windows.

At least “can’t delete it because it’s in use” gives you a clue what’s going on, rather than a mystery amount of missing “free” disk space.

That does depend how the file is opened. If it's opened with full file share rights (read/write/delete) then it won't block access.

Delete is different in the sense that deleting a file is just adding a flag which tells the OS to delete the file once all file handles are closed. However newer versions of Windows 10 use POSIX delete semantics by default on NTFS. This means the file will be deleted immediately.

> However newer versions of Windows 10 use POSIX delete semantics by default on NTFS. This means the file will be deleted immediately.

Is that true? I thought POSIX deletion semantics only happened if FILE_DISPOSITION_POSIX_SEMANTICS is passed to SetFileInformationByHandle function (at FileDispositionInfoEx level), otherwise traditional Windows deletion semantics applied. (If POSIX semantics was the default, that might break legacy apps which were not expecting it – the kind of breaking change that Microsoft is usually careful to avoid.)

If you use the DeleteFileW function then it handles all that for you, including setting FILE_DISPOSITION_POSIX_SEMANTICS. At least it did the last time I tested it. But sure, don't take my word for it. Test it.

Edit: Just tested again. I'm not sure how you create files with FILE_SHARE_DELETE in Python so I used another program to create a file and hold the handle open. I then used `os.remove` to successfully delete the file while the handle was open. I could also delete the file from Windows Explorer. This was on Windows 10 1909 on the machine I use for development, thus it is possible that I've set some experimental option somewhere that affects Windows' behaviour. So further tests are welcome.

On further research, I believe you are correct.

I am just surprised that Microsoft would change the default behaviour of DeleteFileW in a Windows 10 build. It seems like the kind of thing that would cause backward compatibility issues. Indeed, I've even found people reporting things breaking due to this [1]. Maybe Microsoft's culture doesn't put the same emphasis on backward compatibility that it used to?

[1] https://stackoverflow.com/questions/60424732/did-the-behavio...

By default Windows locks a file while it is open and does not let you delete it. Other platforms do not, so code which naively reads and then deletes a file without explicitly closing it will work fine until you try to run it on Windows.

Reply on news.ycombinator.com