Hacker News Reader

Comment by boulos | Hacker News Reader

It’s too bad that the code and correction seem to be locked down, but the linked Twitter discussion has an explanation [1]: the code expects Python’s glob to return a sorted list of files, but it doesn’t guarantee this (the main fix is to sort the result). I had thought reading the Vice article that this would be about case-sensitive vs insensitive filesystems.

[1] https://mobile.twitter.com/bmarwell/status/11818211438352015...

View on HN

Are they locked down? The article links to the paper [0], which has two zip files to download, first of which is the code, including an explanation of the corrections made [1].

[0] https://pubs.acs.org/doi/10.1021/acs.orglett.9b03216

[1] https://pubs.acs.org/doi/suppl/10.1021/acs.orglett.9b03216/s...

Ahh, I failed to click those. They seemed large enough to be “not the code, but the full dataset”. Thanks for the correction!

To me this just raises more questions like "Why does the order of the files even matter?", and "If it does why wouldn't you make sure it was sorted, or better yet sort it in some reliable fashion?".

If you are writing a paper about the order in which a sequence of events happens, the order matters. It's the difference between cause and effect, the heart of science. Filename is often used as part of the data, as a timestamp.

Obviously reliable sorting is important, which is why people are upset that python's glob() turned out to be unreliable

> Obviously reliable sorting is important, which is why people are upset that python's glob() turned out to be unreliable

A quick glance at the documentation didn't mention it, does glob even promise sorting the result? Relying on that to happen when it's not even promised seems negligent.

Most developers look at the result from a function call rather than looking at the documentation.

I mean, I have learned that people simply won't accept that SQL doesn't guarantee the order of a result set if you don't use "order by". I tried to argue it and eventually gave up. It seems vastly easier to change a language than human nature. People are hardwired to ignore claims that something should not be relied on, because in normal human relations, that is a way of pushing away undue responsibility, not something to be taken literally.

> A quick glance at the documentation didn't mention it

What do you mean? It is the very first sentence of the documentation: "results are returned in arbitrary order"

https://docs.python.org/2/library/glob.html

Ah, see, I missed that, I had just skimmed, then grepped for "sort" and "order" (but mistyped that as oder).

It doesn't sort. But people assume it does especially if the files are returned in the order that they were created, which typically will be sorted.

I think this is why it doesn't get noticed; the files are written say

   result-20190901-1303
   result-20190901-1305
   result-20190901-1310
   etc...

IIRC glob.glob() is a wrapper around os.listdir(). I used to develop code on an OS X laptop then push to a Linux box for the actual computational result (big dataset). And discovered at one point that one OS returns a sorted result, the other does not. Sorting is important if you want track/analyse instrument data over the days in a year.

Reply on news.ycombinator.com