StefanKarpinski

On UNIX, paths are UTF-8 by convention, but not forced to be valid. Treating paths as UTF-8 works very well as long as you hadn't also make the mistake of requiring your UTF-8 strings to be valid (which Python did, unfortunately).

On Windows, paths are UTF-16 by convention, but also not forced to be valid. However, invalid UTF-16 can be faithfully converted to WTF-8 and converted back losslessly, so you can translate Windows paths to WTF-16 and everything Just Works™ [1].

There aren't any operating systems I'm aware of where paths are actually Shift-JIS by convention, so that seems like a non-issue. Using "UTF-8 by convention" strings works on all modern OSes.

[1] Ok, here's why the WTF-8 thing works so well. If we write WTF-16 for potentially invalid UTF-16 (just arbitrary sequences of 16-bit code units), then the mapping between WTF-16 and WTF-8 space is a bijection because it's losslessly round-trippable. But more importantly, this WTF-8/16 bijection is also a homomorphism with respect to pretty much any string operation you can think of. For example `utf16_concat(a, b) == utf8_concat(wtf8(a), wtf8(b))` for arbitrary UTF-16 strings a and b. Similar identities hold for other string operations like searching for substrings or splitting on specific strings.

Absolutely right. Deprecating direct string indexing would have been the right move. Require writing `str.chars()` to get something that lets you slice by Unicode characters (i.e. code points); provide `str.graphemes()` and `str.grapheme_clusters()` to get something that lets you slice by graphemes and grapheme clusters, respectively. Cache an index structure that lets you do that kind of indexing efficiently once you've asked for it the first time. Provide an API to clear the caches.

Not allowing strings to represent invalid Unicode is also a huge mistake (and essentially forced by the representation strategy that they adopted). It forces any programmer who wants to robustly handle potentially invalid string data to use byte vectors instead. Which is exactly what they did with OS paths, but that's far from the only place you can get invalid strings. You can get invalid strings almost anywhere! Worse, since it's incredibly inconvenient to work with byte vectors when you want to do stringlike stuff, no one does it unless forced to, so this design choice effectively guarantees that all Python code that works with strings will blow up if it encounters anything invalid—which is a very common occurrence.

If only there was a type that behaves like a string and supports all the handy string operations but which handles invalid data gracefully. Then you could write robust string code conveniently. But at that point, you should just make that the standard string type! This isn't hypothetical, it's exactly how Burnt Sushi's bstr type [1] works in Rust and how the standard String type works in Julia.

[1] https://github.com/BurntSushi/bstr

It was cobbled together from msun and other sources to make a complete, portable, open source, liberally licensed libm, which—shockingly enough—did not exist in 2010 when we put it all together.

I’m curious about the choice to call this Idris 2 v0.6.0 rather than the more obvious and semver-compatible Idris 2.6.0. Anyone know if Edwin had written about this anywhere?

The Julia ecosystem (BinaryBuilder specifically) forbids libraries built with -ffast-math for exactly this kind of reason.

From this description this is fairly obviously the LightGraphs situation, and that's a pretty misleading account of what happened. The person was not kicked out of anything—they were not blocked or banned from any platforms or forums. They choose, of their own volition, to stop participating in the community. I've never seen any evidence of bullying for political views or otherwise; maybe there was some, but if so it was never reported, and it would have been a clear and actionable community standards violation. Whatever their reasons, this person decided they wanted to leave, which is unfortunate—we don't want anyone to feel unwelcome—but it's their prerogative.

That would have been fine, if unfortunate, but they also wanted to "take their work with them" in the sense of archiving their registered open source package repos preventing any further maintenance or development. This desire was not about not wanting the maintenance burden—they were not willing to grant ownership of the repos to other maintainers. In short, the original author wanted to force all development of the packages by anyone to stop. Of course, that would have left all the people who had come to depend on those packages high and dry, since the code they'd come to depend on would get no bug fixes, security patches, etc. Despite the fact that there were active contributors to that code who were happy to take over maintenance.

Imagine if Linus Torvalds got mad one day and decided to insist that no one could do any further development of the Linux kernel. No bug fixes, no security patches, no new features. Linus out. That was the situation here. Fortunately this is not how open source works: open source licenses are not revokable and the ability to fork a project is baked into each license for this exact reason—so that a disgruntled author cannot screw over an entire community of people who have come to depend on their work. They don't have to keep doing work, but they also can't take away they've done. If Linus threw a tantrum and refused to allow any more work on Linux, the rest of the community could take over and continue maintaining the kernel—fixing bugs, patching security flaws, even adding new features. Linus could close down his git repo and never touch the kernel again, but other maintainers could continue to develop Linux and support the vast community of users who have come to depend on it.

Similarly, it would have been perfectly legal to fork LightGraphs and continue development in a new repo with the same name. Out of respect for the original author's wishes, however, the LightGraphs package was allowed to be "frozen" with no further development. But it would have been deeply irresponsible to cease all maintenance and leave all the people who use and depend on LightGraphs hanging, especially given that there were willing maintainers. So LightGraphs was forked and renamed to "Graphs"; the old repo has been allowed to remain frozen, while maintenance and development has continued under the Graphs name in a new repo. The author of LightGraphs got their wish for work on the thing called "LightGraphs" to cease. The users of the package didn't get screwed over since they can do a simple search and replace and keep using a maintained graphs package. Personally I think the community handled it with responsibility and grace.

An arrangement that would make sense would be to separate data collection from publication and make the academic success of data collectors predicated on how many publications use their data. That would incentivize data collectors to make their data as widely useful and available as possible, which is very counter to the current situation. I imagine data collection groups talking to a number of data analysis groups and figuring out what to collect that would be useful for the largest number of interesting experiments. This might also make data collected intentionally or accidentally to give a specific outcome less likely.

Unfortunately, it's absurdly hard to change what counts as credit towards a successful academic career, so this seems like a pipe dream. Academia is still waging a very uphill battle to get Research Engineer to be a long-term viable career path, despite the absolutely critical need for software engineering in all areas of modern science.

This press release is so vague and hypey as to be unintelligible. The paper itself (linked from the article) is much clearer: "Practically Correct, Just-in-Time Shell Script Parallelization" https://nikos.vasilak.is/p/pash:osdi:2022.pdf.

Inspired by this I wrote a post on the Julia discourse forum about the arbitrariness of truthiness in programming languages: https://discourse.julialang.org/t/on-the-arbitrariness-of-tr.... Julia requires actual `true` or `false` values in conditionals. Some people are annoyed coming from languages where they can test if a list is non-empty without having to write `isempty(a)`. But I have no regrets and this is a great example of why. Languages cannot agree on truthiness because it's arbitrary and just becomes a set of random rules you have to remember and which causes nasty bugs if you get them wrong.

Here's a post I wrote on Julia discourse about why one-based indexing is not the culprit here, no matter what people may feel about it aesthetically: https://discourse.julialang.org/t/offsetarrays-inbounds-and-....

Several of the most serious scientifically minded programming languages have also had arrays that can be indexed starting at different offsets, including Fortran, Chapel and Fortress. If Julia were zero-based OffsetArrays would still exist and be highly useful. OffsetArrays is not some "kludge" just to allow indexing from zero. Frankly, indexing from zero isn't improtant enough for that. What really makes OffsetArrays useful is being able to do things like have indices go from -k:k where k may depend on the particular dimension. That way the point A[0, 0, 0] is the center of the array and you can navigate up/down/back from there naturally. Of course you can simulate that with arrays that arrays that start at zero or one, but it's a major pain.

I guess that’s pragmatic but having native arrays and “scientific arrays” be totally different incompatible things is just so unsatisfying.

I’m curious if anyone knows what approach Owl takes to making sure float arrays, complex arrays, etc. are unboxed. (Or if it leaves them boxed.)

Offset arrays aren't a kludge and the package would exist regardless of whether zero or one has been chosen as a default base for indexing. Having arbitrary index ranges in different dimensions is extremely useful in many application domains. When working with FFTs, for example, it's natural for the indices to be symmetrical around zero. Or when doing coordinate transforms like in this example from the excellent Images.jl package: https://juliaimages.org/stable/tutorials/indexing/.

In particular, not a single issue mentioned in this article would have been prevented by static type checking.

That's why interfaces are useful—they save you from that. But they don't actually solve the problem of checking that an abstraction has been implemented correctly, just that you've implemented the entire API surface area, possibly incorrectly. Note, however, that if you have a way of automatically testing the behavioral correctness of an implementation, then those tests presumably cover the entire API, so automatic testing would subsume the benefit that static interface checking provides—just run the automatic tests and it tells you what you haven't implemented as well as what you may have implemented incorrectly.

The big language design problem that I think this post highlights is that the flip side of Julia's composability is that composing generic code with types that implement abstractions can easily expose bugs when the caller and the callee don't agree on exactly what the abstraction is.

Several of the bugs that Yuri reported are a very specific case of this: there's a lot of generic code that assumes that array indexing always starts at one, but that's not always the case since OffsetArrays allow indexing to start anywhere. The older code in the stats ecosystem is particularly badly hit by this because it often predates the existence of OffsetArrays and the APIs that were developed to allow writing efficient generic code that works with arrays that don't start at the typical index (or which might even want to be iterated in a different order).

Fixing these specific OffsetArray bugs is a fairly straightforward matter of searching for `1:length(a)` and replacing it with `eachindex(a)`. But there's a bigger issue that this general problem raises: How does one, in general, check whether an implementation of an abstraction is correct? And how can one test if generic code for an abstraction uses the abstraction correctly?

Many people have mentioned interfaces and seem to believe that they would solve this problem. I don't believe that they do, although they do help. Why not? Consider the OffsetArray example: nothing about `for i in 1:length(a)` violates anything about a hypothetical interface for AbstractArrays. Yes, an interface can tell you what methods you're supposed to implement. There's a couple of issues with that: 1) you might not actually need to implement all of them—some code doesn't actually use all of an interface; 2) you can find out what methods you need to implement just by running the code that uses the implementation and see what fails. What the interface would guarantee is that if you've implemented these methods, then no user of your implementation will hit a missing method error. But all that tells you is that you've implemented the entire surface area of the abstraction, not that you've implemented the abstraction at all correctly. And I think that covering the entire surface area of an abstraction when implementing it is the least hard part.

What you really want is a way to generically express behaviors of an abstraction in a way that can be automatically tested. I think that Clojure's spec is much closer to what's needed than statically checked interfaces. The idea is that when someone implements an abstraction, they can automatically get tests that their implementation implements the abstraction correctly and fully, including the way it behaves. If you've implemented an AbstractArray, one of the tests might be that if you index the array with each index value returned by `eachindex(a)` that it works and doesn't produce a bounds error.

On the other end, you also want some way of generating mock instances of an abstraction for testing generic code. We do a bit of this in Julia's test suite: there are GenericString and GenericSet types, which implement the minimal string/set abstraction, and use these to test generic code to verify that it doesn't assume more than it should about the string and set abstractions. For a GenericArray type, you'd want it to start at an arbitrary index and do other weird stuff that exotic array types are technically allowed to do, so that any generic code that makes invalid assumptions will get caught. You could call this type AdversarialArray or something like that.

I've personally thought quite a bit about these issues, but as Keno has said, there hasn't been time to tackle these problems in the last couple of years. But they certainly are important and worth solving.

On a personal note, Yuri, thanks for all the code and I'm sorry to see you go.

Heck, this is the entire memory management model of PHP, which I found shocking when I learned it, but makes sense given that the language is intended to generate web pages: just allocate, never reclaim the memory, then kill the process when you’re done.

Good idea! One problem is that some people will swipe right on everything in one or both categories. Limiting the number/fraction of right swipes might help.

Startup time for Teams is hilariously bad. If I start trying to join a Teams meeting when it starts I have to warn people I’m going to be a few minutes late. My computer is not especially slow. I’m sure everyone at Microsoft has it running all the time and doesn’t have to wait for it to start but I don’t because it’s not my only or even my main meeting software.

This post is a really good illustration of UTF-8. Very clear! The key brilliance of the design is not only the embedding of ASCII in UTF-8, but the fact that nothing in ASCII can appear anywhere else UTF-8, and more generally that no UTF-8 character can appear as a substring of another character’s encoding. That means that all the byte-oriented libc string functions just work. I wrote this up recently in a StackOverflow answer with some examples: https://stackoverflow.com/a/69756619/659248

Your argument is that “Python can call fast code written in other languages”? That is true, of course, but if that’s your definition of Python being fast, then it makes the notion of one language being faster than another meaningless.

Eagerly awaiting matrix libraries written in pure Python that outperform BLAS.

Sometimes I would imagine (I’m no expert in Rust but do know a bit about compilers), Rust’s ability to guarantee unshared access to memory can probably enable optimizations that are hard to coax out of a C compiler.

Many libc functions are also much slower than they could be because of POSIX requirements and being a shared library. For example, libc’s fwrite, fread, etc. functions are threadsafe by default and acquire a globally shared lock even when you aren’t using threads (you can opt out, but it’s quite annoying and non-standard) which makes them horribly slow if you’re doing lots of small reads and writes. Because libc is a shared library, calls to its functions won’t get inlined, which can be a major performance issue as well. By comparison Rust’s read and write primitives don’t need to acquire a lock and can be inlined, so a small read or write (for example) could be just a couple of instructions instead of what a C program will do, which is a function call (maybe even an indirect one through the PLT) and then a lock acquisition, only to write a few bytes into a buffer. That’s a lot of overhead!

And finally, Rust’s promise of safe multithreading no doubt encourages programmers to write code that utilizes threads in situations where only the truly courageous would attempt it in C.

Awful but plausible.

An interesting aspect of this effort is that it undoubtesly makes modifying these tools much easier. While GNU and the FSF are all about legally ensuring the ability to read and modify their code, they have, in practice, made it almost impossible to do so for entirely technical reasons. I’m a pretty good C programmer and whenever I’ve looked at even simple GNU tools like `cat` or god forbid GNU libc’s `printf` code (this one is actually kind of delightful it’s so nuts), I have despaired of understanding, let alone modifying, the code. GNU coreutils are such a morass of preprocessor defines and bonkers C code to support every legacy system GNU has ever run on, that it might as well only be distributed in binary form. I think it would literally be easier to decompile most GNU tools and modify the decompiled platform-specific source than to make the same modification to the source as shipped. Rust has a steep learning curve, but learning Rust is a picnic compared to figuring out how to modify just one GNU tool, and of course they’re all different. Rust puts the onus of portability where it belongs: in the language and compiler, instead of forcing it on every single programmer and application.

Tip: the first move when trying to write a macro is doing `Meta.@dump` on examples of argument expressions you want your macro to consume and produce. Then write code that transforms the inputs to the outputs.

This is such a bad and US-centric example: for anyone who lives in a place where it is easy to get around without a car, hiring a car only when you need it is a no-brainer financially and owning a car is a totally waste of money.

Is the concept that UTC would become "detached" from a particular location on Earth and the offset of time zones relative to UTC would change over time (very infrequently since it would take on the order of a millenium for a time zone to shift by half an hour).

If I've learned one thing about dealing with exceptional events, it's that the less often they happen, the less likely you are to handle them well. Switching from leap seconds every few years to leap minutes every century, pretty much guarantees we'll screw it up. But maybe that's ok? Y2K was a fat nothingburger, after all the fuss.

Fully agree with this, although one case where Java’s lack of unsigned integers is a quite bad is in representing binary data because the byte type is signed. I recently had cause to write some Java code that did a lot of modular arithmetic with arrays of bytes and oof, what a mess. There were four or five subtle bugs caused by the unexpected signedness of the byte type. I also wrote the same thing in Julia using UInt8 vectors and had no such bugs.