The history of the Schwartzian Transform (2016) | Hacker News Reader

The history of the Schwartzian Transform (2016)

perl.com

54 pointsmooreds21 comments

I wrote a blog post some time ago which contrasts the "Schwartzian transform" with the slightly clearer alternative used by most APL-family languages and (in my opinion) an even cleaner approach which is possible in languages with a first-class table type: https://beyondloom.com/blog/rankingoffruits.html

That was a lovely vignette, thanks. Much earlier in my career I had exactly the sense suggested that “something’s off” about the Schwartzian, even though it became a useful and idiomatic tool at the time. I only really understood the disquiet much later, when working with columnar data, and realising the positive qualities of declaring structs-of-arrays rather than arrays-of-structs, with subsequently excellent mechanical sympathy, and so on from there.

Unfortunately, the only such language to which most workaday devs have access is some dialect of SQL. Fortunately, at least there's that.

I find it interesting that the transform was controversial in the '90s.

Today, it seems like a normal solution to the problem to me, and the controversy seems silly. I have much experience with the map function from Javascript. It is too simple to be objectionable.

But in the '90s, I would also have had trouble understanding the transform. Lambdas/closures were unheard of to everyone except Lisp dweebs. Once I figured out what the code was doing, I would be suspicious of its performance and memory consumption. This was 1994! Kilobytes mattered and optimal algorithmic complexity was necessary for anything to be usable. Much safer to use a well understood for loop. I have plenty of experience making those fast, and that's what map() must be doing under the hood anyway.

But I would have been wrong! map() isn't doing anything superfluous and I can't do it faster myself. The memory consumption of the temporary decorated array is worth it to parse the last word N times instead of N log N times. Lisp is certainly a slow language compared to C, but that's not because of its lambdas!

I believe I first came across this in the early xslt days, when I had a manager that wanted to write all software with an XML requirements spec, then use xslt to simultaneously generate code AND documentations.

1. It did not work because of course it didn't work 2. This meant that all data output of the system would done in XML and transformed, so I got to know xslt quite well 3. It did give me a fun moment at a conference where a speaker asked "Who knows the Schwartzian transform or has used it?" and only me and my coworkers raised their hands

I was curious to see what other languages do by default (or not), so I quickly tested a few:

C# evaluates the lambda only once per item, in input order: https://dotnetfiddle.net/oyG4Cv

Java uses "Comparators", which are called repeatedly: https://www.jdoodle.com/ia/1IO5

Rust has sort_by_key() which is encouraging... but also calls the key transformation many times: https://play.rust-lang.org/?version=stable&mode=debug&editio...

C++20 with std::range also calls the lambda repeatedly: https://godbolt.org/z/raa1766PG

Obviously, the Schwartzian Transform can be manually implemented in any modern language elegantly, but it's interesting that only C# uses it implicitly!

PS: The fancy solution would be a temporary "Hashset<T,K>" used to map values to the transformed sort keys, so that the mapping Foo(T t): K would only be called once for each unique input value 't'. That way the transformation could be invoked less than O(n) times!

Inspired by your behavior, I looked up the Wikipedia article on Schwartzian transform[1]. The article mentioned that Rust not only provides sort_by_key, but also sort_by_cached_key. :D

[1]: https://en.wikipedia.org/wiki/Schwartzian_transform

I keep saying to colleagues that simply knowing that something exists in a library somewhere is 90% of the skill of modern programming.

I suspect the profession of software archeology form the Vernor Vinge science fiction novels will soon become a reality, especially now that old languages can be uplifted mechanically with AI.

This one is particularly easy to chance upon, though, by either browsing the docs of slices’ `sort*` methods (because they’re all clumped together on the page), or even simpler, noticing it when you type `foo.sort` and your editor suggests autocompletions.

The reason Rust has a separate `sort_by_cached_key` is, of course, that it doesn’t make sense to cache cheap projections which are the 95% use case.

The implementation is somewhat more sophisticated than just the naive transform:

> The current implementation is based on instruction-parallel-network sort by Lukas Bergdoll, which combines the fast average case of randomized quicksort with the fast worst case of heapsort, while achieving linear time on fully sorted and reversed inputs. And O(k * log(n)) where k is the number of distinct elements in the input. It leverages superscalar out-of-order execution capabilities commonly found in CPUs, to efficiently perform the operation.

> In the worst case, the algorithm allocates temporary storage in a Vec<(K, usize)> the length of the slice.

Indeed, that works the same as the default C# sort: https://play.rust-lang.org/?version=stable&mode=debug&editio...

Perl was ahead of its time, and likely because of it's Borg-like capacity to absorb a little bit of everything... it was unfortunately nigh on impossible to refactor. This is why Perl6 (nee Raku) took nearly 2 decades to be "done". It's a crying shame that it lost momentum, because many of the techniques that the features of Raku enable are as clever/succinct/revolutionary as the Schwartzian Transform was back in its day... but because the developer community is so very small now, Raku will have a very hard time catching up to the performance of competing languages.

Python's key-functions nicely encapsulate the whole process.

> Some people want Perl to be accessible at first glance to someone who doesn’t know the language.

…oh boy.

In retrospect it doesn’t seem like these people prevailed.

Thank you for posting! I always felt like such a badass in the late 1990s when I used this.

I'm so grateful for the Lisp translation. Perl is such a write-only language, my hair curls up just from looking at it.

To be fair, this isn't what normal Perl usually looks like. Perl allows you to write like this but most people don't. This is more like Duff's Device in that it works and it's clever and you'd can anyone who put it in production code.

That's not to say that idiomatic Perl doesn't have its quirks and oddities - Perl idioms do throw off people who aren't used to them. But this is a bit far even for Perl.

I've definitely seen it in production code, but it was a good deal after 1994.

By then it was a commonly known technique, so it was a pattern a Perl dev would be expected to recognize.

If the production code has to be maximally performant, you'd want the transform. It doesn't copy data around with a bunch of needless assignments. Pipelining the extract/sort/reconstitute in this manner is absolutely the right thing to do in production if production requirements are to keep it fast and with a small footprint. There's no excuse for not commenting it, but strictly-code-speaking there's no reason not to use this for production if the requisite needs are there.

Eh. It's not worse than Ruby, and vice versa.

> X is such a write-only language, my hair curls up just from looking at it.

That would be a powerful language feature, but alas

X.isWriteOnly() for X in aLanguagesIdonotUse

is merely akin to

X.hasTooManyRules() for X in aHumanLanguagesIdonotSpeak

Reply on news.ycombinator.com