Variadic generics, again
poignardazur.github.io
poignardazur.github.io
But soon after you will realize that you need to do some operations with those tensors such as A + B + C. To compute this expression compiler will first need to compute (A + B) sub-expression and store the result into the temporary tensor. Only then it will be able to use this temporary result to compute the final expression by adding the C tensor. When you have a lot of such expressions to compute, and expressions are arbitrarily complex, you will want to avoid inefficiencies caused by creating many temporaries - there's an attached cost to them both in memory and in compute. And sometimes you will be able to fuse multiple operations into a single one.
This is a problem that can be solved with lazy evaluation and in C++ this can be done in compile-time through expression templates. And this is what basically all linear algebra C++ libraries are famous of doing.
So Rust has that and C++ has template specialization, const if, and variadic templates. Would be cool to have all in one language.
> So you can eliminate temporaries at compile time.
> ... but the library user can just write A + B + C ...
> So Rust has that.
Eliminating temporaries in C++ is also done in compile-time through expression templates or more generally through TMP.
I'm pretty sure Rust type system is incapable to express many (?) of the TMP things from C++ - it does not even support partial specializations to begin with so it's hard to imagine that you can do a lot of TMP gymnastics. I think that this is an area where it is generally accepted that Rust is lacking against C++.
You often don't want that sort of specialization. Code like deep neural nets is a pain to handle (unless each layer is identical, which is admittedly moderately common) even with variadic generics and especially without, and it's hard for the compiler to generate the right specializations for (and the right number of specializations; anecdotally, you can expect enough code bloat that it causes problems).
Using branded types you can get a weak form of dependent typing and use that to get features like Tensor<f64, Dim<16>, Dim<MultipleOf<8>>, Dim<SensorChannel>>, where you're allowed to pass in concrete numbers, comptime-known constraints on runtime-known values, or even just weak information like that there's a particular brand (SensorChannel) on the type which allows you to match fully runtime-known values at compile-time and let the type-checker handle it (think of functions like matmul(Mat(A,B), Mat(B,C)), where the B has to be the same for both, but you very well might want to know it at runtime). It's still the library writer's job, unfortunately, to use `inline` and other such keywords appropriately to guide the compiler to the right number of specializations, but the API presented to people using that tensor library can be reasonably nice.
Incidentally, Zig _also_ makes that second thing (branded types) easy to write. I hacked a demo together [0] if you'd like to play around with the concept, but a dedicated tensor library should eventually offer something less verbose IMO.
If you're doing this with ML, my preferred strategy is to go the other way. Only write specializations that are easy for the chip to execute (e.g., appropriately aligned and length-aligned large powers of 2), and build code on top of that. If the computational kernel has to do padding or shifting or other garbage, even at the head/tail of a vector, then something in your model is horribly wrong anyway; just make that unrepresentable.
However what you can't do without dependent types (as far as I know) is infer or verify statically that the transpose of a Tensor<f64,20,10> is a Tensor<f64,10,20>.
Edit: I think that dependent typing is needed only of the matrix bounds are only known at runtime.
Focus on the compile error story. In particular, the design should avoid anything that leads to post-monomorphization errors.
Rust is already incredibly complex, and most of that complexity comes from dozens of different systems that share few concepts all with their own syntax (modules, macros, traits, e.t.c.), so adding yet another syntax for static loops begs the question if some of this can't be unified.
Zig is a seriously flawed language, but comptime is the right abstraction for metaprogramming in an imperative language.
The fact that you call it "dogma" (and your other comments about Zig) suggests that you aren't going to be very open to changing your mind on this.
But as far as "the touted advantages" go, I've experienced them firsthand.
Being able to write generic code and know that if the code compiles in your library, it will compile for people who depend on it, is a pretty nice workflow. Being able to change that generic code and still be confident it will still compile for downstream users who download the new version is pretty nice too.
Being able to download some random dependency full of generic code, and to instantiate that code without having to dread reading mile-long call stacks that end with "foo.bar is not implemented in this function you didn't write" is pretty nice too.
(And yes, post-monomorphization errors can still happen in theory; I've never seen one happen in practice, certainly not with the frequency I saw the mile-long call stacks in C++.)
Those aren't some aesthetic details that language designers cling to out of some perfectionist desire to keep the language theoretically pure. It's part of what makes pulling dependencies especially painless in Rust.
Rust as is, is a de Bruijn criteria fulfilling proof language, with the macro system ending up as the meta language.
However I do overall agree with you, that rust needs to accept post-monomorphization errors. I see no way to get usable const-generics otherwise.
I think what it boils down to at the end is where you want your pain to be, pains from constantly hitting limitations and providing an escape hatch like macros, or pains from having a system that is too powerful and all the chaos that brings.
I'd rather reign in the chaos, after all we also choose to be Turing-complete, and not always terminating like Coq, but mmv.
This results in miscompilations, broken buildins, features added that are then ripped out again.
I think fast iterations and experimental stuff are ok if you have a stable core. I.e. I don't mind if zig decides to change its for loop syntax after 5 years, so long as there is a simpler stable construct like `loop { break; }` that people that use the language in production and library maintainers can use.
You need a certain critical mass for a language to be self sustaining, and that requires people to actually use your stuff, which in turn requires at least some stability guarantees for some subset of the language.
Zig's response to that is "I'm just my inventors toy yet, but it will be grand when version 1.0 hits in 10-50 years."
The fact that they now want to replace LLVM with their own backend makes me think that even 50 years would be optimistic.
To be fair go has never used llvm and was delivered in less that 50 years.
Getting to 70 percent of llvm will be relatively easy for zig and like all others, they will congratulate themselves and pretend it validates their choice. They will then discover the other 30 percent happens 0.1% at a time, because there are no silver bullets. This will take them 10+ years because it always does.
And if Zig was a google project instead of a 3 person show, I'd be far more inclined to believe that they can pull this off.
But as previously said, the current style of development makes it hard to attract contributors and more importantly funding.
Not linking to LLVM would also make the barrier to entry on contributing to Zig itself lower, because now you can just compile Zig with Zig, and not worry about other dependencies.
All of this would've come for "free" (lets and lots of developer hours) if introspection is further developed. It's truly a tragedy that things went down how they did.
When you're talking about functions operating on mixed, variable length, types, what meaningful logic can you actually apply, that doesn't fall into requiring the types to implement an interface anyway?
When you're talking about struct/classes/etc, what do you need over and above defining a class/record/dataclass? It would seem to me, you're just skipping the part where you give your tuple a name, but is that even a good idea to do?
To be clear, I did not find the examples in the post convincing, as they represent mostly wrappers/adapters for implementing or consuming arbitrary types, but there are no examples for how that might actually be useful downstream.
Imagine each packet requires a size header. When you want to send multiple packets at once, you now want to optimize that and only write a single initial header to the interface, preceding the data.
With variadic generics, you can enable a syntax like `interface.write_packets(packet1, packet2, packet3, /* ... */);` which writes the packets in the desired optimized way. It can internally construct a serializable data tuple from the variadic generics and add the correct header only to the beginning.
Without variadic generics a similar syntax is only possible with macros, which means that it cannot be implemented as a reusable trait.
> It would seem to me, you're just skipping the part where you give your tuple a name, but is that even a good idea to do?
Sometimes you explicitly don't want to require naming the tuple for flexibility, like in the example above.
A lot of the value is to derive complex structs programmatically that may never be realized concretely but feed other metaprogramming algorithms that slice and dice them and make decisions based on the properties of the individual types. You don’t want to name them because (1) there may be hundreds from an unbounded set and (2) your may never need to reference them directly, so it saves a mountain of code and eliminates a massive amount of edge cases. At their best, you can use them to express insanely complex things with surprisingly little code.
It is not the kind of feature you use profligately in a code base. But when you need them, you really need them. Most of what they can express in ordinary code would otherwise require complex code generators, compiler magic, or outrageous amounts of boilerplate.
Rust has enums, which are so much better it isn’t even funny.
impl Trait for (T1)
impl Trait for (T1, T2)
impl Trait for (T1, T2, T3)
...
Someone, somewhere cursed there weren't variadics there.
Would I use variadic generics regularly? Probably not. But in places where they'd be useful - boy, would they really be useful!
because you can make that a single macro which kinda is a bit like variadics. Through it still is limited to some specific number of your size (typical 16 or 32).
Anyway not having varadic generics by itself is in most cases just a small inconvenience with limited benefits and there are many far more important priorities IMHO.
The main benefit for varadic generics is that they might unblock some other improvements on top of them.
Even if you could think of one, I personally think the rust community does a poor job of weighing the tradeoff between supporting a niche feature and adding complexity to the language. Rust is already so huge that all but the most dedicated have little hope of understanding even half of the language features. I'm not sure I understand the incentive structure that makes it this way, but it is hard for me to see how rust avoids becoming the next c++ unless they learn how to say no. Honestly I believe it's already too late and that rust's best ideas will be adopted by a smaller language with better ergonomics, compiler performance and learnability that will eat its lunch.
- it is duck typed, which to be fair macros are also, but variadic generics would be type checked at the definition;
- it interacts badly with traits, that is if you allow both querying for implemented traits and creating new trait `impl`s then this can create cycles in the execution of comptime blocks, meaning the compiler needs to compute a fixpoint or arbitrarily give an error.
Anything higher-order. At the most basic level you want to be able to take a bundle of values and a bundle of functions and apply the functions to the values, or take a bundle of functions and another bundle of functions and make a bundle of composed functions. You will probably complain that these examples are trivial, but if we don't have the syntax or vocabulary to handle the trivial cases then how can we even start to think about the more complex cases?
> When you're talking about struct/classes/etc, what do you need over and above defining a class/record/dataclass? It would seem to me, you're just skipping the part where you give your tuple a name, but is that even a good idea to do?
It is, if only because you want to be able to transform existing records generically rather than having to concretely write out every specialised possible version of your record. For example, think of writing a generic "patch" method for use in the "update" route of a CRUD webservice - something that lets you call it with an existing record (e.g. a user) and some representation of a few deeply nested fields of that record (e.g. the postal code that's on that user's address) and get an updated version of the record, without having to write out the type for that "patch" input longhand. This is a basic, trivial thing that people do in TypeScript every day, but you still can't even write a type for it in Rust, yet alone implement it.
> I think variadics aren’t necessarily useful for application developers, or people working on libraries with mostly concrete types (eg web libraries). But for people working on libraries with lots of type manipulation, variadic generics have obvious benefits. They let you go from manipulating types to collections of types.
One example is in SQL libraries. If you have a trait `DatabaseColumnStorable` for types you can store in a database column (integers, strings, etc), then with variadics you could have a `struct DatabaseRow<...Columns: DatabaseColumnStorable>(...Columns)` to represent rows of a database table in a type-safe manner.
Consider having multiple (heterogenous) Promises and wanting to wait for all to complete, and return the results.