Why not Rust?
Edit: I wonder if Swift could be replaced with Rust for iOS development?
Edit: I wonder if Swift could be replaced with Rust for iOS development?
We believe that Rust supports all the ingredients necessary to implement the techniques in this paper: it has a strong static side, and its traits system supports zero-cost abstractions which can be provably eliminated by the compiler. It has a great pointer aliasing model, a suitable mid-level IR, a vibrant and engaging community, and a great open language evolution process.
A concern with using Rust is that a strong goal of this project is to appeal to the entire TensorFlow community, which is currently pervasively Python based. We love Rust, but it has a steep learning curve that may exclude data scientists and other non-expert programmers who frequently use TensorFlow. The ownership model is really great, but mostly irrelevant to the problems faced by today’s machine learning code implemented in Python.
From my point of view, a weak justification is worse than no justification in cases like this.
Rust is much better suited to this task than Swift from a technical point of view. The far superior platform support for Windows and Linux is ample reasoning to say Rust is better suited for this task, since very few data scientists will be training models on macOS. However, that's only one of several areas where Swift has shortcomings for a project like this. Swift is great for iOS and macOS development, of course, since it was designed for that. I don't think Swift is a bad language by any means, and with enough effort, it can be reshaped to be good for Tensorflow... the GitHub document just provides zero useful justification for the work required to make it good for Tensorflow.
EDIT: to some of the replies talking about Rust's learning curve, that mostly applies when you start trying to design efficient, interlinked data structures involving ownership. For most applications of machine learning, this simply wouldn't be a problem. The library would provide the data structures, you just have to use them. Rust can provide simple interfaces to complicated things.[3] The compiler's error messages are usually incredibly helpful.
The learning curve of Rust should not be relevant here, compared to Swift, which is also full of idiosyncrasies. Swift and Rust both have a large learning curve for someone coming from Python. This is because they're statically typed languages that are just different from a scripting language. For an application like this, I would say those learning curves are roughly equal at the language level, but as I pointed out in my comments, Swift has an enormous learning curve of requiring many data scientists to either install and learn Linux, or throw out their current computer, buy a Mac, and learn macOS.
My point here is not that Rust is the most suitable language for Tensorflow (although it could be), but rather I'm making the point that Rust is more suitable than Swift for a project like this, and therefore this document is just annoying. It would be better for them to delete this document and just say "we're using Swift because our team has a lot of experience with it and because the creator of Swift is leading this project, so we would lack enthusiasm and momentum if we were using something else, even if it were more suitable."
Julia would be really interesting to see explored further, since it would appeal much better to many existing data scientists who would be transitioning from Python. The times that I've played with Julia, I was amazed at how slow the JIT is for even tiny scripts. LLVM is powerful stuff, but it is painfully slow at everything. It would be nice if Julia offered an alternative backend for rapid development.
[1]: https://github.com/tensorflow/swift/issues/3#issuecomment-38...
[2]: https://github.com/tensorflow/swift/issues/3#issuecomment-38...
[3]: http://kiss3d.org/
Having done user research on this by speaking to data scientists, I can say that static typing is desired by a nonzero number of who practice what we would consider to be data science and machine learning. Much like how TypeScript is seen as a revelation to hordes of JavaScript programmers who have never used static types before, the ability to get some level of correctness verification at design-time matters.
Enter stage left: Julia
Julia is already pretty great, I'd really love to see what cool stuff we could have with a swell in community size and investment!
[1]: https://github.com/JuliaArrays/StaticArrays.jl
[2]: https://github.com/davidavdav/NamedArrays.jl
In brief, it is not the duty of the automatic differentiation package to favour a specific array type – it just works for all of them, which is something that I find fairly magical with Julia.
2) StaticArrays would be a good starting point, but the point of it is to optimize Arrays by unrolling for loops and triggering SIMD (IIRC) and there are performance penalties when your arrays get really large, which they do, in ML. Something LIKE the staticarrays typesystem but without the overoptimization would be welcome.
3) (kind of tangential) I have beef with how GPU is handled as GPUArray in julia. It really should be handled as a worker node using the ClusterManagers-type semantic; you should be async sending tasks to the GPU as if it were a remote agent (which it kind of is, due to PCI bus bandwidth and latency bottlenecks) and waiting for the result to come back as a Future.
This is being explored with "Live Checking" in F#[0], which offers a form of static typing over TensorFlow without actually forcing you to express every complex interaction with data in types.
[0]: https://github.com/fsprojects/TensorFlow.FSharp#live-checkin...
Who would trade static typing with fast prototyping any time.
Data science is a really nebulous term covering many drastically different domains of CS. Many DS I talked with, don't really produce code, they do coding to produce analysis, which is the actual delivery. For them, code is ad-hoc and disposable, created on demand and left in the dust until rediscovered when mission comes.
Some of the code do survive and enter production stage, I guess that is where they would seek some assurance from static typing. But I do think they could learn to mitigate most of pain if they can commit themselves to write some unit-tests/functional tests, yet such awareness is rare among the DSs I know and worked with.
So all in all, yes static typing MIGHT help, in some way, but I don't think it addresses the underlying pain point as much.
So gradual typing like in Julia is already a good thing for having the best of both worlds.
These need not be at odds. Many ML languges like F# or OCAML, by use of type inference, get you type safety without having to type a bunch of stuff and sacrifice faster prototyping. And certainly in F# there is a history of having productive tooling that lets you prototype easily. Simply writing some F# code in an F# script in an IDE, hitting alt+Enter, and letting it execute in an interactive shell is hugely productive for exploratory tasks. And features like Type Providers build out types for an arbitrary data set that let you guarantee your code is actually correct for the data.
What I've mentioned isn't without its flaws, and eventually someone is going to reach head-scratching problems just as they would in any other environment. I don't think there's an objective way to measure productivity across a wide range of professionals, but I do believe that some subset of them would prefer static types for their work. This is backed by conversations with some of them about problems they encounter.
I do agree with your criticism of the document here, though. It feels very much like Swift happens to check many boxes, but the lack of Windows support is baffling. It's simply table stakes to be able to run, fully supported, on Windows, macOS, and major Linux distributions. That should be the very first thing anyone considers.
But beyond that, I think even with Rust's macro system it could be difficult to make it work for Tensorflow in a way that feels appropriate for Rust programmers _and_ for TensorFlow. This was explored in F# for Tensorflow research[0] and a completely different approach[1] was taken because making a type system suitable for tensorflow got too unweildy.
[0]: https://github.com/fsprojects/TensorFlow.FSharp
[1]: https://github.com/fsprojects/TensorFlow.FSharp#live-checkin...
If you're talking about matrix shape compatibility (matching up rows from one with columns from another) I'm hopeful about const generics here: https://github.com/rust-lang/rfcs/blob/master/text/2000-cons...
"Java / C# / Scala (and other OOP languages with pervasive dynamic dispatch): These languages share most of the static analysis problems as Python: their primary abstraction features (classes and interfaces) are built on highly dynamic constructs, which means that static analysis of Tensor operations depends on "best effort" techniques like alias analysis and class hierarchy analysis. Further, because they are pervasively reference-based, it is difficult to reliably disambiguate pointer aliases."
Do you have any insider knowledge that Chris Lattner had the unilateral power to choose Swift for this project? I would imagine with the importance of TensorFlow at Google, the decision to go in this direction had to be agreed on by a number of people.
> The learning curve of Rust should not be relevant here, compared to Swift, which is also full of idiosyncrasies. Swift and Rust both have a large learning curve for someone coming from Python.
How exactly would Rust-Python interoperability work? Swift for Tensor Flow allows any python library to be called like a native library in Swift. Could you do that in Rust?
Yes, and companies are even doing it in production. Sentry probably being the best well known.
If you like the pain of using a non supported language without all the XCode, UIBuilder, CoreData, Instruments, Metal Shaders debugging,... goodies then yes.