Swift: Google’s Bet on Differentiable Programming

The greatest possible barrier to adoption.

And speaking of Lisp - wasn't symbolic differentiation a fairly common thing in Lisp? (basically as a neat example of what you can do once your code is easy to manipulate as data).

In essence there are cases outside the well developed uses (CNN, LSTM etc.) such as Neural ODEs where you need to mix different tools (ODE solvers and neural networks) and the ability to do Differentiable Programming is helpful otherwise it is harder the get gradients.

The way I can see it being useful is that it helps speed up development work so we can explore more architectures, again Neural ODEs being a great example.

If we already have useful phrases like "embedded programming", "numerical programming", "systems programming", or "CRUD programming", etc, I'm not seeing the pretentiousness of "differentiable programming". If you program embedded chips, we often call it "embedded programming"; likewise, if you write programs where differentials are a 1st-class syntax concept, I'm not seeing the harm in calling it "differentiable programming" -- because that basically describes the specialization.

>the quotes are ridiculous ("Deep Learning est mort. Vive Differentiable Programming.";

Fyi, it's a joking type of rhetorical technique called a "snowclone". Previous comment about a similar phrase: https://news.ycombinator.com/item?id=11455219

There is definitely some need for an EDSL of some sort, but I think a general method is pretty useless. Being able to arbitrarily come up with automatic jacobians for a function isn't really language specific, and usually much better results are obtained using manually calculated jacobians. By starting from scratch you lose all the language theory poured into all the pre-existing languages.

I'm sure there'll be a nice haskell version that works in a much simpler manner. Here's a good start: https://github.com/hasktorch/hasktorch/blob/master/examples/...

I think it's pretty trivial to generalize and extend it beyond multilinear functions.

Swift was railroaded in Google by Chris Lattner, who has since left Google and S4TF is on death watch. No one is really using it and it hasn't delivered anything useful in 2.5 years

Google is still taking themselves very seriously while everyone else is starting to get bored.

The problem with being 25 is that you have about 8 years ahead of you before you figure out how full of shit everyone is in their twenties, and maybe another 8 before you figure out that everyone is full of shit and stop worrying quite so much about it.

This is not really about swift. Swift seems to have been chosen because the creator was there when they picked the language, even though he left.

    #include <numeric>
    #include <iostream>
    #include <chrono>
    #include <vector>
    
    int main(int argc, char** argv) {
      for (int i = 0; i < 15; i++) {
        std::vector<int> result;
        auto start = std::chrono::system_clock::now();
        for (int j = 0; j < 3000; j++) {
          result.push_back(i);
        }
        auto sum = std::accumulate(result.begin(), result.end(), 0);
        auto end = std::chrono::system_clock::now();
        std::cout << (end - start).count() << " " << sum << std::endl;
      }
    }

It's a problem.

C++ and javascript are languages of professional software engineers, of which there are many many more languages with various pros and cons.

Python has been the defacto standard in scientific/data/academic programming for decades. The only other language you could say rivals it would be MATLAB which is even more simplistic.

My point is that simplicity and clarity matters to people who don't care that much about programming and are completely unfocused on it, they are just using it do get data for unrelated research.

'if __name__ == "__main__"' is not in the example code nor is it a required part of a python program so not really sure what your point is here.

Yet swift is open source, and Apple and the community can fork it if they so choose. This is great news for me personally as an iOS developer and an ML noob who doesn't want to write Python. I can't comment on Julia because I have no experience with it, but I applaud the efforts to build the Swift ecosystem to challenge Python.

I think a lot of the criticisms so far are that it's early days for Swift in ML, and that's one point the author is emphasizing.

Does anyone force anything on Google? This seems to express little confidence in the competence of Google and their people. Perhaps Google chose Swift and brought Lattner in for his obvious expertise.

While other people will do the sensible thing and learn Rust. Because it runs circles around Swift, it offers many paradigms and can be used in almost any industry, operating system and product, not just developing apps for Apple's ecosystem.

Swift will take over the world when Apple will take over the world which is safe to assume it will never happen.

I am not saying at all that is bad to learn Swift and use Swift, but have correct expectations about it.

Some ideas in the language or parts of its core concepts are really good. First class optionals and sum types, keyword arguments, etc., I liked all of those.

Unfortunately, by and large, Swift is lipstick put on a pig. I have never used any other language that regularly gave me type errors that were WRONG. Nowhere else have I seen the error message "expression is too complex to be type-checked", especially when all you do is concatenating a bunch of strings. No other mainstream language has such shoddy Linux support (it has an official binary that works on Ubuntu... but not even a .deb package; parts of Foundation are unimplemented on Linux, others behave differently than on macOS; the installation breaks system paths, making it effectively impossible to install Python afterwards[1]). Not to mention, Swift claims to be memory-safe but this all flies out of the window once you're in a multithreaded environment (for example, lazy variables are not threadsafe).

In addition, I regularly visited the Swift forums. The community is totally deluded and instead of facing the real problems of the language (and the tooling), it continues to bikeshed minor syntactic "improvements" (if they even are improvements) just so the codes reads "more beautifully", for whatever that is supposed to mean.

But the worst thing is how the community, including your post, thinks Swift (and Apple) is this godsend, the language to end all language wars, and that everyone will eventually adopt it en masse. Even if Swift were a good language, that would be ridiculous. There was even a thread on that forum called "crowdfunding world domination". It has since become awfully quiet...

[1]: https://bugs.swift.org/browse/SR-10344

Even though the article talks about the "why not Julia", which is the highest comment at time of typing ... Choosing a cross-compatible language would have kept more people interested in the long-run. Why should I as a Windows user want to learn a language just to use it with Tensorflow; when I'm not sure if such language support and other tooling will generally come to Windows?

I know this is a hot take but... I doubt Google has the capability to be frank. They created Dart and Go (a.k.a generics ain't necessary). They created Tensorflow 1 which is totally different from Tensorflow 2.

Swift may not be the best but Swift is starting to become such a large part of Apple so it will have backing no matter the internal politics.

The language is not where the battle will be, it will be the tooling.

https://swiftforwindows.github.io/

Using `..<` and `...` is pretty simple to figure out from context. The former produces an exclusively-bounded interval on the right, while the latter is an inclusive interval. This is functionality that more languages could stand to adopt, in my opinion.

I agree that the names themselves are not very transparent. However, they become less opaque as you learn the Swift ecosystem. Admittedly, this makes them not as immediately user-friendly as Python's simple names, but it's not as though they're some gigantic obstacle that's impossible to overcome.

Personally, I like Swift a lot (even though I never use it). It has a syntax that has improved on languages like Java and Python, it's generally fast, it's statically typed, and it has a large community. The fact that implicit nullable types are discouraged directly by the syntax is phenomenal, and the way Swift embraces a lot of functional programming capabilities is also great. If it weren't so tied to Apple hardware, I would likely recommend it as a first language for a lot of people. (I know that it runs on non-Apple hardware, but my understanding is that support has been somewhat limited in that regard, though it's getting better.)

I find this notation slightly clearer than the python version. It took me some time to remember whether range(15) includes 15 or not.

A current and more readable way of expressing this would be

  let start = Date().timeIntervalSinceReferenceDate

  let start = Date()

It's more important that the language can accurately and succinctly represent the mental model for the task at hand, and the whole point of this article is that Swift can offer a syntax that is _more_ aligned with the domain of ML while offering superior performance and unlocking fast development of the primitives.

Python's no saint when it comes to time stuff either. I had some code using time.strptime() to parse time strings. It worked fine. Then I needed to handle data from a source that included time zone information (e.g., "+0800") on the strings. I added '%z' to the format string in the correct place--and strptime() ignored it.

Turns out that if you want time zones to work, you need the strptime() in datetime, not the one in time.

BTW, there is both a time.time() and a datetime.time(), so even that line that needs no explanation might still cause some confusion.

Is it? Does that range start at 0 or 1 or some other value? Does it include 15 or exclude it?

> - start = time.time(): doesnt need any explanation

Doesn't it? Is that UTC or local time? Or maybe it's CPU ticks? Or maybe the time since the start of the program?

You've basically just demonstrated the assumptions you're used to, not any kind of objective evaluation of the code's understandability.

So, is that evaluated when that statement is run, when the value is first read (lazy evaluation, as in Haskell), or every time ‘start’ gets read. For example, Scala has both

  val start = time.time()

  lazy val start = time.time()

  def start = time.time()

CF clearly is a prefix for a library. I'll take an educated guess it means Core Foundation? Pretty common pattern of naming things, with +/- to be certain. And once you've seen it, it is just there, and you know precisely what it means. So 10 minutes of your life to learn what CFxxx() means.

Let. I like lets. Some don't. Surely we can coexist?

x..y is also not unique to Swift. It has a nice mathematical look to it, is more concise.

Btw, is that 'range' in Python inclusive or exclusive? It isn't clear from the notation. Must I read the language spec to figure that out? .. /g

Furthermore, it's obvious what `time()` does in this context, but if I was writing this code I would _absolutely_ have to look up the correct time function to use for timing a function.

First of all if you are doing "for i in range(N)" then you are already doing it wrong, for ML and data analytics you should be using NumPy "np.arange()", Numpy arange doesnt even run in "python" it's implemented in C. So it may even be faster than swift '..<' . Let me know when you can use swift with spark.

time.time() (as well as datetime.datetime.now() and other stuff like that) always looked extremely ugly to me. I would feel better writing CFAbsoluteTimeGetCurrent() - it seems more tidy and making much more sense once you calm down and actually read that.

"range(15)" vs "0..<15" could go either way.

"let" vs "var" in Swift is indeed something that adds verbosity relative to Python, and adds some cognitive load with the benefit of better correctness checking from the compiler. Very much a static vs dynamic typing thing. That's where you'll see the real friction in Swift adoption for developers less invested in getting the best possible performance.

I feel there is an impedance mismatch here.

Note that to be useful you need backward differentiation (a function with one output but several outputs) and compatibility with tests and loops.

See the Zygote.jl library (Julia) for a working example in a different language (their whitepaper is a great starting point if you want to implement something like that).

That question can be generalized to "since <Turing Complete language> can do <anything-we-want-via-library/function/macro/codegen/customcode/extension/whatever>, can't we just do that instead of modifying the core language?"

Well yes, but the advantage of having 1st-class language syntax includes:

+ more helpful compiler error messages because it has intelligence of the semantics

+ more helpful IDE because the UI has intelligence about the intentions of the programmer

+ interoperability because programmers are all using the same syntax instead of Programmer Bob using his custom macro-dialect and Progammer Alice using her idiosyncratic macro-dialect.

While C++ might not be the ideal language, choosing one that only works properly in Apple platforms, dismissing the complaints of lack of any solid roadmap in platforms that researchers actually use and hand waving any usability problems to just use notebooks in Google Compute cloud is not how a language gains adoption beyond its initial niche.

Regarding the tensorflow dev summit, it was supposed to be 2 days long initially, with the S4TF talk taking place on the second day. One week before the summit, the whole day 2 got scraped due to covid19 though. So the intention was there at least.

• I fail to see how differentiable programming (the idea of expressing the desired computation in terms of differentiable objective functions) is any less of a "paradigm" than logic programming (the idea of expressing the desired computation in terms of logical predicates).

• Depending on the expressiveness of your programming language, every paradigm can seem like it's "a great use case of a library, but hardly worthy of being called a new programming paradigm."

It's also not quite declarative programming either. In the same way that you build ordinary executable programs out of smaller executable parts. You build differentiable programs of smaller differentiable parts. You aren't declaring what outcome you want, you are simply restricting what you are building your program out of so that it has properties that allow you to interpret the program differently from how it will run "normally".

Declarative programming is more or less orthogonal to differentiable programming. You're right that declarative as a paradigm leaves the implementation details up to the "compiler," and so you can have arbitrary implementations created in response to one declared specification. And often what that means is that under the hood it's possible that the compiler can be tweaked to output better results. But the thing is that those compiler changes and optimizations, are just that: arbitrary. You can't "know" or "prove" anything about them, and that means it mostly requires human creativity and intelligence to make progress.

That is fine and all, but differentiable programming is asking/answering the question: Ok, but what if we COULD prove something here? What if the units of computation could be guaranteed to have certain mathematical properties that make them isomorphic to other formalisms?

And why do we care about that?

Well, in math what happens is that some people will start with their favorite formalism and prove a bunch of stuff about them, and figure out how to do interesting calculations or transformations on them. Like "Ah, well if you have a piece of data that conforms strictly to the following limitations, then from this alone we can calculate this very interesting answer to a very interesting question."

But a lot of the time those mathematical paradigms can't talk to each other -- in programming terms, their apis just aren't compatible. Like raster vs vector images. Both image storage/display paradigms are "about" the same thing, but a lot of the operations you can do on one don't even make sense to try on the other, and our ability to translate back and forth between them is a little wonky. Math formalisms are a bit like that, a lot of the time.

So it's very interesting in math world when someone proves that a formalism in one paradigm can be transformed perfectly into a formalism from another paradigm. All of a sudden all the operations available in either paradigm become available in both paradigms because you can always just take your data, transform it, do the operation, then transform the answer back.

(Side note: this is why some people are excited about Category Theory: it's like a mathematical rosetta stone. Ie. A lot of things can be translated into and out of category theory, and in turn all those things that were previously in separate magisteria are interchangable.)

Ok so, back to differentiable programming. If you suddenly have a way to conceive of your program / unit of computation as a differentiable function, then right off the bat you get access to all the tools ever created for calculus. The optimization thing where you find the gradient of the program and follow it toward some target is just one of the things. You also get a huge suite of tools that let you enter the world of provably correct programs, for example.

You also get access to all the tools of all math that can translated to and from calculus, which is... a lot of them. I wish I knew more about math so I could rattle off the 100 ways that would help, but I can't, so instead I'll just say that I think it would be a game-changer for creating optimized, robust systems that work way, way better than our current tech.

You'd like to be able to write your functions using the language's standard function syntax, but have access to both the function and its differentiated form. You can achieve it with macros (in an ad-hoc way) or as a custom language feature (what's being done here, again kinda ad-hoc), or you can use an algebra+interpreter style but at the cost of having to use a less natural syntax (Haskell do notation or similar). The thing that I'd say is closest to the ideal solution is something like stage polymorphism (which is genuinely an exciting new paradigm in my book: it squares the circle of macros versus strong typing), where you can write a function definition in natural function syntax and have access both to the function itself and the AST of the function in a much richer form than what a macro gets (which can be interpreted to produce a differentiated version of the function).

Sure, there are some analogies, but it is stretching the definition a bit.

To be fair, he does answer the question a minute later.

doStuff({ foo: 3, bar: 7 })

I'm sick of this pointless and never-ending upgrade cycle. I started a new project this week, and it's in Clojure. I find it a much more productive language, and I'm confident I can run it on nearly any computer from the past 20 years.

(This is probably not my most productive comment ever, but I'm pretty frustrated right now with having chosen Swift in the past.)

You do need to use the Xcode-bundled toolchain when archiving for the App Store, but not for development or off–App Store releases. Even for the App Store, you could get away with just one computer on your team having the latest macOS and Xcode, or with using a separate build server instead of archiving on your workstation.

[0] https://swift.org/download/

Supplementals: http://taichi.graphics/wp-content/uploads/2019/10/taichi_lan...

IBM may have counter-balanced this.

What do other think is the reason for the slow backend adoption of Swift?

You can't use Swift on Windows. You can target practically target most operating systems with C#. C# is not only open source, but an international standard. Ecma (ECMA-334) and ISO (ISO/IEC 23270:2018)

Although both started as proprietary languages, both are open source now, both are developed mainly by one big company, the difference is that Microsoft is interested seeing C# on many platforms, while Apple doesn't have that interest.

I think Kotlin, Go, Rust have greater potential to rise in popularity than Swift.

I would've been excited about Swift if it weren't for these things.

From my understanding, the author is someone from outside Google who doesn't know anything about Google bets more than what is publicly available, and the author is biased toward the Swift language.

JAX makes more sense, as it uses Python, but more importantly, Google works at a lower level of the stack with TPU, MLIR and XLA. It doesn't really matter what language is on top of that stack.

Doesn't matter anyway, it's a google project, it will be abandoned before 2022

As the article points out, Google did consider it. IIRC it came down to Julia and Swift in the end. And, given Chris Lattner was leading the effort, there was only really going to be one answer. There's clearly some merit to that: they were expecting to make changes in the compiler (again, if I remember, for e.g. optimising GPU code). If you're going to change the compiler, it's pretty compelling to opt for the language that one of your team designed. And it's not clear (to me) what the implications of commits into the Julia master tree would have been.

That doesn't generate a community though. It's yet to be seen whether that will happen. It would take the level of resource that very few firms can afford to dedicate. Google is one of them: though its patchy record on committing to long term endeavours means it's definitely not a slam dunk. And Lattner leaving further detracts from that confidence.

I'll be interested to see what they come up with. Still think it's a pity they didn't choose Julia. But it's not my project so I don't get to choose.

2 - S4TF is only usable in Google Cloud notebooks

True, but I didn't encounter it in the wild yet. Usually Swift used in "Python mode" is fast enough.

The objective of the demo was not to see which language could sum up a bunch of numbers the fastest. You could keep optimizing that until you are left with just `print(<the resulting number>)`. The objective was to have a simple example of looping over an array a bunch of times. The only reason I ended up summing the numbers in the array and printing them was so that LLVM wouldn't optimize it away and be unfair towards python. I actually wrote it first in Python tbh.

    let clock = Clock.system
    let now = clock.thisInstant()

Because for mobile devices you don't want to ship LLVM. Plus it's easier to package binaries than scripts which usually require Docker to deal with dependencies in a sane way.

Needing shared memory paralellism is a weird reason now? Pretty much any parallel algorithm that's not embarassingly parallel is going to perform better with threads able to share memory than with message passing between processes.

Is it because it's easy to write?

Honestly I am curious as an outsider to the Python world.

For research this seem indeed like a bad fit.

(Side note, named functions in Swift are indeed also closures:

    let x = 10
    func f() {
        print(x)
    }
    f()    // Okay; prints 10 as you'd expect

)

But even so, Python is a little slow when you start thinking about threading. You'd be better off using Rust or Go or something rather than the half-baked support found in scripting languages.