Hacker News Reader

Comment by high_derivative | Hacker News Reader

S4TF from my (external) perspective was always more a (very cool!) programming language research project than anything else, at least since Lattner left.

I would personally assume the shutdown was due to a combination of reasons:

- There simply being no good reason for Python users to ever move to Swift. There is no big painpoint being solved for the broad ML user community

- Organisational momentum lost with Lattner leaving

- General disarray of TensorFlow as a project, the parallel rise of Jax from/within prominent Google orgs, it being the cool new thing

View on HN

- There simply being no good reason for Python users to ever move to Swift. There is no big painpoint being solved for the broad ML user community

As a swift and python user, I would have been really happy to be able to use swift for ML applications. Having a half way decent type system solves so many problems. But while I can see that from my vantage point, I know for a vast majority of the ML community python is "good enough" and there would be quite a lot of inertia to overcome in that regard which is probably not realistic.

I'm not sure if the Swift type system is half way decent in terms of machine learning applications. At least not without a lot of work. The data tends to be in specialized in-memory formats that tend to be opaque to the type system. The actual matrix operations involve dimension parameters that are likewise opaque to most type systems. So you still need to run code for the actual tensor graph compilation to validate if everything is correct. Python in notebooks does that instantly, Swift I suspect has a much longer delay.

I don't see any reason the swift type system could not accommodate the types required for ML. I did a lot of work with computer graphics in Swift, which is all linear algebra also, and the type system was a huge benefit.

Also I don't see why Swift would perform worse than python in any case. If python is fast, it's because it's wrapping a C++ library. There's no reason Swift could not wrap the same library, but on top Swift could communicate better type information back to the user.

In addition to what the sibling said, a big part of the S4TF project was proposing changes to Swift's type system. I'd also point out that there are not one, not two, but three type checkers looking into tensor typing for Python. What's more, Guido is on that committee!

Which is sort of my point. If you have to rewrite or expand significant parts of the the type system anyway is there any advantage over just doing the same work for Python?

There's many other advantages of Swift over Python for ML (concurrency and performance) but I just don't see the type system as one. At least not anymore.

I mean yes, because a) there's less to change, b) you still have nice features like protocols and c) Python's type system is pretty unsound. My point was that saying ML is too complex for type systems is really a cop out, and that has been borne out by how many languages have started experimenting with more complex types for this exact purpose.

> Having a half way decent type system solves so many problems

What problems does it solve for you? When building ML applications I mean. Because that's what S4TF was supposed to be.

ML researchers produce models that are used by ML applications for a particular purpose. The code to train the model _could_ be statically typed, sure, but I really don't see what the improvement would be. It would be more verbose, less readable, have significantly more noise. Just try reading some TensorFlow C++ code equivalents of their Python API.

From the "WhySwiftForTensorFlow.md" document (https://github.com/tensorflow/swift/blob/main/docs/WhySwiftF...):

.. static types: * can catch bugs at compile time instead of runtime. People get very frustrated when a silly type error brings down a long training run hours into it. * directly improve tooling experience like code completion/intellisense, jump to definition, etc. * make it easy to know what operations are Tensor operations.

> can catch bugs at compile time instead of runtime.

OK, but what bugs? For most use cases of TensorFlow that I've seen, if there is a type error somewhere, you will catch it long before it hits production, and if you only catch it 100hr into training, your training code is either non-deterministic or there's some other issue that is not type related. Unless we're talking about dependent types a-la Idris, I don't see how Swift's type system can help catch these kinds of bugs.

> directly improve tooling experience like code completion/intellisense

Sure, but type-hints provide the same improved experience with language servers like PyLance [0]. Not a reason to jump to a new language.

> make it easy to know what operations are Tensor operations.

Easy. Anything that starts with "tf." is a tensor operation.

Yes, I'm being a bit trite, but I was never convinced by the premise of S4TF, because I've never heard an ML researcher or engineer say "static typing will fix my problems".

[0]: https://devblogs.microsoft.com/python/announcing-pylance-fas...

My guess is that's because ML researchers typically have not spent much time with statically typed languages.

I was also once convinced that static typing was not so valuable - when I was working a lot with js, python and ruby, but the more time I spend with static typing the more I like it.

There is an "activation energy" to overcome with static typing: when you first start it feels like a tedious burden to have to add type annotations, when your python code ran just fine without them. But in my experience, with a little bit of practice the type system starts to feel completely effortless, and it saves you from all kinds of strange runtime issues you run into when nobody is checking your types.

Also type systems encode a lot of intention into the code. If I look at a function signature in Swift, I know exactly what has to be passed in, and exactly what will come out. In python, I have to rely on documentation, or even maybe I have to dig into the source to see what this function expects.

In my experience, the real difference is seen when you come back to a weakly-typed language after working with types for an extended period. Returning to Python after spending time with Rust or Swift feels like walking down the street with one of my senses missing. There are whole categories of dangers I now have to watch out for which the compiler would have caught for me automatically. It is not pleasant to go back.

I've done a lot of statically typed code and have gone to conference where Haskell was on the keynote. I've done ML work in both dynamically and statically typed languages. For the ML work I do I don't honestly see the advantage. Most of the complexity is in the matrix and data operations which existing type systems don't check. And the type annotations and IDE support is good enough to catch most of the other boring errors. In my experience most deep learning ML models are also not particularly complicated code from a code perspective.

I am totally with you on statically typed vs. dynamically typed languages _in general_. I'm not denying that there are major advantages to using one for most kinds of programs. But I just don't see a benefit for ML.

You can easily find out if you like it or not, you can use the C++ API for TF/PyTorch and you'll have a statically typed program running your model for you. Sure, it's not Haskell, but C++ is statically typed, and although it's not the most popular or loved language on HN, you can use it to get stuff done.

And with C++20 relatively pleasing to use.

Reply on news.ycombinator.com