Swift for TensorFlow | Hacker News Reader

Swift for TensorFlow

tensorflow.org

206 pointsgok52 comments

They want to build up the language compiler itself to better support machine learning. That requires a language with enough type information to support sophisticated analysis. It should be built on LLVM. It needs an existing ecosystem of tools and libraries. And having a better performing language than Python is likely a win.

Swift fits the requirements. As Chris Lattner is driving this, no one could ask that he choose something else.

But Rust also would have been a plausible choice. As the Rust team is very interested in applying the language to exciting new use-cases, it's a bit of a loss for it to miss out on collaboration here. Perhaps this could inspire similar work on the Rust side; many of the concepts would likely transfer straightforwardly.

The principle reason Python flourishes is because it's highly expressive and readable. (There are things it misses, such as sensible lambda, any reasonable multi-threading model, etc.)

I can't see Rust competing on similar merits. Swift however (like Kotlin to which it is extremely similar) seem to be in the right sweet spot in terms of languages designed for usability, with a lot of useful f.p. constructs.

It depends on if all your writing cutom functionality. If all you have is one big main function, rust is pretty readable. Function definitions and structs are where things get a little messier.

I can't agree - there's too much going on in Rust - most people using Python want something approaching Matlab or R for ease of use.

I'm not saying Rust doesn't have it's sweet spot - but I can't see that as a general purpose language. (Likewise Haskell and Scala - that's just my opinion though :))

For better and worse, I wouldn't say anything so absolute about Scala. Scala contains multitudes ;)

You can write Scala as Java or Haskell or anything in between.

>And having a better performing language than Python is likely a win

I was under the impression that most Python libraries for machine learning (numpy, scipy, tensorflow, etc) are running C code under the hood. How much is Python's performance really holding it back?

>They want to build up the language compiler itself to better support machine learning. That requires a language with enough type information to support sophisticated analysis. It should be built on LLVM. It needs an existing ecosystem of tools and libraries. And having a better performing language than Python is likely a win.

You literally just described Julia. The Julia core developers are all part of Julia Computing (which contracts for ML work) or are somehow associated with data science and machine learning. For this reason there's a lot of on tooling for building computational graphs / performing AD, compiling for TPUs, etc. It's built on LLVM. It has loads of packages for optimization, machine learning (like 4 NN libraries, a few different ways to work with GPUs), data science, etc. And type-stable Julia code is as fast as C. So it's interesting that this gets some kind of noteworthy announcement when it's already been done and already been done well.

Watching the talk, I thought the problem statement was exactly right. Of course our answer is a bit different, but I'm glad people are coming around to seeing ML as a programming language problem. We talked about this a bit in a blog post a few months ago: https://julialang.org/blog/2017/12/ml&pl.

The talk is here: https://www.youtube.com/watch?v=gplTc2F5Wvk&feature=youtu.be...

Also see Chris's introductory post for the project: https://groups.google.com/a/tensorflow.org/forum/#!topic/swi...

Is it me or did the audience had absolutely no clue what that presentation was about ?

It seems like Lattner's part of the show is gone from that video now, though it had been there yesterday when I clicked the above link (or maybe I'm just doin' it wrong now?). Anyway, looks like Lattner's part is here now: https://www.youtube.com/watch?v=Yze693W4MaU.

I like both Swift and Tensorflow. But how is that going to work?

As far as I understand MacOS has no official Nvidia support (=> no Cuda), which is (at least) advised if you want to use a GPU for computing. Using OpenCL instead of CUDA would require building Tenfowlow from source. The OpenCL support is not as mature as with CUDA, so I imagine you could run into unexpected (performance) problems.

On windows, you have an excelent CUDA but lackluster Swift support.

Will they add OpenCL as a "first class backend" for Tensorflow or rather expect a "first class" support of Swift on Linux and Windows? Otherwise who is going to use it?

People rarely train ML models on macOS for the reason you mentioned. Most machine learning work happens on Linux, so this should work well there.

TensorFlow supports a standalone server mode where it receives computation graphs and executes them. This is nice because then you can remotely execute on any accelerator (Cloud TPU, multi-worker multi-GPU) from your laptop.

In their demo, they did exactly that with a Cloud TPU: it connected to a TensorFlow server that executed the machine learning training part of the program.

I agree, I just had in mind that Apple just now added/announced support for external GPUs. Besides Image+Video edditing, I though general computing tasks is a use case they had in mind. It's not like Gaming is big on MacOS.

>TensorFlow supports a standalone server mode where it receives computation graphs and executes them. This is nice because then you can remotely execute on any accelerator (Cloud TPU, multi-worker multi-GPU) from your laptop.

Where can I find more documentation on this? I’ve been looking for something exactly like this.

CUDA has been available on macOS since 2010:

https://www.nvidia.com/object/mac-driver-archive.html

I currently have an NVIDIA 980 GTX in my 2010 Mac Pro and it works great (no additional drivers needed but NVIDIA also have downloadable drivers).

Coincidentally, macOS added support today for external graphics cards so you could add a set of external cards to any Mac with Thunderbolt 3 ports.

In any case, Swift on macOS has built-in support for Metal-compute (which works on any card) and CoreML which can be integrated with Tensorflow.

https://developer.apple.com/metal/

https://developer.apple.com/documentation/coreml

Training on an Apple device (read: overpriced device) is idiotic, nobody does that.

Shouldn't it be called "TesnsorFlow for Swift"? Like in "MySQL driver for Java", not "Java for MySQL driver".

I'm not native English speaker, so I wonder...

I think they're making that grammatical choice deliberately because they're modifying the language to better support TensorFlow integration, as opposed to just providing bindings for it.

It comes off rather arrogant, doesn't it? Not unlike Windows Subsystems for Linux, which is named exactly backwards.

It depends on what you're focussing on. If I've got a MySQL backend and using different languages, I may be looking for a MySQL driver for Java, or a MySQL driver for PHP. If I've got PHP backend and am moving from MySQL to PostgreSQL, I'm going to be looking for a PHP driver for PostgreSQL. Since it's the TensorFlow community, I trust they named it like they wanted. I can see it either way.

It's using Swift to create a TensorFlow (or XLA) graph which can be evaluated on a TPU. From that point of view, the name makes sense.

Curious why not Go. Google has much more control over Go than Swift.

This is actually surprising. Are the machine learning engineers at Google (besides Chris Lattner) admitting that Swift is a better language than Go? :) I would be surprised to find out if he was the only one involved in the decision

Looks like some decisions at Google are made somehow chaotically, reminds me TypeScript Angular and Dart Angular.

Yes, they're admitting that Swift is a way batter language for data science work.

To be perfectly frank, most languages probably are.

So, i'm the closest thing to Chris's counterpart in language land (I own all of Google's production and cloud programming languages, including Go). We also go way back to nearly the beginning of llvm days. Chris and I certainly talked about this.

I guess i don't really understand why anyone would think this is surprising. Like, i understand the knee jerk reaction, but I guess i don't understand why i would try to force him to do something else.

Chris is a high level engineer who decided he wanted to prototype and build a system in a language he's familiar with and feels is a good match for his problem space. People generally agree it's a good match for his problem space and his familiarity/etc makes him more effective in getting things done there. This doesn't mean anything else is a bad match (general purpose languages are often good at multiple use cases, hence the reason they are considered general purpose)

Why would I try to stop him and force him to use another language?

What tangible benefit would such a thing bring Google?

It's not like Google is a bunch of language zealots who think everything must be written in language X - it already supports a number of production languages, because different languages tend to have different tradeoffs for different use cases.

Outside of that framing, i have trouble thinking of a sane goal that would be served by trying to force him to do it in a different language.

But if i'm missing something, i'd love to know!

Well initially preferring to use a homegrown solution (like a programming language) isn’t necessarily a sign of zealotry. Dogfooding is good :)

Go just seems like a much worse match for this problem than Swift. Wrong type system, wrong compiler architecture, wrong runtime design, wrong language evolution strategy. This is in so many ways a great match for Swift.

"Well initially preferring to use a homegrown solution (like a programming language) isn’t necessarily a sign of zealotry. Dogfooding is good :)"

Sure it's not necessarily.

Building on top of something that is yours, simply because it is yours, is in fact a form of zealotry.

It is fairly unrelated (IMHO) to building on top of something that is yours so you can make sure you experience what your customers will experience.

IE Building your own ML on top of your Java thing because you want your customers to build ML on top of your Java thing would make sense. Building your own ML on top of your Java thing just because it's yours does not by itself make sense. (IE it has to provide some particular value over not building it yourself, etc, otherwise it's pretty clear zealotry)

Regardless of whether it is zealotry or not, Google certainly has a habit of developing everything (software side) of its own.

When Google started doing it, there weren't a lot of good options to solve the problems it has. As that has changed, so has Google (but like any large company, it changes slowly)

Also why not kotlin/kotlin-native?

Because it's not about control?

I don't understand why anybody would ever use Go outside of backend development. It's just language lacking lots of features without giving any benefits.

Google is not monolithic. The Go designers have an emphasis on keeping the language simple (or lackluster in my opinion), and they would never agree to such a thing if they are still in power.

As DannyB mentioned in his post, Google is very technology driven, not usually driven by zealotry. Using Swift for this purpose has specific technical motivations that will be spelled out in the white papers we are publishing in April.

-Chris

I didn't mean to blame anybody in zealotry, it is purely technical considerations: it is much easier to maintain and develop codebase when you have more organizational control and can push changes to mainstream, comparing to forked project, when your changes may not be accepted to main branch.

Very interesting. They are baking auto differentiation/gradient computation directly into the compiler ? How will this be work, since you invoke TF and TF computes gradients (at least on the loss functions) using Python/C++ from first principles.

It'll be nice to use the Xcode build system, but given that Xcode isn't open source, I wonder how this will play out. Is there a way to use Bazel natively with Swift ?

Exciting times though !

> Is there a way to use Bazel natively with Swift?

Yes. https://github.com/bazelbuild/rules_apple/blob/master/doc/ru...

The key thing here is that the Swift compiler is computing gradients using SCT techniques, not TensorFlow. This has a number of advantages that we'll go into in the white paper we are going to publish in April, so I'd prefer to wait until that spells it all out rather than go into detail here.

All this works no matter how you choose to build your code: Xcode, Bazel, SwiftPM, makefiles, whatever. Also, since ML scripts are frequently small, a simple #! script like we showed in the demo is perfectly reasonable for many things.

-Chris

Great! Thanks Chris. Look forward to the white paper.

It's not clear what this is.

Is it related to Apple's Swift language?

Yes, it’s language level support for machine learning (specifically TensorFlow) in the Swift programming language.

Lattner’s post makes this clear:

https://groups.google.com/a/tensorflow.org/forum/#!topic/swi...

Am I the only one seeing this is a deliberate April's fools joke? My bet: The project is real, but "Go for Tensorflow" instead of swift.

Quoth: "Is it me or did the audience had absolutely no clue what that presentation was about?" One day too early but it couldn't emphasize "April" more often in the article.

Quite a genius take on Apples openness and and wannabe-AI-ness.

I think you're reading too much into it. It was announced at the Developer Summit.[0][1] Seems like a long ways to go for an April Fool's joke.

[0]: https://medium.com/tensorflow/highlights-from-tensorflow-dev...

[1]: https://www.youtube.com/watch?v=Yze693W4MaU

Will you buy me a beer if they rename it into "Go for Tensorflow" tomorrow evening?

@Google: Hire me or ask nicely and I'll remove this so that your genius joke is not spoiled.

I am willing to bet 10:1 agaist this being a joke, that is, I give you $20 if this is a joke, you give me $2 if not. Can pay with cryptocurrency. Mail me (address on HN profile) if you are interested.

This isn’t a joke.

Definitely not a joke.

Lattner works for Google Brain now, not Apple.

Reply on news.ycombinator.com