Hacker News Reader

Comment by bitL | Hacker News Reader

Still a bit more verbose than Python. What would you gain by doing the same in Swift actually? If you have to type more code to do some experiments, and still have to import Python libraries for extra functions as they do in that example, what's the selling point? (beside it being cool)

View on HN

Here's some context: https://github.com/tensorflow/swift/blob/master/docs/WhySwif...

Some benefits vs Python are static types (catching more errors at compile-time vs hours into a training run), no GIL, don't have to drop to C++ for higher performance.

Some benefits vs some other (non-Python) languages are a shallow learning curve, small boilerplate, safety-by-default, and a growing community.

Obviously there are places where Python and others have advantages over Swift too, and several of those are called out in the paper.

Is it really likely that you'll catch a type exception hours into a training run? The way tf works it'll catch a shape exception before you even start training.

Its not likely, but I've had it happen to me where 2200 batches in it crashes because the graph has dynamic sizes and the input is not well formed. Static typing would not have helped.

But, static typing could help provide much more useful error messages than what you get from the Python code.

I don't have experience of a type exception hours into run, but for me, deploying code to the test environment is long, so the edit-run cycle is very long. If the compiler can catch the errors locally, then edit-(compile)-run is shortened.

This sort-of feedbacks on itself. I use estimators, so type errors occur far into a run, but not during training itself, and checkpoints mean very rapid recovery, so ...

I think I actually stopped caring so much because the recovery is so fast.

Big help for me was coding a little script that means I can just send a notification to my android from from cli and/or python, so I know when these things need attention.

I think the primary advantage of using this is that you get both the usability of writing imperative code (like PyTorch or TF Eager) and the benefits of having a computation graph (easier distributed training, using TPUs and stuff).

Static typing is nice.

But no GIL is the most important part. Multithreaded preprocessing data input to the network is almost a must these days, but that is hard to achieve in Python. Multiprocessing or that PyTorch magic alleviates the problem, but they are not as high performant as multithreading.

In Keras you can just use a Sequence-based generator, set a large number of workers, or even put in multi-GPU callback, and you can parallelize pre-processing as much as you like.

I doubt that Keras can break the limit of the CPython interpreter. Its parallelization is probably also done with multiple processes, in no way as efficient as multithreading.

I might take a look at Keras source just to know :) But still, if you do image augmentation on the fly, you are probably fine even with processes if your batch size is sufficiently large.

Reply on news.ycombinator.com