Hacker News Reader

Comment by NickHoff | Hacker News Reader

Maybe I'll give TF another try, but right now I'm really liking PyTorch. With TensorFlow I always felt like my models were buried deep in the machine and it was very hard to inspect and change them, and if I wanted to do something non-standard (which for me is most of the time) it was difficult even with Keras. With PyTorch though, I connect things however how I want, write whatever training logic I want, and I feel like my model is right in my hands. It's great for research and proofs-of-concept. Maybe for production too.

View on HN

TF's deprecation velocity was way too high for my taste. Things we wrote would stop working randomly with their updates. I feel very similar to you about the models being "buried too deep" in their (ever-changing) machine. I much preferred how easy it was to hack Caffe V1 (once you got past the funky names, etc).

These days, I really like mxnet. Torch was a disaster, but Pytorch is much better. It's not bad in production, definitely my #2.

> TF's deprecation velocity

That's Google on a nutshell. In fact, they may drop TF altogether next month. You never know ...

Test

I am curious, what do you like about mxnet?

Can't explain it but for some reason Tensorflow never felt "right" to me, even work keras.

Pytorch on the other hand feels so much more natural...

Maybe I'm not up to speed with the latest PyTorch, but to me Keras feels much more natural. In Keras if you want to define a deep learning network, then you just do that, you specify the first layer, the second layer, etc, then you calibrate over some test and validation samples, using a certain flavor of gradient descent, for a given loss function. In PyTorch, you have to define a class, with a constructor, some method called "forward", I don't know, maybe if I follow an example to the end I get the hang of it. My problem is that I don't want to write object oriented programming, I want to do machine learning. Keras doesn't force me to know what a class it, or what it means to inherit from nn.Module, or that a constructor in python needs to contain 'self' as a variable. PyTorch, at least the exmples I saw online want me to do just that, and that's a turnoff.

On the other hand, in Keras I can't (easily) change the architecture of a learner after I defined it. I can't prune some nodes and split others, maybe that's easy in PyTorch. If that's the case, I'll take a second look. Until then, when I have some time, I'm really tempted to invest some time in MXNet, as the book "Dive into Deep Learning" appears to be quite good.

in Keras I can't (easily) change the architecture of a learner after I defined it

I'm not sure what you mean here, because only PT lets you change architecture after you define it, while TF/Keras uses a static precompiled graph. Now that's changing with eager mode, but that used to be the main advantage of PT.

That was what I heard too, that in PyTorch you could adaptively build your graph. In Keras you could in principle emulate this in a brute-force way: you define the graph, train it, saving the weights, analyze them, decide what nodes to remove and what nodes to add, and then rebuild the graphs from scratch and use the adjusted weights from the previous round. I say in principle, personally I never did it. Some people hint that in PT this is a breeze, I'd be curious to see some example. If you have any links, that would be much, much appreciated.

Again, not sure I understand your example. What do you want to do?

In a large NN a lot of nodes end up being useless. You can remove them without degrading the performance of the NN.

To be more concrete, here's a link [1] to Google's neural network playground. I built a network with 5 layers and 37 hidden nodes. It trains quite well, but the last layer has 2 nodes that contribute with very little weight to the final output. The app allows you to change their weight (you click on the corresponding line and edit). If you change the weight to zero (effectively dropping the node), the classifier, if anything, gets better. My guess is that you can easily remove about half of the nodes. Conversely, if you look at the nodes with the highest weights out, you can in principle clone them and halve the weight out both for the original and for the clone. With this configuration, the network output is exactly the same, but if you continue training, it allows more flexibility, as the original and the clone are allowed to diverge.

This type of operations are not possible in Keras. Are they in PyTorch? If not, then what type of dynamic graphs are possible? What can one do with PyTorch that one can't do with Keras?

[1] https://playground.tensorflow.org/#activation=relu&regulariz...

Your first example is commonly referred to as network pruning, and is typically used to compress a model (the nodes are still there, but a sparse weights network can be stored in compressed form). It's also possible to remove nodes themselves, rather than individual weights. This is typically done on a filter level (for convnets), so that entire filters are removed.

The second example (cloning the nodes) is typically performed to improve network robustness (by avoiding important nodes a single point of failure).

To do either one during training you need dynamic graphs, so either PyTorch, or TF eager mode. Here's one filter pruning implementation: https://github.com/jacobgil/pytorch-pruning

I just hope they don't screw it up in the process of integrating PT with Caffe2.

Reply on news.ycombinator.com