TensorFlow Model Analysis – A library for evaluating TensorFlow models
github.com
github.com
This notebook here shows some helpful cases: https://github.com/tensorflow/model-analysis/blob/master/exa...
I still don't see the point, but maybe someone needs this. To me, it would be a lot more useful to evaluate things like memory consumption and CPU/GPU bottlenecks (for each node in the graph).
This library is about bringing Apache Beam into play. Tensorflow is not really distributed (distributed tensorflow requires you to start all the servers and manage all the endpoints yourself). Right now, we use Spark for pre-processing pipelines and for launching both parallel experiments in TensorFlow and distributed training. Spark is a key part of TensorFlow for us (www.hops.io). Google want an open-source alternative (not just managed TF, which is distributed) that enables ML pipelines. This is it. Spark is fighting back, with Image pre-processing now part of Spark 2.3. We recommend Spark for pre-processing, spitting out .tfrecords, which then are used as training/test data.
When hyperparam search is this easy with PySpark, why wouldn't you do it?
def model_fn(learning_rate, dropout):
[TensorFlow Code here]
args_dict = {'learning_rate': [0.001, 0.005, 0.01], 'dropout': [0.5, 0.6]} experiment.launch(spark, model_fn, args_dict)
For distributed training, we favor Horovod. It requires minimal changes to code, and scales linearly on our DeepLearning11 servers: https://www.oreilly.com/ideas/distributed-tensorflow
Perhaps because I don't have hundreds of GPUs laying around? I'm a DL researcher fortunate enough to have a 4 Titan X workstation for my experiments. Most of my peers at other labs only have shared access to such workstations, and with the current trends in GPU pricing, that's not likely to change in the nearest future.
More importantly, the lack of compute power is rarely a bottleneck in my research. Reading papers, brainstorming new ideas, and writing/debugging code are the most time consuming parts, it's not like I'm sitting idle while waiting for my models to train.
Distributed TF is the standard in Google, will be the standard amongst everybody else within a couple of years.
By "everyone else" you mean a few dozen corporations, which can justify spending millions of dollars on hardware to speed up their mission critical ML research?
SoTA in Cifar-10 and ImageNet is now model-architecture search. This paper/blog was an eye-opener for me: https://blog.acolyer.org/2018/03/28/deep-learning-scaling-is...
Current estimates are that 75% of ethereum mining is done on 1080Tis. So, if that stops overnight - will they sell them or switch to some other bitcoin? My guess is that they will flood the market. Mostly they are under-volted when mining, so they won't burn out within a few weeks if you buy used.