Tensorflow 2.0: models migration and new design
pgaleone.eu
pgaleone.eu
For anyone interested in research in deep learning with a python-numpy like environment, I can only recommend to switch.
I can do everything in both pytorch and tensorflow and when I have to define really efficient input pipelines (tf.data is a great thing), parallelize and distribute the training and export a trained model to production... With tensorflow everything is easier.
Moreover, pytorch in 1.x will have static-graph too, exactly like Tensorflow.
Both frameworks are converging to something really similar. I don't see a reason to switch (right now)
For point 2: I don't see the problem. I use almost daily a wrapper (defined inside tf.contrib, that in version 2.x will go in core [I hope]) around the optimizer that in 2 lines allows me to distribute the training on multiple GPUs on the same machine
https://www.logicalclocks.com/goodbye-horovod-hello-tensorfl...
What actual product(ive) things?
To mitigate this complexity, PyTorch and Tensorflow are heavy weight machine learning frameworks that give your software a lot of structure as well as tooling to monitor the progress of training and debug your models as well as some deployment tooling.
Any neural net based component of software will likely be developed in one of these frameworks.
Or generative models: same workflow (python definition and train. Export and use in other languages).
I use Tensorflow basically everyday. I use pytorch only when I read a model implemented in this framework to reimplement it in Tensorflow (in order to use all the tools I developed to simplify the train to shipping to production phases)
I'm constantly surprised by how much attention it gets given how narrow the scope is. I guess a lot of people need to classify images.
> My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners.
This is not correct. TF is mainly used because it's designed for industrial-strength machine learning. It provides primitives with an eye to scale from the outset (because it's Google).
It's probably true that prototyping / research was not the main audience. That's exactly why Keras was adopted, as well as features like tf.eager, to abstract away the underlying computation graphs and make it easy for people to try different things.
Well-designed primitives / abstractions are important; Tensorflow does this well.
Not sure if he develops and maintains during company time, but wouldnt be surprised
Sounds like every Google developed technology. Angular, Go, Android, the list is infinite.
Sonnet is based on TF
The reason I bet on Tensorflow originally I admit was it was built by Google. But another really strong reason is it had a fully integrated toolchain from research to production. That toolchain has gotten much stronger with the continued investment in tf.data and tensorflow-serving. There was a binary alternative to CSVs (TFRecords). Also given some of our needs with some pretty esoteric components the tf community is actually quite deep.
I had only minor python experience before starting with tensorflow so I literally did not want a python-numpy environment. I am still sad that the world has converged on python as the language of choice but that ship seems to have sailed.
I will say tf.while is something I won't miss, as is tf.cond. I didn't find either really difficult, but it was more typing than was necessary. If there is something I really think should be invested in it is documentation. What should be in a SessionRunHook and what is the best way to write them? If you dig through some of google's repos that use TF estimators you find a lot of useful hooks as well as utility functions to help build them. But learning how to use them was more painful than it should have been.
All that said I am not a researcher. I would suspect that some of my workflow overlaps with researchers as I have found myself implementing a number of papers. I also need the ability to rapidly move a model from training to production.
We used to be a TensorFlow shop. Last year we started playing around with Pytorch, only for R&D projects at first... and it felt like a breath of fresh air. Whereas TensorFlow always felt like it was designed and engineered from day one for massive scalability and deployment flexibility (at the expense of easy/fast development), Pytorch felt instantly like it was designed for easy/fast development by AI researchers who need to experiment and iterate through models as quickly as possible with as little hassle as possible. Almost overnight, we stopped tinkering and experimenting with TensorFlow.
Earlier this year, after a blog post announcing the production-friendly features of Pytorch 1.0[a], we decided to switch our production systems from TensorFlow to Pytorch. So far we're happy with the decision.
I use TensorFlow via Keras because there are a lot of concise examples, Q&As and tutorials on it for different use cases while Pytorch looks a little bit like a magic black box you can't use before you take a course and study it as a whole. I believe this is a major point for many. Also the first Pytorch [official] tutorial I've found mentioned an nVidia GPU as a prerequisite and I don't have a GPU, just old built-in Intel Graphics with Core 2 Duo and TensorFlow seems having no problems with this.
I never felt like I was not re-learning tensorflow! A constant series of breackage, deprecations, new apis etc.
I think that’s a property of Google though, because there’s never been a google service for which I’ve read the documentation and not been like “I do not understand what’s going on at all” for a non-trivial amount of time.
And all their examples for Python are just like “run this magic-code-ridden python file, congrats you did the tutorial”.
The breaking changes are quite well known, see Angular 1.x ~> 2.x, which is arguably a huge reason (but not the only one) why React took a big chunk of their market.
But mostly I don't believe such criticism has much of chance of changing anything: The creators are already the most qualified you could have working on it, and it's unlikely they intentionally made mistakes in the early rounds of API design. Maybe it would have helped to spend a few more weeks on specifications rather than coding but, again, I wouldn't feel very comfortable second-guessing Google's project management.
I think the most likely outcome if such complaints start to pile up would be future projects just remaining proprietary, either longer, of completely.
Also, I don't think I am ungreatful. It's a serious criticism of a software i use almost daily. I really don't consider tensorflow to be great or even some kind of software, but for some tasks it's the only option.
Also, google profits a lot from releasing tensorflow into the wild.
But, all in all, tensorflow 2.0 seems like a step in the right direction.
https://www.tensorflow.org/swift/
And no.. it's not simply a Swift wrapper for tensorflow, it's literally Tensorflow (or generally speaking: differentiable calculation graphs) built into the Swift language.
Overall, it looks like Pytorch is forcing TF devs to focus more on users and usability, and I'm excited to see how they continue to spur each other's growth.
I'm pretty sure there will be plenty of incompatible API changes in every 2.* release, and a completely new "canonical" way introduced in TF 3.0.
Does this mean we can get a pure C API for training and running? Right now, IIRC you can only train in Python and C++ which ignores a large part of the programming community. If not a pure C FFI API, what is the approach being taken?
(Of course, you could also write Tensorflow protobuf directly, but that would be tedious.)
should i still use those outdates TF slim models that are for some reason kept in models/inception?
please, tensorflow team, make this stuff easier for us non-Phds.
As a programmer, I don't want to think in terms of dataflow graphs. That is what compilers are for!
However, migrating to a keras-like approach, you can work thinking about "objects with variables inside" and then the graph will be built for you by the abstraction introduced by `tf.keras.Model`.
However, for automatic differentiation, graph is always required (as you can see from the example that uses eager)
Can you explain what you like so much about manipulating graphs, over just writing statements like z= x⋆y, where ⋆ is some tensor operation?
Really, the graph makes me feel like I'm stacking Lego bricks with chopsticks, rather than with my bare hands ;)
The same description (the graph) can be taken and used in every other language. I can move a trained model to production just picking a single file (a `tf.train.SavedModel` IIRC), give it a tag and I'm ready to go, with different models with a native support for the "tagging" (hence the model versioning).
The worst thing about tensorflow is being locked into having to use python for graph description and training (in my opinion) ...
- Using a JVM to compile mainly C++ and python, implying to deploy a JDK/JRE just for that.
- Awful RAM consumption, compiling tensorflow takes 16GB of RAM to create a python wheel
- Try to download the world and compile everything without allowing to specify external dependencies.
- Compile time that are crazy long due to previous reason.
- Invasive. Make very hard to integrate with anything else that does not build with Bazel
- Make very hard, or sometimes, impossible to tune compiler flags
- Just not reproducible.
- Unstable, try to build a recent Google package with a 6 month old Bazel and good luck.
I strongly advise you to watch this video of the last FOSDEM : https://archive.fosdem.org/2018/schedule/event/how_to_make_p...
Many of your other ones are byproducts of the fact that Bazel is primarily a build-from-source system. This has some benefits, particularly in a C++ ecosystem where binary compatibility across versions basically doesn't exist. But it also has some big drawbacks when it comes to compile times.
I do see Bazel seems to support depending on a prebuilt .so, though I have not tried this: https://docs.bazel.build/versions/master/cpp-use-cases.html#...
> Invasive. Make very hard to integrate with anything else that does not build with Bazel
I think your main options here are:
1. prebuild the other projects, then depend on the .so from Bazel (https://docs.bazel.build/versions/master/cpp-use-cases.html#...)
2. write a BUILD file for the external project. Here is an example of a project that builds a bunch of non-Bazel deps by writing Bazel BUILD files for each of them: https://github.com/googlecartographer/cartographer/tree/mast...
> Make very hard, or sometimes, impossible to tune compiler flags
"bazel --copt=<your options here> :target"?
> Just not reproducible.
What isn't reproducible? I tend to think of reproducibility as a strength of Bazel. Because all of your dependencies are explicit and Bazel fetches a known version, the build is less dependent on your system environment and whatever you happen to have installed there.
Disclosure: I am a Googler. I have some gripes with Bazel, but overall I think it gets some important ideas right. You have a BUILD file that is declarative, then any imperative code you need goes into separate .bzl files to define the rules you need.
Nix, Guix and Spack packager managers solved the C++ ABI issue a long time ago already without the crazy needs of Bazel in term of resource consumption, integration and compile time. They even supports binary distributions for some of them.
> I think your main options here are: > 1. prebuild the other projects, then depend on the .so from Bazel (https://docs.bazel.build/versions/master/cpp-use-cases.html#...) > 2. write a BUILD file for the external project. Here is an example of a project that builds a bunch of non-Bazel deps by writing Bazel BUILD files for each of them: https://github.com/googlecartographer/cartographer/tree/mast....
I know that. But all of them are terrible options. I do not want to depend of SQLite, OpenSSL, libxml or whatever other system library compiled by Bazel, nor I want Bazel to takes 45 minutes to recompile them. Additionally, this will cause diamond dependency problem with other softwares that use Bazel artefact without compiling with Bazel.
> "bazel --copt=<your options here> :target"?
Can I use that to specify a flag to some target and not some other ? Without having to build sequentially each of them ?
Concrete example of the Maddness: SQLITE will not compile if you enable some options that would make tensorflow faster.... Bazel recursively compile both.
> What isn't reproducible? I tend to think of reproducibility as a strength of Bazel. Because all of your dependencies are explicit and Bazel fetches a known version, the build is less dependent on your system environment and whatever you happen to have installed there.
Bazel try to build in isolated environment but do half of the job. It still depends on system compiler, and do not chroot nor "compiler-wrap" ( c.f Spack ) making the build still very vulnerable of system side effect and update.
> Disclosure: I am a Googler. I have some gripes with Bazel, but overall I think it gets some important ideas right. You have a BUILD file that is declarative, then any imperative code you need goes into separate .bzl files to define the rules you need.
I can understand that Bazel is very convenient in Google environment with Google resources. But it's a nightmare for everyone I talked to outside of Google.
Yes this is certainly something that is solvable at the package manager level also. And this approach will certainly have shorter compile times.
I agree it would be nice if Bazel integrated with package managers like this more easily. I hope Bazel adds support for this. There is a trade-off though: with less control over the specific version of your dependencies, there is a greater risk of build failure or bugs arising from an untested configuration. Basically this approach outsources some of the testing and bugfixing from the authors to the packagers. But it's a trade-off I know many people are willing to make.
> Can I use that to specify a flag to some target and not some other ? Without having to build sequentially each of them ?
You can put copts=["<opt>"] in the cc_library() rules in the BUILD file. This will give per-target granularity. You can add a select() based on compilation_mode if you need to define opt-only flags: https://docs.bazel.build/versions/master/configurable-attrib...
> Bazel try to build in isolated environment but do half of the job. It still depends on system compiler, and do not chroot nor "compiler-wrap" ( c.f Spack ) making the build still very vulnerable of system side effect and update.
Bazel allows you to define your own toolchain. This can support cross-compiling and isolating the toolchain I believe, though I don't have any direct experience with this: https://docs.bazel.build/versions/master/toolchains.html
> I can understand that Bazel is very convenient in Google environment with Google resources. But it's a nightmare for everyone I talked to outside of Google.
I hear you and I hope that we see some improvements to integrate better with package managers.
FWIW, I have been experimenting with auto-generating CMake from my Bazel BUILD file for my project https://github.com/google/upb. My plan is to use Bazel for development but have the CMake build be a fully supported option also for users who don't want to touch Bazel.
Thank you for your answer.
I really wish the auto-generation of CMake build scripts will solve the problem(s).
But even that I have doubts. The way Bazel works will make very hard to auto-generate "standard modern" CMake that allows external dependencies. This is orthogonal to the way Bazel works.
Tangentially related to package manager integration: Bazel can now build with CMake in-tree [2].
[0] https://github.com/bazelbuild/rules_nixpkgs/
- It can take advantage of many cores which you have in your computers.
- It's polyglot. We have several languages in our project, and mixing several build systems were a nightmare.
- It's correct. Very rarely you really need to run clean builds.
It has its learning curve, but once you got it, you start to love it.
Which is a bad thing when you're on a shared server. I remember it wasn't straightforward to limit the number of cores it utilizes and it even ignored the flag I passed it. I eventually resorted to waiting until the weekend when no one in the lab was running jobs.
"--jobs N" limits the parallelism during the execution phase (when Bazel executes compilers, etc.) but doesn't limit the parallelism of the analysis phase (when Bazel loads BUILD files and constructs the build graph). For the latter there is a "--loading_phase_threads N" flag.
We are actively trying to make Bazel better use the local resources, and also clean up the current mess that the flags are in this area. See the discussion at https://groups.google.com/d/msg/bazel-dev/7pcrJY6Xo78/JDe256... for some context.
(I work on Bazel)