Hacker News Reader

Comment by XCSme | Hacker News Reader

Not only upgrading is hard, but also installation (on Windows at least). For each Tensorflow version you need a specific python version, a specific CUDA version, specific tensorflow-gpu version, and many other easy to get wrong things. The problem is not the requirements, but that it's very hard to know what versions are compatible. There are endless threads on Github of people trying to use Tensorflow but failing after spending days trying to install it.

View on HN

Try using these containers[1] from my peer team at IBM. They run on a variety of architectures (x86, ppc64le, with and without GPUs in most cases).

In addition if you don't want to fiddle with the containers, there's also a conda channel[2] that lines stuff up. I work on a peer team in machine vision, and use these for personal and professional projects.

[1] https://hub.docker.com/r/ibmcom/powerai/

[2] https://www.ibm.com/support/knowledgecenter/en/SS5SF7_1.6.1/...

Tensorflow sounds like an ideal candidate for running in a container. List out all your approved compatible versions in the Dockerfile and distribute it with your source code and anyone can reproduce your results with the exact same setup.

Yes, but for personal use, unless someone else already made those containers, I will still have to go myself through the trial-and-error process of finding the right combination of versions. Yes, second installation will be easier, but if I just want it on my PC it doesn't really help.

You can `docker pull username/mytensorflowcontainer` and start from someone elses' work. Looks like Tensorflow has a how-to on the site: https://www.tensorflow.org/install/docker including working cpu-only and gpu-enabled examples.

I've been using containers for this, and mounting the source code.

I've been using Anaconda and it finally made Python work without endless rounds of 'incompatible libary bingo'.

It also works well for machine learning projects.

+1 for Anaconda, worth trying for anyone who has a problem with package versions.

As a software engineer and non-data scientist, I hate Anaconda because it feels like it's a tool that tries to be the be-all, end-all package management tool for everyone in the data science field, yet it feels like a sloppily built, bloated whale. It's even managed to overwrite PATH on some of my Linux machines, which is where I drew the line.

I vastly prefer creating hermetic environments with either venv or Docker. They're much cleaner and easier to work with. I wish data scientists would adopt these tools instead.

Sadly, many of the ML models I investigate on Github don't even have their package requirements frozen. It's an uphill battle...

> I vastly prefer creating hermetic environments with either venv or Docker. They're much cleaner and easier to work with. I wish data scientists would adopt these tools instead.

I suspect you have a lot of time on your hands. But for me the 'batteries included' approach really nails it, why repeat the headache over and over again when a single entity can take care of that in such a way that incompatibilities are almost impossible to create? The hardest time I've had was to re-create an environment that ran some python code from a while ago, with Anaconda it was super easy.

I'm sure it has its limitations and just like every other tool there are situations where it is best to avoid it but for now it suits me very well.

I would suggest you try out Miniconda (https://docs.conda.io/en/latest/miniconda.html). It comes with just the basics, and let's you install TF with GPU support by simply doing:

conda install -c anaconda tensorflow-gpu

it's incredibly slow.

What’s incredibly slow? Installing things with conda?

conda install for our environment.yml: about 3-5 minutes solving, then 5-10 minutes installing.

pip install with almost exactly the same set of packages: 3-5 minutes total.

Anaconda may work well as a virtual environment for some ml projects, but it is by no means a solution for getting a gpu-working installation of tensorflow on Windows.

> getting a gpu-working installation of tensorflow on Windows

I think part of your problem is the last word there. Windows is a bad match for such an environment. On Ubuntu it is pretty much painless.

I agree, and prefer Linux myself, but some clients only allow for solutions based in Windows (and no containers) :/

They're (slowly) making this better. Starting with 1.15, there is only one tensorflow pip package, no tensorflow/tensorflow-gpu hell any longer.

Really? https://www.tensorflow.org/install/gpu says to `pip install tensorflow-gpu`.

https://groups.google.com/a/tensorflow.org/forum/#!topic/dev...

Ah I guess it's on the way but not fully there yet

Reply on news.ycombinator.com