TensorFlow: Large-Scale Machine Learning on Distributed Systems (2015) [pdf] | Hacker News Reader

TensorFlow: Large-Scale Machine Learning on Distributed Systems (2015) [pdf]

download.tensorflow.org

201 pointsmax_28 comments

Not to sound overly critical, but I don't enjoy the tone of this paper: presenting the concepts (e.g. graph operations) and features (e.g. automatic differentiation) of TensorFlow as new and novel when in fact very similar systems like Theano have existed since at least 2010 (which they finally get round to mentioning 14 pages in). They really should have shorn the paper in half and just focused on the distributed bit, which is the really novel and exciting bit.

Addendum/Edit: With that said, I don't want to disparage the amazing technical achievement of the Google Brain team and the way TensorFlow really smartly reuses and improves on the concepts that make Theano really powerful. I'm following the project very closely and TensorFlow has a very high chance of being the foundation for my future projects :)

Not to sound overly critical ;), but the word "novel" doesn't appear in the paper and the word "new" only appears in tehnical contexts like "quickly experimenting with new models".

Where does it say that its automatic differentiation feature is new and novel? It actually calls out Theano in the paper:

"Like Theano and Chainer, TensorFlow supports symbolic differentiation, thus making it easier to define and work with gradient based optimization algorithms."

EDIT: Okay, I see that you already saw the reference. I guess I didn't find the tone as bad as you did. :-/

It's confusing but automatic differentiation, symbolic differentiation, and finite differences are all different approaches. "Automatic differentiation" typically refers to reverse order chain rule. Higher level libraries do really neat stuff with it. For example autograd can differentiate much of your numpy calls, even of composed of many complex functions.

Even on distribution, I fail to understand where the novel idea is. I must miss something.

Sadly, most are only even paying attention because of the 100kg gorilla in the room…

>> "A computation expressed using TensorFlow can be executed with little or no change on a wide variety of hetero- geneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards."

Are there other comparable systems that deploy using the same build across all platforms?

If not, putting aside other advantages like doing computation across multiple machines, being able to using the same build is a major step forward in my opinion.

Are there other comparable systems that deploy using the same build across all platforms?

SystemML[1] doesn't, AFAIK, claim to scale down to mobile devices, but it does scale from a single desktop machine up to an arbitrarily large distributed cluster, using the same code. If you write code in DML or PyDML, the runtime will take care of distributing it across the cluster. And it's "smart" enough to take the cluster characteristics into account, and can also dynamically re-evaluate stages of the computation as the job is running, so it's always trying to use resources as efficiently as possible.

[1]: https://systemml.apache.org/

Preliminary White Paper

A Google Beta by any other name would smell so sweet.

Are there APIs to do the training in C++? I can't get away with doing what I want to do in the cloud (nor do I really want to).

Edit: I see http://mlpack.org/ - anyone have experience using this?

More C++ support is on our roadmap:

https://github.com/tensorflow/tensorflow/blob/master/tensorf...

Does anyone know what they used to generate those technical diagrams? Che bella!

I drew the diagrams in Inkscape. (Naturally, this doesn't include the screenshots of TensorBoard or EEG.)

it's not a white paper, it's a full scientific publication.

It is not a full publication yet. The experiment section is left as to-do and that is a very important section in any paper. They will most likely publish it in the near future.

Do you mean this: http://arxiv.org/pdf/1603.04467v2.pdf

This is same as the one in the link. Section 8 is todo. Putting on arxiv is not exactly publishing, I meant publishing at a conference.

Literally the third line of the article says: "(Preliminary White Paper, November 9, 2015)"

The line is wrong.

The arxiv paper is just a repost of the preliminary paper, which is really more a design outline and rationale -- we may work on a more substantial paper later, we've been busy building the software.

It would be nice if you changed it from "white paper" to something more accurate, like "provisional".

The reason is that "white paper" has a very specific connotation, and this paper is not consistent with the white paper.

Personal opinion: in science the word publication now means "to make accessible" rather "to publish in a conference" (the conference = publication thing is mostly CS) or "to publish in a journal" (like biology and a lot of non-arxiv physics). Pre-pub peer review isn't even required any more. arxiv (and scientists putting preprints on web sites) changed this. Of course, this is personal opinion, based mainly my observations and doesn't necessarily represent the mainstream opinion (yet).

If I'm not supposed to trust that line, how do I know what lines I'm supposed to trust.

> it's not a white paper, it's a full scientific publication.

I believe the parent referring to the common usage of "white paper" to refer to B2B marketing materials that are not often written in an academic style.

I was pleasantly surprised to find this was a self-published scientific article and not what I and dekhn usually think of when we hear white-paper.

The name of the file on the server is "whitepaper2015.pdf"

Is it peer reviewed?

No. But in this case, it doesn't matter much - the paper isn't really making any claims in relation to other systems. The authors can reasonably be trusted to describe the design of their own system, particularly since the source is out there. Peer review would improve the paper, undoubtedly, but wouldn't have any effect on the system design itself or Google's reasons for building it. The latter two are the focus points of the whitepaper.

That doesn't matter.

It does if you are claiming it is a full scientific publication.

Reply on news.ycombinator.com