Yes indeed! Thanks for the follow up. I just wanted to give folks a survey of the things they should look at when evaluating things like these charts to see how fast something is.
The casual reader won't know to look for these things (it's actually not their fault, a lot of this knowledge is very field specific)
I've made comparisons to deep learning flow graphs and spark's computation graphs in the past, but there are very different considerations and use cases for both.
I think what I was commenting on was more of the minute details that get lost in summaries like this.
If anything this is a pitch for having an end to end specialized system that understands your data pipeline all the way to your linear algebra calculations.
That way all of these variables are taken care of.
A simple example where this matters a ton is just the ETL in to a neural network.
You're loading data from some disk (that may or may not be networked).
What's the input format of that data? What's the block size?
Say it's image blobs stored on hdfs. You know ahead of time your gpu can fit an optimal batch size of 32 in to memory because your image resolution is high.
You then need to go back and prepare the data for loading.
To get the optimal performance for data loading, you need that data loading to be aligned with hdfs block size (otherwise you cause an additional hdfs seek) and saved in whatever your favorite vector format is.
Then to evaluate how fast a "minibatch" will be processed, you need to figure out how fast your architecture is in addition to how the communication with the gpu is handled.
I think overall this is just a hairy subject to cover in a small blog post.
While these "programming flavors" are very similar, they are optimized differently in practice.
If you think about spark, the "nodes" are more general purpose operations with a ton of JVM based code gen happening.
It knows about different "edges/ops" in addition to having a registered schema. It's lazily initialized with a DAG built with code gen after the declarations are done.
A "tensor graph" doesn't have types beyond knowing what device it's running on and perhaps a data type like float/double/half.