As someone who is mostly working in the field of data warehousing, ETL has a very meaning to me (Extract, Transform, Load). Is this the same as you’re talking about ?
I hope that answers your question. I consider the process of extracting records, transforming them, and loading them for training to be "ETL". I understand ETL also applies to other data consumption.
*I should also note that if you wanted to use datavec for ETL and do not wish to train a deep learning model, it is quite useful for columnar data.
I am using TF, and in my workflow I first do all ETL in some separate process, dump all training/validation data into TFRecord file, and then my training program consumes it. Clear separation of concerns without any performance penalty.
And I can iterate over training logic with various parameters as many times as I want without touching ETL.
A lot of our user base are people who want to take what the python folks did for creating a feature vector from raw data -> import a model (note this is all in java) and run the training pipeline and test pipeline in the same place.
The goal is to provide an opinionated set of tools for doing this.
Upside here: It's obvious how to both pre process data and create ndarrays from them. Tighter integration also allows us to make some optimizations under the hood with how memory allocation is handled as well as allows us to target different data types like images + sound as well as databases in the same place.
What I'm guessing here is: You're a data scientist focused on building the models. Someone has to take that code and put it in production. If you work at a startup, production might be "TF serving". If you're a fortune 500,
you're likely not deploying that. People we've worked with are usually constrained in some way (especially by the JVM) .
You aren't our target audience, it's your colleagues.
We'll work backwards from that by adding our python interface and the like, but largely we want to solve a cross team concern after models are built.
It is also easy once you have a proper language that supports multi-threading. Since you do it "full speed" with full utilization of the GPU, there is no disadvantage of doing the ETL while training.
Beyond that, you already "change in ETL" when you experiment. I'm not sure how this changes anything.
Out of nowhere you've sprinkled in "multi gpu support" which I'm not sure is relevant here. Do you mean as part of training?
We handle the gpu bits for you. All you do is define your transform logic, it runs on one of our backends like spark and then when you go to allocate a tensor boom gpu.
There's no special compilation or process needed to make this happen.
Dl4j supports multi gpu training out of the box. All you need to do is uses our parallelwrapper module.
We can also do distributed training with gpus on spark as well (yes this includes cudnn).
You've also for some reason decided to attach an open ended coding library where you can do whatever you want to a database?
Baked in ML in the database servers is already notoriously bad. The whole point of what we're doing is to provide a middle ground.
Data engineers have to do this anyways.
"You've also for some reason decided to attach an open ended coding library where you can do whatever you want to a database?" I didn't catch this part.
The "open ended coding library" is datavec here. Comparing them here is only really semi valid. The processes there are definitely brittle.
https://deeplearning4j.org/datavec
The thing to remember is that this is ETL focused on machine learning. It's not any old set of transforms. It's transforms that help us normalize, standardize and finally tensorize various data types, be they images, video, text or time series.