Comment by crockpotveggies | Hacker News Reader

Comment by crockpotveggies | Hacker News Reader

Parent Full thread

crockpotveggies

Yes, so one of the core libraries within the DL4J project is datavec which is ETL-focused. One key problem that we discovered - and fixed - was that reading and transforming data for training could bottleneck a multi-GPU process. You spend a lot of $$$ on a deep learning computer, but making the library performant enough so that you could load data at the same rate the GPUs could consume it was challenging. This scales to about 4+ GPUs (depending on datatype) and we're building a datavec server so this can scale much larger. There are still good returns if you clean and transform your data and presave to disk, which helps with large machines such as a DGX. However, other bottlenecks still apply (which we are solving right now).

I hope that answers your question. I consider the process of extracting records, transforming them, and loading them for training to be "ETL". I understand ETL also applies to other data consumption.

*I should also note that if you wanted to use datavec for ETL and do not wish to train a deep learning model, it is quite useful for columnar data.

Why mix ETL and training?

I am using TF, and in my workflow I first do all ETL in some separate process, dump all training/validation data into TFRecord file, and then my training program consumes it. Clear separation of concerns without any performance penalty.

And I can iterate over training logic with various parameters as many times as I want without touching ETL.

Integrations and deployment. We are targeting data engineers with this not data scientists.

A lot of our user base are people who want to take what the python folks did for creating a feature vector from raw data -> import a model (note this is all in java) and run the training pipeline and test pipeline in the same place.

The goal is to provide an opinionated set of tools for doing this.

Upside here: It's obvious how to both pre process data and create ndarrays from them. Tighter integration also allows us to make some optimizations under the hood with how memory allocation is handled as well as allows us to target different data types like images + sound as well as databases in the same place.

What I'm guessing here is: You're a data scientist focused on building the models. Someone has to take that code and put it in production. If you work at a startup, production might be "TF serving". If you're a fortune 500,

you're likely not deploying that. People we've worked with are usually constrained in some way (especially by the JVM) .

You aren't our target audience, it's your colleagues.

We'll work backwards from that by adding our python interface and the like, but largely we want to solve a cross team concern after models are built.

Because you might want, as I definitely wanted, to also iterate over the feature engineering, not only the network and training parameters. Thus, doing ETL "inline" is really cool, and speeds up your iteration.

It is also easy once you have a proper language that supports multi-threading. Since you do it "full speed" with full utilization of the GPU, there is no disadvantage of doing the ETL while training.

But what if your training logic needs a change in ETL - You have to iterate on ETL too. So doing it inline makes a lot of sense. Microsoft has done something like this with SQL server + R offering. Personally I find the MS approach quite appalling. You have to load your model in a PLSQL script, so the ETL + running the model is seamless, but imagine debugging and oh forget about multi GPU support.

The part of the ETL you'd want to "crystallize" is just the transformation from raw data to feature vector.

Beyond that, you already "change in ETL" when you experiment. I'm not sure how this changes anything.

Out of nowhere you've sprinkled in "multi gpu support" which I'm not sure is relevant here. Do you mean as part of training?

We handle the gpu bits for you. All you do is define your transform logic, it runs on one of our backends like spark and then when you go to allocate a tensor boom gpu.

There's no special compilation or process needed to make this happen.

Dl4j supports multi gpu training out of the box. All you need to do is uses our parallelwrapper module.

We can also do distributed training with gpus on spark as well (yes this includes cudnn).

You've also for some reason decided to attach an open ended coding library where you can do whatever you want to a database?

Baked in ML in the database servers is already notoriously bad. The whole point of what we're doing is to provide a middle ground.

Data engineers have to do this anyways.

Adam, sorry guess I was not clear. What I said was that the MS Sql server + R strategy of baking in ML in PLSQL is appalling and doing something like what you are doing with DL4J is perhaps the right approach. The multi GPU support rant was for SQL Server + R, not DL4J.

"You've also for some reason decided to attach an open ended coding library where you can do whatever you want to a database?" I didn't catch this part.

Right so I think we were agreeing that how database servers bake in the ML is a bit weird and not the way to go.

The "open ended coding library" is datavec here. Comparing them here is only really semi valid. The processes there are definitely brittle.

I think he agreed with you..!

Yeah sorry about that..I was a bit confused on how to read it. There was a lot mixed in there. I think we're clear now :D.

Reply on news.ycombinator.com