Hacker News Reader

Comment by pbourke | Hacker News Reader

Everyone wants to fire up Tensorflow, Keras and PyTorch these days. Fewer people want to work in Airflow and SSIS, spend days tuning ETL, etc. This is the domain of data engineering, which bridges software engineering and data science with a dash of devops. I’ve been working in this field for a couple of years and it’s clear to me that data engineering is a necessary foundation and impact multiplier for data science.

View on HN

Don't forget data cleaning. A huge issue I've seen is just getting sufficient data of a high enough quality.

Also, (for supervised classification problems) labelling is a big problem.

It is almost as if we need a "data janitor" title.

Phht you don't want to call it data janitor; no-one good will want that title. At least call it Data Integrity Engineer or something reasonably high-status.

At my company we have just created a position called Data Steward.

Data Sanitation Engineer :)

Machine Learning Data Integrity Engineer

This is a hugely important point. Having the right data is most of the battle in the typical project. I'm fortunate to run the data science department at a company with an amazing and supportive team of devs. We work together to make sure that all data is recorded in the most useful way and we talk regularly. Without their support, my models wouldn't be very useful.

Data cleaning always sounded suspicious to me.

As a data engineer I've often billed myself as the Charlie Kelly of data

I'm finding myself really enjoying this type of work and I think I would like to specialize in it. Any good learning resources you used on your path to where you are now?

Any formalized paths I could take to enter this field?

I got into data engineering by starting in ETL (DataStage) and learning about cloud services (AWS) on my own and getting my next job in a cloud-based SaaS startup.

How might you recommend moving into this field more?

Reply on news.ycombinator.com