Hacker News Reader

Comment by fnord123 | Hacker News Reader

If you work in a team mixing data engineers and data scientists then python has been a superior choice for a decade as the different team members are all using the same language to build the platform and to use it.

We've had a lot of success weaning people of R and MATLAB in finance (so replace "data scientist" with "quant"). Of course if you work in a field that doesn't have the libraries and can't build them in house then your mileage will certainly vary.

View on HN

This is a great point. Python has been a solid data engineer/pipeline language for years now. It makes it easy for data scientists and data engineers (or people who wear both hats) to work in the same environment and codebase.

The only area of un-touched by Python is web frontends and mobile apps.

Hope Web assembly would change that.

Off-topic, but what is a "data engineer"?

In my team, "data engineer" refers to someone who deals with the ETL process, but also some feature engineering (ie, designing and extracting features. There's a lot of cross over here with the work Data Scientists do, but often the tools are different: for example, in the Python world the data engineer will use Pycharms while the data scientist will use Jupyter.

I'm sure someone can come along with a better description, but it's kinda like devops/sysadmin but specifically for data storage and access.

You could also look it up...

At some point, people realized that 90% of data science work was building pipes and keeping them clean. And then data engineering was invented so that the data scientists wouldn't have to get their hands as dirty. :)

A snappier version which gives the flavour: Data scientists use {tensor flow, scikit-learn, parquet, hadoop, spark, scrapers, etc}. Data engineers write {tensor flow, scikit-learn, parquet, hadoop, spark, scrapers, etc}.

I'd include database engineer in that bin. We used to call people DBAs but that seems a bit limiting now.

That's a good description for a data engineer from my end.

Reply on news.ycombinator.com