Hacker News Reader

Comment by rm999 | Hacker News Reader

I'm glad to see the quick convergence on Python. I've evaluated Python every ~2 years since 2006 for "data science" tasks (machine learning, statistics, data munging, and visualization). I'd argue that Python only properly covered this full data science stack 1.5-2 years ago. R covered this stack adequately probably around 2011-2012. Matlab had this before 2005.

What makes Python a superior language to Matlab and R is the ease of software development. It's an easy, pleasant language to work in, and I trust it for production tasks (I've written production R code and it's fairly hard to read and fragile).

What's even better is data science is moving 100% into Python 3 (from 2) by 2020:

http://www.python3statement.org/

View on HN

If you work in a team mixing data engineers and data scientists then python has been a superior choice for a decade as the different team members are all using the same language to build the platform and to use it.

We've had a lot of success weaning people of R and MATLAB in finance (so replace "data scientist" with "quant"). Of course if you work in a field that doesn't have the libraries and can't build them in house then your mileage will certainly vary.

This is a great point. Python has been a solid data engineer/pipeline language for years now. It makes it easy for data scientists and data engineers (or people who wear both hats) to work in the same environment and codebase.

The only area of un-touched by Python is web frontends and mobile apps.

Hope Web assembly would change that.

Off-topic, but what is a "data engineer"?

I'm sure someone can come along with a better description, but it's kinda like devops/sysadmin but specifically for data storage and access.

You could also look it up...

That's a good description for a data engineer from my end.

I'd include database engineer in that bin. We used to call people DBAs but that seems a bit limiting now.

At some point, people realized that 90% of data science work was building pipes and keeping them clean. And then data engineering was invented so that the data scientists wouldn't have to get their hands as dirty. :)

A snappier version which gives the flavour: Data scientists use {tensor flow, scikit-learn, parquet, hadoop, spark, scrapers, etc}. Data engineers write {tensor flow, scikit-learn, parquet, hadoop, spark, scrapers, etc}.

In my team, "data engineer" refers to someone who deals with the ETL process, but also some feature engineering (ie, designing and extracting features. There's a lot of cross over here with the work Data Scientists do, but often the tools are different: for example, in the Python world the data engineer will use Pycharms while the data scientist will use Jupyter.

> What makes Python a superior language to Matlab and R is the ease of software development.

In practice I think the trends we're seeing have more to do with the fact that most universities now teach CS and data science using Python.

Given that Python is approximately as good as R, and it's becoming increasingly much easier to find good people to hire, there's very little reason not to be a Python shop.

That, and people started transitioning into data science from other fields where Python was already more popular.

Another reason why I'm glad python is winning (and a fun read):

http://arrgh.tim-smith.us/

Love!

"R is a shockingly dreadful language for an exceptionally useful data analysis environment. The more you learn about the R language, the worse it will feel. The development environment suffers from literally decades of accretion of stupid hacks from a community containing, to a first-order approximation, zero software engineers."

Yup.

The first time I tried to read R code I was immediately disgusted with how '.' is a valid variable name character. Not only valid, but inexplicably also in common use.

That's in part because, unaccountably, '_' denotes assignment in S.

> unaccountably

Maybe because in ASCII-63 the _ codepoint meant left-arrow (which was preserved in at least smalltalk systems much past the 60s)?

My theory is that they might have been trying to protest against the non-mathematical theft of = (equality) to mean assignment, as = is used in most other languages. (Not that that's an excuse for violating the shit out of the Principle of Least Surprise in a trillion different ways.)

It's immature to rail on R for things like how its syntax doesn't look like that other language you already know, but yes, it's extremely quirky. And its unchecked flexibility, while great for experts providing DSLs and such, enables non-programmers to produce garbage implementations. The vast majority of R's myriad community packages (R's biggest strength!) seem to be written by a grad student who read half a tutorial.

How do you feel about Julia? Data analysis in Python has plenty of warts.

Julia is great, but IMO, it needs to reach 1.0 first before making a splash. It would be frustrating if you found your tools developed on v0.4 stopped working on v0.5 released a year later. API/ABI stability is critical to the adoption of new programming languages/libraries.

I've heard they plan to release Julia 1.0 this year. So keep an eye on it.

Yeah, I heard about it from others, too. Look forward to it. Hope the dev team stick to the plan.

Rumor is it's going to be this summer at the annual conference.

http://danluu.com/julialang/

+ The versatility of Python. It is one of the "funnest" languages to work with because I'm always surprised by the number and variety of packages available to do all kinds of tasks.

Reply on news.ycombinator.com