What Data Scientists Really Do, According to Data Scientists

A large number of hedge fund analysts are data scientists. Just because computational finance models used a tool that abstracted away a large portion of the programming doesn't mean that they weren't using applied statistics and domain specific modeling to solve problems.

>> why can't data science simply be about applying the scientific method in the realm of data analysis? >That's what a statistician do.

Mmmh.

Run that experiment for me next time you meet a statistician:

    - ask him if he can apply Chi-squared to a decision problem

    - ask him if he can *explain* how and why Chi-squared works.

In my experience, all statisticians can do the first, almost none can do the second.

Learning how to use a screwdriver to screw screws without understanding notions of torque and moment doesn't mean you're applying the scientific method.

In my humble experience as a data scientist at a big tech company, a big differentiating factor is familiarity with the R or Pydata stack. It's not just its own language and library, it's a more general idiomatic way toward approaching and solving problems from a "software first" perspective.

I sometimes work with people who are formally trained in stats at a much higher level than myself. But with my lesser ability combined with my capability to write production grade Python code using the pydata stack, my code and solutions are more likely to make it in.

if you really understand ML/DS, then you would know that ML is founded on all the same concepts.

> "... they tackle data is radically different from statistician ..."

the short answer is, before we had limited data, compute, and the problems businesses needs increased substantially.

now we can do better things, and if you cant do more than what companies were doing in the 1950s, then your value proposition is significantly less than someone that's using industry practices established in 2017 / early 2018.

> "I wish statistic is better at marketing. Oh well."

as a data scientist, I have to know stats, ML, big data tools (spark/splunk/hdfs), programming, and domain specific knowledge. some data scientists are just rebranded statisticians, which is fine. the role of a data scientist varies to the needs of the org they are in. I mostly do anomaly detection and classification, while others may focus solely on AB testing.

> > "> why can't data science simply be about applying the scientific method in the realm of data analysis?"

the statement doesnt make sense to me. if you are doing data analysis then by default you are doing observations and measurement on empirical data. As far as experimentation goes, almost no one has the resources or the standing to do that. Some do, like those AB testers I mentioned before.

> "Actually my last day of internship I've met a few statistician interns some of them are from Cal (UCBerkely) and they came to the same conclusion (we have a lot of complaints). The ML/DS group is really just doing black magic (nicest way of putting it)"

Then they don't appreciate ML and the problems they are trying to tackle. It reminds me of a stats PHD co-worker that was upset about the number of parameters in an image classification problem we had. they didn't think a model could be ran that had more parameters than observations. I was like we are not running linear models ... As an experiment, go to kaggle and try the spooky author classification competition. See how far you can get with basic stats and then see how much further you can get with ML. hopefully it will give you an appreciation for some of the ML tools available.

It's not black magic, like many other domains; its just a dense subject that takes a lot of time to understand

In social sciences (speaking as an economist), the goal of statistics was always testing theories and hypotheses against empirical evidence. Once you have a solid theory, then you have an understanding of the subject matter, which allows you to make predictions and counterfactual hypotheses. So the primary goal was understanding the data generating process, finding causal relationships between variables, and so on. The goal was specifically not making predictions. One of the first things I learnt was that a bad model can make good predictions (for a while). The ML crowd appears to be taking the opposite approach, focusing on making predictions and disregarding everything else. I suppose it has its uses, but I wouldn't call it science.

This, and your other responses in this thread, come off as rather disappointing to me, as someone who considers the work they do as "data science". Machine learning as a method clearly has quite a bit to contribute to the business world based on revenue alone. My argument could rest here.

But it's also disappointing the way that you belittle all machine learning practitioners, even those with academic credentials, for their work not being worthy of serious consideration. This also sounds a little defensive and projective, and I can't imagine it's easy seeing the forest for the trees with your head in the clouds.

How good ML and Datascience people that you have met were at statistics and in math in general?

The amazing thing is that the black magic gets results (for certain categories of problems).

I think databases and analysis have been around a little longer than 20 years...

exactly. Optimising queries becomes really a critical part of the job when you make complex JOINS on millions of records. Just getting the data can take a huge amount of time before you can even consider models.

I’ve been told by IT from numerous organisations that Hadoop will solve all of our team’s query inefficiencies.

Also hence why we introduce new members of the team to learn how to do efficient queries and joins. And spend time upfront to structure their problems.

I work as a biostatistician and I've been tasked recently with querying large databases using SQL, in addition to analysing the data. However, my programming background is very limited and thus I'm sure my queries are very inefficient.

Could you point me to some materials/texts about how to improve querying efficiency for SQL? If it's oriented for beginners then that would be ideal.

Thanks in advance.

No. Splunk is it's own proprietary thing. They actually work somewhat similarly (in that they're both query engines over distributed data using a map-reduce paradigm) although a closer parallel is lucene/elasticsearch.

Sounds like poor role definition? You don't have your laboratory scientist cleaning lab equipment or ordering reactants.

so I checked out the website and I still don't get what Dremio actually is...

I know what I need which is a wide open easy to use (think CEO's personal assistant who has zero background in anything other than basic business processes easy) visual ETL tool.

We have found some decent VERY expensive and VERY user unfriendly tools but nothing cheaper than just hiring a person to custom handle each batch.

Basically...we're stuck with Cognos, et al.

I think, perhaps, you misunderstood the OP's objection, particularly with respect to feedback loops.

It may not be safe to assume the inputs are biased [1], but, then, it's also not safe merely to assume that they aren't.

What's particularly dangerous is to assume unbiased inputs if the subsequent conclusions (the outputs) then influences future inputs [2]. That's the feedback loop that can serve to amplify biases in the data/conclusions with each cycle.

> must that be anything-ist? Or just a fact?

To answer your question, no, it doesn't have to be anything-ist and is, of course, a fact. However, to characterize it as just a fact ignores that it may well be an anything-ist fact, if it had been based on anything-ist data.

Ultimately, this is just a GIGO [3] problem, a concept from early in "computing" (aka "data processing").

[1] which I'm substituting for "flawed", since "unbiased" was the OP's term

[2] for example, by setting interest rates which have an influence on loan repayment likelihood

[3] https://en.wikipedia.org/wiki/Garbage_in,_garbage_out

Big data may conclude that the data shows black people are more likely to commit crimes, but it doesn’t tell you the why, which may be that black people are systematically opressed, more likely to come from a poorer background, discriminated upon so as to reduce their options in life. The AI, based on statistics can tell you the effects, but not the causes.

So if i were to go and beat up every person named John, and then machine learning takes a crack at it and tells us that people named John are more likely to get injured, we may end up discriminating against Johns without realizing it was I who just have a thing against people named John. If this happens in a system and creates a feedback loop, It can lead to something becoming a self-fulfilling prophecy when it need not have been.

Based on a naive conclusion the solution may be to deny health coverage to people named John, but obviously the real solution is to put me in jail.

This topic has been discussed at length on HN and has been the topic of many flame wars.

If you're genuinely curious for an answer, in the United States it's absolutely the case that you are legally

> obligated to throw out the conclusion

when it comes to many of the things you mentioned (e.g., you can't consider race, or often gender and sexual orientation depending on the state, when you evaluate a loan/employment application).

The reason for those laws is that, historically, overtly and openly racist people used the loan application process to discriminate in housing.

Asking to be able to use any accurate statistical model for any application is an inherently political request, because bad actors in the past have attempted to exclude people based upon skin color. We can't make that discrimination legal whenever someone can dream up a plausible mathematical model justifying an ultimately racial intent.

The real point of my post is that a discussion on this topic devolves into a flame war unless everyone agrees that we can't turn p-hacking into a legally justifiable way of allowing racial discrimination in important social processes like housing loan or employment applications.

Please explain. If you removed "profiling features" from your analysis, but then through other means ended up with a "similar performant model," then aren't you still profiling, just with less obvious data inputs?

If Group A is low credit risk and Group B is high credit risk, won't your new model still make it much harder for Group B to be approved? If not, it's not "similar." If you are able to dissect the Groups and pull individual high/low risks from within the groups, that would be a superior model, which is not what you claimed. So how are you not profiling, and how does it make a difference in terms of who gets approved/rejected?

It's illegal to use them. Although it's strictly suboptimal for the firm.

So you’ve replaced first-order proxies with higher-order proxies.

Sounds like it wouldn’t make any difference (other than PR).

The distinction, as you've restated it, still isn't useful:

'From a business perspective, the thing that's different about "machine learning" compared to other things you do with data is that it's possible to take the human out of the loop.'

There are many things you can do with data that take humans out of the loop, that don't involve machine learning. For example, software that automatically re-orders stock in a supermarket once stock (calculated based on starting stock less sales) goes below some level.

You could argue that this still has a human in the loop (to define a threshold) and that you're not removing the human from the loop until the thresholds themselves are automatically calculated.

But then you're just moving the job of the human from deciding the threshold, to deciding what % of the time it's acceptable to be out of stock of that item. Sure, you can automate that, too, but then the job of the human still exists: she's just deciding the objective function that stock-out percentage must satisfy, rather than deciding the stock-out percentage for each SKU directly using a jupyter notebook or Excel sheet.

Also Andrew Gelman

Hadley Wickham, Andrew Gelman, and maybe Pete Warden are who data scientists look up to, but I think Nate Silver is the first data scientist to be a household name (admittedly, NYT-reading upper middle class households, but still).

Don't forget David Donoho. His paper "50 years of data science" provides a great historical context of data science and a vision of what "Greater Data Science" should be.

https://courses.csail.mit.edu/18.337/2015/docs/50YearsDataSc...

Also Tukey, Chambers, Cleveland (all three from Bell Labs), Breiman, Tufte.

I would also submit Judea Pearl to the list.

They are statisticians.

Hum, or maybe I meant data analyst, like you say who knows.

I haven't, atleast not since my last university class. I'd love to find some kind of course that trains me on the basics along with applying it through programming.

I do know programming. I know how to build a backend and frontend. What I don't know is how to take a large dataset and start making predictions based off of it.

Since we're on the topic of "what data scientists really do", I have comments about these common recommendations:

1. Kaggle.com competitions incentivise chasing a metric, where very large amounts of man-hours are used to improve a model a tiny bit, which may not have dramatically different performance in a real-world situation than a baseline model. (Kaggle Kernels, which consist of Python/R Notebooks, are better for teaching how to analyze datasets: https://www.kaggle.com/kernels)

2. Andrew Ng's Coursera course is good, for an introduction to terminology only. You won't be using anything from the course in real-world use cases (e.g. hand-implementing the matrix algebra behind backpropagation), but it's good to learn what backpropagation is. And please don't say you're an expert in machine learning because you took the course.

Science is more of an institution than what you are describing like some isolated act of making a prediction

Less black box, more reproduciblity / generalisation is what people ask for these days, so ablative studies exposing how the bricks in the model work individually? In old terms, sensitivity studies.