Hacker News Reader

Comment by ataturk | Hacker News Reader

It was all hype. All of it. I enjoyed learning about data science, but in the end, there were no jobs I could actually apply for and realistically get. I also don't think they were providing the big wins for the company that would justify what they were getting paid. Again, all hype.

View on HN

same here. i work great in teams, masters in computer science, do really well in kaggle competitions, I understand when to use the algorithms, how they work, etc.

but in the very few ds interviews I had, I was tanked as soon I got asked questions like: whats the formula for a T test. I know what the test is, and when its not appropriate to apply it, but I don't memorize those kinds of formulas. the field is just too big to do that

Thanks for sharing your experience!

Were you making it past the initial stage/screen interviews where every question is a buzz-word test? Were you ever interviewed by people who you'd actually work with and know what their team needs? What geographical area do you reside roughly, if I may ask?

it was initial screenings, she was a data scientist (but really a statistician). I passed the first two screenings (code interview, database / sql interview). every question was about statistics, and probability.

This position was in NYC.

I'm sorry but a standard t-test is like the most basic thing in statistics after means and variances. It's not like they want you prove the CLT or something. To me it's more like a statistics fizz-buzz.

To which, of course, the correct answer is "Unless you have good reason to believe the data is normally distributed, you should be using Mann-Whitney", then explain that :-)

Excuse me, I think that at this juncture a Kruskal-Wallis would be more appropriate (assuming multiple distributions).

Thats not really true though. The vast majority of t-tests are done using a sample average as a test statistic, and by the CLT, as the sample size goes up, the distribution of a sample average becomes approximately normal. So unless you have a really weak sample size, the t-test assumption holds even in the absence of normally distributed data.

For many companies, data science means statistics. Statistics are used for experiments and to analyze data. If you don't know what a t-test is, you cannot be hired as a statistician. However, you may be hired as a machine learning engineer.

Try looking for a company that does take-home assignments (these are mostly Kaggle-type problems). Then you don't need to learn formulas by rote, and the interview can focus on the assignment you did.

Experimental design (statistics) is important, but data science is a team sport. The people specialized in statistics may not be able to code up a deep net, and vice versa.

Computer scientists make great data scientists, especially if you know data infra. If I were to start a team from scratch, I'd focus on software engineers and physicists. Most are willing to pick up some ML to broaden their skills, and the ability to go end-to-end from conception to production is invaluable.

This is hyperbole. It's not all hype. A lot of it is hype, yes, but there are many examples of great value being derived from advanced analytics.

The real issue is setting expectations with those on the outside trying to get in - a 3-month boot camp with no prior programming or statistics knowledge is just not going to get you to the level of competence required for this type of career. All of these MOOCs for ML/AI/DS are bullshit if you haven't the foundation underneath, which, based off of those people I know who've taken those courses, doesn't exist for many MOOCers.

The fact of the matter is that most companies don't need "artificial intelligence"; they need intelligence - people with domain and statistical knowledge and enough programming skills to be dangerous. As cool as some AI/ML tech is, you could get so far as a data scientist with great knowledge of a few statistical approaches and above-average programming skills.

You're right in one sense, though: there are a lot of data science practices/applications that aren't worth the value they provide. There are others that are worth exponentially more than their operating costs. As with most things in statistics, whether data science is effective "depends."

Reply on news.ycombinator.com