How to Become a Data Scientist – On Your Own (2015) | Hacker News Reader

How to Become a Data Scientist – On Your Own (2015)

datasciencecentral.com

141 pointsrbanffy44 comments

Becoming a data scientist on your own is exceedingly difficult because, despite their purported adherence to objective data above all else, the practice of data science is full of people who consistently appeal to authority via educational credentials. You can see it in this thread. They regularly make the mistake of thinking that because the skills necessary to be successful in the field correlate highly with advanced degrees that means that only people with advanced degrees should be able to participate in it. They generally make it very difficult to objectively evaluate an individual's skills because their injection of bias into the candidate evaluation process.

Its regressive and completely out of step with the supposed meritocracy we like to think we follow in tech. Its also the path towards cartels. I get the feeling a large portion of data scientists would like to create the American Data Scientist Association, with credentials and bar tests.

Your comment is really refreshing, thank you

Yeah I kinda get that feel as well. The thing that makes me suspicious is the amount of unnecessary and obfuscating jargon that gets thrown about. They've even invented new jargon to replace perfectly confusing old jargon (e.g. your model's "error residual" is now your "function cost"). I've spent the past couple of weeks doing a bit of ML vision stuff. Most of the terminology was lost on me, at least until today when I discovered "Machine Learning is Fun": https://medium.com/@ageitgey/machine-learning-is-fun-80ea3ec...

I think I learned more in an hour than I did in the past week, thanks to this series. The author actually bothers to explain concepts (that turn out to be fairly simple btw) like 'gradient descent'. Highly recommended read if you have the time and interest. Just to whet your appetite:

...current machine learning algorithms aren’t that good yet — they only work when focused a very specific, limited problem. Maybe a better definition for “learning” in this case is “figuring out an equation to solve a specific problem based on some example data”.

Unfortunately “Machine Figuring out an equation to solve a specific problem based on some example data” isn’t really a great name. So we ended up with “Machine Learning” instead.

First of all, errors and residuals are different things.

Second, "cost" is not new jargon. What is relatively new is thinking about probabilistic modeling in terms of abstracted cost functions, but only relatively.

There are dozens of tutorials, courses, etc that are clear and don't introduce unnecessary jargon. Nobody is trying to keep you out of data science.

As for the term "machine learning," it's because what we today call ML gree out of actual AI research. It so happened that a lot of progress was made very quickly by the ML researchers, so the ML-oriented terms became popular as some older statistics terms were subsumed.

This is true of most domain. The jargon often gives the impression things are really complicated, but often time it's just that you don't know the domain language. Even with math, 60% of the challenge is remembering the notation and substituting it in your head with the real construct.

Now, as you get into advanced topics of any field, you'll start to find difficult concepts, but 90% of the time, the foundation just appears hard because of the language you don't know.

Also, keep in mind Machine Learning is not "Machine Figuring out an equation to solve a specific problem based on some example data”. Machine Learning is a subset of AI which focuses on teaching the computer how to perform a task without explicitly programming the task execution logic. Currently, the known practical technique of doing so can be described as "Machine Figuring out an equation to solve a specific problem based on some example data”. This might not be true in the future, as better ML techniques are researched and discovered.

Also, it's good to keep in mind ML is a software engineer discipline, and data scientist just benefit from modern software, the same way Excel created jobs, this technique of ML created jobs. In the future, different ML techniques might create more jobs or replace the current ones. This puts the data science field at the mercy of ML research. Already I think it's been hard to keep up as a data scientist, since ML research is being financed greatly and advanced quickly.

One of the main aspect of Data Science is... Science.

I know some people who I would qualify as scientist without a PhD, but they are extremely rare. On the other hand, I've seen countless unqualified people apply and get (lousy) data science job because the "title" itself is very vague. I'd be more inclined to hire a PhD (who's been through a fairly painful scientific training) in, say, material science, then train her to data/programming, than get a good programmer and train her for science.

I'm not for a "Data Scientist Association"[0], but I'm not for diluting the value of all the effort I made to effectively become a data expert AND a trained scientist.

[0]Rant: there is a world outside America.

I have the opposite problem: I constantly feel inadequately prepared for my job as a data scientist because I never got a PhD. I know mid-career doctorates are considered risky but I still have my mind made up to get one.

The PhD is irrelevant. What is relevant is the experience of doing science in a rigorous fashion. Today's "data scientist in a box" lessons don't give people the basic understanding of how to approach a problem or even what a null hypothesis is. Some of the videos on Youtube make it seem like any correlation is relevant when they don't even know what they don't know.

I can't agree more: PhD shows that you went through the pain of scientific training, you can however train yourself to science in different manners.

Start from the start though: epistemology is grossly underrated, but the scientific method (aka: calling bullshit) is my most valuable tool.

We (Class Central) have been working on a Wirecutter-style guide on Data Science. Instead of presenting a list of resources, to try to recommend the best resource (mostly a MOOC).

Its a six part series, and so far on the first two parts have been published:

Part 1: The Best Intro to Programming Courses for Data Science [1]

Part 2: The Best Statistics & Probability Courses for Data Science [2]

Any feedback would be appreciated.

[1] https://www.class-central.com/report/best-programming-course...

[2] https://www.class-central.com/report/best-statistics-probabi...

Welp.. you just implemented one of my 100's of startup ideas. And by the looks of it, you did a pretty good job too :) So, kudos the site looks awesome and I think will definitely fill a need people have.

A "what you will and what you will not yet be able to do afterwards" would be nice. Especially in regards to jobs and projects, not just details about what is covered in the courses.

I think the author of this post is unintentionally misleading. Becoming a data scientist is not a passive activity that can be taught solely through coursework, the only time the author mentions real-world applications is in the point mentioning competitions.

Every data scientist that's worth anything has either done a PhD or would be capable of doing a PhD, the distinguishing characteristics between PhD's and standard coursework is the incremental effort navigating uncertainty.

In the end, Data Science entails a great deal of uncertainty that makes most people uncomfortable.

disclaimer: you may need to have a masters in CS or Statistics to be taken seriously. For every success story you hear of someone "doing it on their own", scrutinize it enough and you'll see that they had either a decent educational background or support from a career facilitator (bootcamps).

Any hard Engineering branch, Mathematics, and some of the more rigorous Biology stuff will do as well.

I honestly do not understand why there appears to be so much desire to get into Data Science when becoming a Programmer is equally lucrative and substantially easier to bootstrap into.

Edit: Seriously, programmers make as much if not more than Data Scientists for what ends up being substantially less stressful work (all things being equal). I suspect if the people pursuing DS actually ended up doing the work and living with the responsibilities they'll regret their time investments.

there is no 'programmer' role in the c-suite, but there are chief data scientists. data scientists have the ear of top management, and have direct interaction where that is rarely true with programmers, which leads to...

there isn't a silicon ceiling on a ds pay like there is with programmers, and I disagree they are equally lucrative. I have never seen ds roles that were not substantially paid more than programmers; although with the explosion of the ds role, there are plenty of sub-par ds positions out there. (according to glass door, the average programmer makes 70k, the average ds makes 128 in san francisco). That disparity even holds for large tech like facebook.

as far as 'less stress', I believe that is subjective. some people would like to program, others more ds stuff, and often ds and programmers get to do a little of both.

> there is no 'programmer' role in the c-suite

CTO?

> but there are chief data scientists. data scientists have the ear of top management

Have you worked in or near a c-suite before? Because both of those statements don't align with the reality that I've observed.

>there is no 'programmer' role in the c-suite, but there are chief data scientists. data scientists have the ear of top management, and have direct interaction where that is rarely true with programmers, which leads to...

After working closely with dozens of tech companies, I have to say I've never seen a single "Chief Data Scientist." I also can't say I've even heard of a single company that has one (I'm sure some exist though). I have seen a Chief Technology Officer in virtually every tech company, which is essentially "programmer role in the c-suite" for the purposes of this discussion.

Furthermore, in the companies I've worked with that had in-house data scientists, they always treated them less well than the software engineers developing products.

I guess what I'm trying to say is that your statements don't match my experience, or the experience of anyone I personally know in this industry, and I'd be interested to see where your experience is coming from.

I've never seen a CTO write any code. That's not their job.

I anticipated a comment like this one, which is why I explicitly said "for the purposes of this discussion."

You're right, a CTO doesn't usually write software, a CTO manages programmers who write software (or VPs managing teams of programmers, etc). But a CTO generally comes from a coding background, and how much data science do you think a "chief data scientist" is really doing, as opposed to managing other data scientists? People in the C-Suite typically don't really do anything other than manage people managing others in the same background they came from.

I think the spirit of my point still stands, pedantry aside. There clearly exists a commonly used and recognized c-suite role for programmers, whether they use their programming ability hands on or in managing others. It's not at all clear to me that there is a commonly used nor well recognized c-suite role for data scientists that would be distinct from CTO.

As a category of employee and work division, data scientists have not yet become distinct enough from cross-polinated disciplines to have that sort of representation.

I know that at least one former Amazon CTO was an excellent coder, although he didn't really do much (if any) while as CTO.

I don't know of many companies that have a "Chief Data Scientist" that reports directly to the CEO. In all honesty, they're more likely to report to a CTO.

Also, there's a reason why the C-suite people have the word "Officer" in their title as they're officers of the corporation and that implies additional legal responsibilities. It's not necessary that it be in their official title, but it typically is.

http://techblog.netflix.com/2015/09/john-carmack-on-developi...

John Carmack, CTO of Oculus and programmer extraordinaire, to bring our TV user interface to the Gear VR headset.

Well, honestly, John did most of the development himself(!), so I've asked him to be a guest blogger today and share his experience with implementing the new app.

Okay, so I'm equipped, but how do I find those jobs? By thew way, I'm making more than the 128 as a developer already, and not in expensive Cali, so is it even worth it?

I think any time you're talking with c-suite type people, the stress level is pretty high. That's been my experience anyways.

When did you last look at Glassdoor?

The avg entry level software engineer makes 110k in San fran, and 95k in the US as a whole.

It's 128k in San fran for a data scientist and 113k in the US as a whole.

I'd suspect the Data Science entries are skewed upwards, as most of them are employed by big companies, while software engineering has a larger small to medium sized employment prospect which probably bring down the average in comparison, but brings more job opportunity.

I think you searched for programmer on glass door. Programmers don't even need a degree, just a boot camp and you're good to go. I'm not sure data science has an equivalent, maybe business analyst? That averages 69k in San Fran.

Also, if you look for Machine Learning Engineers, a specialty of Software Engineering, they make more then Data Scientist, with a 140k avg in San Fran and 122k national.

For me anyways, data science seemed like more interesting work (building predictive models, investigating trends and telling stories through analytics, working on deep learning) than pure programming.

I dunno which is easier to get into with a hard science background, but it took me about 1 yr of being an analyst to get a good ds job.

so do you have a "masters in CS or Statistics"?

No, phd in physics

ok, but still some science phd. thanks!

> data science seemed like more interesting work

That is the only good answer to why one would want to be a DS, if you don't find the work interesting you'll never succeed.

So if I have a masters in CS, but in a different discipline, will that be okay if I'm now wanting to shift focus to data science through a series of my own side projects utilizing ML?

My advice is unless you can locate jobs doing ML stuff, it simply isn't worth it.

Agree that an advanced degree really helps, but in my experience it could be in any quantitative field. If you're getting a MS just to become a data scientist, I agree that CS or stats are the most natural choices, but I've seen great candidates whose degrees were in physics, biosciences, psychology, math, economics, etc. (some of whom we've hired or given offers to).

The key is that the degree has to involve actual work with messy big real-world data with a lot of uncertainties. It's possible to build that kind of experience on your own without doing an advanced degree, but it seems to be rare based on candidates who apply to work with us.

And ... where does it say learn maths?

this article has a great list of people to follow on twitter, if anyone has ones to add be sure to post below =)

Olson is a great one, especially in GIS here is my contribution: https://twitter.com/randal_olson

I'm going to give you all a small dose of reality regarding Data Science. Are you ready? It's being hyped to the max to sell courses, books, seminars, and what have you. So far, the number of data science jobs available in most cities is at least for now not in line with the MASSIVE hype taking place.

I would love to have a job doing Data Science: I have a PhD in a relevant field so I recently pushed down hard on this area. I took the Coursera course, I'm learning all the various Python libraries, I learned R, and do anything I can every day to pick up a skill here or there. I even have a "Kaggle" account. What I don't have are job leads because there aren't actually that many jobs and the ones that do exist say "data science" but really mean other things.

To illustrate how ridiculous and self serving this is, replace the word "data" with "cancer." You cannot become a "scientst" by watching ted videos. At a minimum, it requires a credentialed degree. Otherwise, I'm a data scientist too.

Do you need a degree to become a computer scientist / programmer?

I will position this first before I go on: I think in theory, you can learn almost anything on your own, I'm now going to talk about the reality that I feel is true for most people and things based on my observations and observations I have seen, heard, and/or read about online.

I would argue not programmer, but perhaps computer scientist.

They are fundamentally different things to me. Programming, and its human facilitator, the programmer, can certainly be learned without a degree. I can teach myself to program fairly well in say, Python, in a few months.

What I can't teach easily, in my opinion or rather what can't be taught easily without some uni resources (going to college, maybe not, but to be honest i think the learning format has some advantages here), is say how to proof formal methods, Computational geometry, higher levels of information theory. Quantum Computing. all realms of computer science. Yes, lots and lots of CS depts teach you how to program in languages, but the ones I find that don't burn out in the long term aren't merely programmings, but have a strong understanding of the discrete mathematics that make up a lot of our modern systems.

I could go on, but I feel like its going to go into rant like an old grump territory.

I do have a bone to pick with this particular article as well:

"I will write separate articles on Data Science Books (I’ve read 127 of those in last six months)"

Unless those books are 20 pages long, you have not read them. Skimmed maybe, but completely read and logically understand the implications of those books? I have to call foul on this.

Hotly debated subject these days.

I personally learned programming on my own, and after about two years of doing it, I went back and started taking some computer science courses in data structures, discrete mathematics, algorithms as well as some other topics. I took some coursework through the University I got my undergrad from but most through local community colleges because they were 1/10th of the cost.

In my experience, I do not think you need a degree to be a programmer. You need to have extreme grit and motivation to learn it on your own.

I took the coursework after doing it because trying to learn advanced computer science topics on top of work in my own time simply wasn't working. It's not incredibly fun to learn, dissect and implement algorithms. At least for me it wasn't. Having no one to ask about advanced mathematics also sucked honestly. For those reasons, a quality education or professor is worth their weight in gold.

As someone who came up through universities with the full traditional CS background, and as someone who has hired and been a tech lead over many developers, I can count only one person I know who didn't get a degree who is a great developer. The people with degrees all had to learn a lot after school, as did I, but the one who is self-taught is some kind of savant, I kid you not. And as great a developer as he is, he had some holes in his knowledge that I ran across from time to time.

In the old days (a millenia ago) Universities (teacher's guilds, literally) didn't test students - the Church tested the students and credentialed them, not the teachers. The great advantage of this was that there was no grade inflation or other funny business, 'cause there was no conflict of interest. Now, the academy grades its own work, in effect. Back then, if you could get the study in on your own and pass, great. If not, you found a professor and settled on a fee. Now, the academy is an old boys' club, and can get very inbred, and be a decade behind the market, too. In computer science, if someone's a prof at most Universities, it's often a sign they aren't competent enough to survive in the wild.

I know many profs who decided to switch from industry to academia because they had enough money and wanted a less hectic lifestyle. A very large number of professors actually do contract work for companies.

Even a former Master's supervisor of mine (who is officially retired) still does consulting with companies (typically game developers as he specializes in splines in computer graphics).

I'm actually in Engineering and at least in Canada, there's actually an accreditation body that reviews the engineering programs to ensure they live up to a standard. If the program meets that standard, the graduates only have to take a law and ethics exam (in addition to the experience requirement). If you're outside that system, you have to take a series of exams on the material. I know that it's different in the US, though. I believe that you do have to take a set of exams on the core material for your type of engineering as there's too many programs to properly vet all of them.

Certainly (many) tests being taken out of the hands of individual profs and individual Universities is a good thing, so that's heartening - but in fast-moving fields I admit it could mean the test lag knowledge and best practice a bit more. As for excellent peeps choosing to be profs for the lifestyle: for a long while I've been saying that engineers/coders not be exempted from overtime; your response suggests that the unintended effect of that might be to harm teaching quite a bit, if those profs go back to a friendlier industry.

Reply on news.ycombinator.com