One Year as a Data Scientist at Stack Overflow

Depends on the stage the company is at... I've a similar background, but took up a Data Scientist role. However and here's the rub. The company has very little data-pipeline setup and functioning , that much of the work is laying out the plumbing, ensuring it works without getting blocked etc.. I've tried my hand at a few predictive models and cross-validation accuracy looks good, but can't test them on real-life scenario, as some of the data-source it depends on is not coming in. What's worse is they don't have a capable and dedicated devops, so it's the CTO who had started this data puller project and hasn't delegated/handed-over either. So I'm reduced sitting on my ass, warming the seat/chair.

You could absolutely learn all the skills and knowledge to be a good data scientist, the real question is if you could get hired. I have been looking into data science and I think the demand for data scientists is not as large as it seems, but it is growing. There are very few junior data science positions. Especially without a PhD or an inside contact it could be challenging to get your foot in the door. My advice would be to check out Kaggle, try out a problem and see if you enjoy it.

There are multiple Data Science boot camps. The fact that these places exist and claim to be able to get people jobs after three months learning suggests you could do it. Look at Signal Data Science.

I agree with the previous messages. There are basically two paths: A) Learn from others. There are some good programs like Data Science Retreat. B) Self-learn and use Kaggle competitions to prove it.

Totally realistic. It'll depend on what the company is, and what they're looking for (and hopefully, this will match what they should be looking for).

While people can often focus on applying the latest deep learning thought-vector approach to their BIG DATA, there's an enormous gulf between the common condition of data and this aspiration.

You don't need PhD level stats and machine learning to apply the things that many companies actually could benefit from. Storing, maintaining and managing the data properly is a start. Then working with the people in the business to get insights from the data they have. Often simple aggregations and visualisations can provide enormous benefit. Being able to show correlations, and sometimes even just being able to show how noisy things are can be important.

Big questions: How is reality different from what we think? How are these differences important for the company?

That might be a correlation that we don't expect, or a lack of one we think is there. Part of a next step might be to become more "pro-active" and design new experiments to answer questions that can't quite be answered yet due to a lack of data. Beyond that you're heading towards bringing a new feature into a product.

Does your current company have some data? Do they spend a lot of time emailing spreadsheets around, or have salesforce or a proper database? See if you can make something useful for your work that's based on an analysis of that data (and possibly do some work to smooth those workflows of passing data around).

To take a page from the authors playbook...create some artifacts. Find interesting data sets, do interesting stuff with said data sets and write about it :)

Absolutely realistic, though it helps to have good intuition about math. I've seen smart people without data science experience make the leap after a 12 week boot camp, or finishing the Johns Hopkins Coursera certificate. If you're good at picking stuff up online, the latter is probably the best start.

There is such a shortage of data scientists that most companies will settle for 2 out of 3 of the trinity of Stats, Programming and Domain Knowledge. If the candidate is smart enough, sometimes 1 out of 3 is enough.

Hey, I'm the Jason Punyon drob's gushing about in the post. I wanted to tell you that this is 100% totally realistic, everyone on the Data team at Stack Overflow besides Dave Robinson is living proof.

In the beginning the Data Science Team at Stack Overflow was just me and Kevin Montrose. We're both ~30, we had some skills that were data-ish, but they were general critical thinking and math skills. By no means were we statisticians or data scientists by trade like Dave Robinson. However, the data at Stack Overflow was very amenable to analysis, and we were able to ship a personalized machine learning product after a year of work.

That year was plagued by (my, mostly) engineering failures (http://jasonpunyon.com/blog/2015/02/12/providence-failure-is...) and a bit of unfortunate lack-of-empiricism, not "data science" problems. And post launch of Providence, the engineering remains the bigger issue. It's just hard to get this shit right. Our record on getting experiments right the first time is spotty, and at Stack Overflow levels of traffic you might be dumping a week or more of time on a single screwup. We've had to rerun many experiments because of various things going wrong with the code (screwed up the A/B testing code, not counting the right things, visual issues with the ads we didn't catch during design, etc etc etc). There's a litany of things you've gotta get right engineering wise before data even comes into the picture.

So if you're gonna take the plunge, know that it's 100% possible but 100% difficult, and maybe not for the reasons you think.

I hire plenty of data scientists - if I had to give a single piece of advice: gain a deep understanding of the techniques you're using.

it's not sufficient to say "oh, we used a support vector machine", and when prompted for more detail about how it works to shrug and say you just copied and pasted code from the internet and tweaked parameters until it worked.

I'm also a self taught programmer. I recently started using https://www.dataquest.io/ (not affiliated with them) and have found it to be a great data science resource for people who prefer a hands-on approach to learning. Previously I'd tried various MOOCs but the lecture format with only a few questions to work on was not helpful for me.

I think the concern is less that it's a good tool, and more that it's one invariably receives emails with attached documents that open well only in Excel.

Excel is a great tool, but its graphing ability leaves a lot to be desired. I feel that graphing in Excel has gotten worse as they have tried making it easier to use as well. I'm 99% sure the graphing engine in Office 2007-2016 is the same one as in 2003 and earlier, but all of the damn menus they have added slow down the graph making process so much. If you are trying to use Excel to make a graph for a reasonably technical audience you are going to need to make a lot of tweaks to the graph to make it acceptable, and tweaking each part of the graph takes dozens of clicks where it used to take one or two.

I personally think that Excel is one of the best tools I have ever used. I do about 40% of my work in Excel. I graduated from college with a degree in biology, and I spent hundreds or thousands of hours in Excel manipulating data and graphing. Excel can and will graph almost anything you need to graph, but anything more complex than the simplest line graph requires getting creative with the formatting of your data and how you use series and data sets. The maximum complexity of a graph in Excel is technically restricted by the 254 series limit, but creating a graph with 254 series will probably take the better part of a week.

So ultimately, Excel isn't a bad tool, but when graphing it's a bit like using a shovel to dig a hole. It has a time and a place, but at a certain point using an excavator will be faster, safer and less expensive than digging the hole by hand.

Nothing "religious" about that observation; quite the opposite, in fact.

In particular: it's not so much whether the tool "works" or not, but the side effects (how well it plays with other tools; and hence, the extent to which it can be integrated into larger, maintainable projects) that need to taken into consideration, also. Which is why better engineers detect more than a whiff of bad "design smell" when they hear someone say "just use Excel" (most of the time).

I wouldn't call it waste if it helped him land the job.

The article is a very nice write-up, but in the end he is still optimizing advertisement click-through rates (a zero-sum game).

While interesting from a technical point of view, why not make yourself 1000x more useful to society by working, for example, on "cognitive health" problems? These are problems that lean heavily on statistics, and are interesting and imho more rewarding at the same time.

I'm one of the Ad Server devs that David mentioned in his post. We have plans in the works for allowing a user to specify what technologies/tags they're more interested in seeing jobs for, as well as things like customizing the geographical location (if any) you'd like to see prioritized. We hope to roll those out this year (we're a small team - just got our 3rd dev)

If I were to work in your place to try to match people to jobs I should study some psychology and NLP trying to predict from what an user write and answers what is his mental state and how that mental state fits in the jobs.

Unfortunately hiring seems to be a very difficult problem and I can imagine that many key features are hidden and can't be obtained since the candidate can't be put into a controlled experiment that should allow us to obtain deep information that is usually hidden, perhaps a way to gain more information and wealth is to communicate with your users and clients in such a way that what is now hidden can be measured and new features can be obtained. That is you need to think about a model for your users, and that model must use features related to mental and human capabilities.

That is a heavily-filtered network: if he didn't drop most of the weak connections he'd end up with everything connected to everything else. On the other hand, if you don't use a sophisticated way to define what a "noisy" edge is, you'll end up with some curious cases like the one you point out. He might have used a naive global threshold -- it's the easiest way to go about it: drop all connections with weight lower than x. But it's also very wrong most of the times :-) Something like the disparity filter [1] works usually well, although you have sure that its null model hypothesis is aligned with what you think is the generative process of your network. The field is "network backboning" and it's a nice one from network science.

[1] https://en.wikipedia.org/wiki/Disparity_filter_algorithm_of_...

Remember all this data comes from Stack Overflow. Some of the clusters might look a bit odd, but that's because they're all derived from the relationships we've found in Stack Overflow content.

mikk14 is right; the connections were filtered out below a particular threshold (set that threshold too low and everything is connected to everything; set it too high and you miss meaningful connections).

But note that precisely because git/github is used in combination with almost all other tags, it doesn't have a high correlation with any particular technology. A correlation (roughly) means "If I know you use tag X, you're more likely to also use tag Y." But knowing someone uses git doesn't let you guess what other technologies you use, because as you note they can use almost anything.

In short, if you're connected to everything, that means you're correlated with nothing.