Show HN: Real world (Jupyter notebook embed) way to assess data scientists
hackerrank.com
hackerrank.com
The hard parts of data science include the following:
- choosing the right input data (rather relying on regularisation)
- figuring out what the consequences are if you're wrong in a specific way, and avoiding the bad cases
- wrangling your data into a nice CSV format
- handling missing data
- spotting biases in your data collection methodology
This ShowHN was to showcase the platform. Technically any csv/datasets/notebooks can be loaded in, and candidates for interviews can be asked to do any of the things you listed. The challenge you took was to showcase one specific example (regression)
Do you think the data set we have used doesn't do it to the extent you'd expect from a data scientists?
The only good end-to-end "technical" data science assessment I can think of is to pose a broad question or business problem that's addressable by applying data science techniques to publicly available data. But a nontrivial version of that assessment would take half a day on the very low end, and long assessments anti-select against good candidates.
IMO, when it comes to evaluating data scientists, the only thing that online coding assessments are good for is to ensure that they can perform basic coding and data manipulation tasks. (I'd include tasks like web scraping, image manipulation, API calls, and ORM stuff in this category). Everything else needs to be evaluated in person.
We optimized these challenges to allow candidates to show as much of their skills they can show in a timed window, without killing their creativity. I'd be curious to know what do think is a good way to interview data scientists.
If representative isn't an option, highly correlated is the next best thing. In practice, for my team specifically, this means screening for math aptitude and general business acumen during a phone screen, data manipulation (moderately complex SQL + tidyverse/data.table/pandas) during a "take-home", and delving more into problem solving approach, model selection and validation, etc. during an onsite. Broad business questions (e.g., "How does a life insurance company make money?") and communication skills generally weed out the candidates who picked up the bare minimum math and programming background through Kaggle + MOOCs.
As an aside, I absolutely think that the sort of assessment in the OP kills creativity. I care a lot about whether a candidate would think to include covariates like Internet usage and segmented urban population when predicting mortality rates; I don't care at all whether they're able to write the trivial amount of code that's needed to include those covariates in a model, given a data set that already contains them.
Tools to hire data scientists and going to continually fail until they realize that the interesting, hard part of being a data scientist is closer akin to a business lead (which can't really be tested in 60 minutes).
Concrete feedback:
- You ask for writing and descriptions on why a model was chosen, why features matters - are you grading this automatically? That would be a feat.
- The task is waaay to easy (even if you do believe there is a market for identifying people who can predict well).
- Python is overly limited. Why not SQL or R?
> You ask for writing and descriptions on why a model was chosen, why features matters - are you grading this automatically? That would be a feat.
Grading is apparently not automatic, which is good as I am not a fan of the Kaggle approach in this demo.
> The task is waaay to easy (even if you do believe there is a market for identifying people who can predict well)
You'd be very surprised about how candidates can respond to these types of questions!
> Python is overly limited. Why not SQL or R?
The full product allows Python, R, and Julia, with popular packages preinstalled for Python/R.
- We are not automatically grading it. We have learned that in the past that trying to automatically grade candidates on such challenges biases their approach, which breaks the point of a good data science challenge.
- that's good to know. we are not really focusing on the final outcome but how creatively a candidate can go about the problem. the dataset allows for some good amount of creativity.
- Ah. can you elaborate what you mean by overly limited? We do support R.
If I'm given this clean dataset with all of the features properly set in columns and data types labeled, I could spin up Azure or Google Cloud's ML capabilities and have them run gridsearch and optimize my model.
To test data scientists, it seems like it's generally falling more into the buckets of people that can pull analytics and query databases to create the datasets and features OR people that can build the infrastructure to serve models, engineer pipelines, etc...
FYI though we're working on this now at https://www.interviewquery.com to try to start creating suitable tests to assess data scientists without having them do 10+ take homes every month.
That's the fault of the test design allowing such techniques without scrutiny, not with the Notebook format.
That said, this demo should have a several hour time limit.
Average marathon time is about 4.5 hours. Let's say you expect you're hired to last around 9 years (average is 8 according to GlassDoor but you're woebegone)
So scaling linearly, for every hour you get to assess a job candidate, you'd get 0.2 seconds to evaluate a potential marathoner.
Assuming a typical candidate gives you maybe 12, 24 hours tops, you have all of 3 to 5 seconds to evaluate a marathoner.
The futility of such an exercise is obvious.
The only solutions are to:
* insist on a longer evaluation period
* hire them on a probationary basis
* only hire people who have ran official marathons before, with proof and such
I leave it as an exercise for the reader to determine what's right for their own evaluation process.
I would want to see a short script to clean and predict a dataset, plus a small description of why choices were made.
Wouldn’t care much about the performance of the model.
It would work great with other learning tools, like MOOCs, datacamp, dataquest.io, as part of an overall data science learning process.
I'm more skeptical of its ability to help companies select candidates, but I could be very wrong about this and if I am then it's a huge win, so thanks for developing it.
I am super interested in seeing how you all develop this in the future,there's a lot of potential here. Is there a data science specific mailing list I can sign up for? I honestly, have zero interest in hiring for other roles so I am not going to sign up for a general mailing list.
https://blog.hackerrank.com/hackerrank-projects-supports-dat...
In fact I'd say it is a bit aggressive for a 60 minute challenge.
Quite a bit of data wrangling is expected to complete modeling on all columns. Some regex knowledge would help here too (for example for wrangling internet_users column)
What was the idea behind asking for 20 most important features when we have 16 columns? Is it expected to do some feature engineering?
Disclaimer: I teach Python and basic Data Science to adults and I'd say most people would struggle to complete this in 60 minutes including myself.
There is indeed some feature engineering involved. The challenge in the test can indeed be solved in the most obvious way possible, as well as in the most creative fashion. We believe how a Data Scientist goes about solving the problem was more important than a fixed outcome.
The best way to assess technical problem solving is a structured hackathon. That is, to be given
* a problem with multiple subproblems and solution milestones
* with both objective and subjective criteria
* freedom of tools
* a "junkyard" of resources
* a fixed amount of time for each deliverable of the problem
And then you observe the process and the results.
For data science, the subproblems should be:
* requirements gathering / understanding the problem
* data acquisition, prep, and analysis
* refinement of requirements / communication
* feature engineering
* modeling
* presentation / storytelling / viz
- a single question is difficult to evaluate. "Answering a business question" is at the very end, usually, of a bunch of exploratory steps
- 60 min is reasonable but not much time to evaluate real work. You either need to expand the time (also a problem, for interviewing) or allow scoring of "what I'd do next"
- tooling familiarity is going to be a huge factor with short time. Are you testing general knowledge or environment knowledge?
- too focused on models, too "kaggle-like". That covers about 20% of the skills and job.
Here are the sorts of things I look for. Do they understand:
1. How to verify & validate data, clean inputs, handle coding errors and ELT type issues
2. How to evaluate data set issues like bias, missing data and outliers, and account for that (and when you can't)
3. (situational) How their infrastructure works an what they need it to do (e.g. for distributed training, if appropriate). How to use it effectively.
4. How to control data and code throughout lifecycle, so you don't waste time and experiments
5. How to choose between approaches and models
6. How to evaluate performance rigorously
7. How to monitor performance over time
but here is the kicker
8. How do you know you are trying to solve the right problem?
For junior people, the emphasis will be on the earlier points. For senior people the last point is key.
Your question partially addresses some of the early points only.
Off the top of my head suggestions.
- Have separate stages. Cleanup & verification can have objective and subjective issues (missing & corrupt data? Outliers?)
- Don't focus too much on modeling, it's the least interesting part.
- Allow different toolsets possibly (e.g. R)
- Initial cleanup/eval stage on a CSV, but following stage pull from SQL?
- Possibly allow multiple inference choices from same or a few data sets. Give a short list of things the "business" is interested in, they pick and describe why
- good idea to focus a bit on producing one/two graphics/tables to communicate to a lay audience.
- more focus on verification
- add a validation discussion requirement. How are you going to know what you did is worth doing?
- add a "next things I would try/do"
The latter is going to be text heavy but no way to avoid this unless there is a follow on voice/personal interview.
There isn't any way you are going to auto score this stuff reliably, so that's probably ok. Consequence is your evaluators are going to actually have to be good at this.
Counter-intuitive in regards to the phrase but good ds is people work first and data work like 9th.
Can you elaborate?
For certain data science usecases, evaluation is possible by using a CSV output bu a user, and comparing that to an expected CSV.
(I worked on the product).
I should have asked this before, but "data science" is a pretty broad term - who are you hoping to target with this? I'm guessing for pretty junior positions but want to clarify.
Oh, and one other question, do you /can you enforce the 60 min time? [edit: never mind, I answered this experimentally, you do cut it off at 60 min]
What skills would you consider important for senior vs junior Data Scientist?
For what it's worth, I think junior and senior DS roles should have fairly different evaluations & interviews.
The question then is how do you algorithmically (or even just consistently) distinguish a great data scientist from one who can accurately model answers to a question that was badly thought out?
Plus as pointed out before, the length of a take home could reduce applications from the most qualified candidates.
Wonder if this should be even shorter and more quiz/fun like so it intrigues rather than annoying more senior applicants, and still wondering the best way to identify the data scientists who ask better questions.
This meant we couldn't automatically score/rank a candidate's solution. We do provide them with an evaluation metric in the problem description (Mean Absolute Error). Here is scoring rubric we provide to the interviewers when they review the submission - https://d.pr/i/hNYY0u
Would love to hear more opinions on our scoring rubric
[edit]. Initial thoughts:
- "data wrangling" scoring difficult given this task - more weight to "rationale", that's more important the "performance", here.
- not enough focus on communication capabilities
- really need something on validation
- "proficiency" measure you use is pretty much impossible to accurately evaluate from your example question
- way too much weight to modeling section overall
Of course more abstractly you want to validate the entire process - but I was referring to those two, as it is hard to see how to address the latter in a format like this.