- a single question is difficult to evaluate. "Answering a business question" is at the very end, usually, of a bunch of exploratory steps
- 60 min is reasonable but not much time to evaluate real work. You either need to expand the time (also a problem, for interviewing) or allow scoring of "what I'd do next"
- tooling familiarity is going to be a huge factor with short time. Are you testing general knowledge or environment knowledge?
- too focused on models, too "kaggle-like". That covers about 20% of the skills and job.
Here are the sorts of things I look for. Do they understand:
1. How to verify & validate data, clean inputs, handle coding errors and ELT type issues
2. How to evaluate data set issues like bias, missing data and outliers, and account for that (and when you can't)
3. (situational) How their infrastructure works an what they need it to do (e.g. for distributed training, if appropriate). How to use it effectively.
4. How to control data and code throughout lifecycle, so you don't waste time and experiments
5. How to choose between approaches and models
6. How to evaluate performance rigorously
7. How to monitor performance over time
but here is the kicker
8. How do you know you are trying to solve the right problem?
For junior people, the emphasis will be on the earlier points. For senior people the last point is key.
Your question partially addresses some of the early points only.
Off the top of my head suggestions.
- Have separate stages. Cleanup & verification can have objective and subjective issues (missing & corrupt data? Outliers?)
- Don't focus too much on modeling, it's the least interesting part.
- Allow different toolsets possibly (e.g. R)
- Initial cleanup/eval stage on a CSV, but following stage pull from SQL?
- Possibly allow multiple inference choices from same or a few data sets. Give a short list of things the "business" is interested in, they pick and describe why
- good idea to focus a bit on producing one/two graphics/tables to communicate to a lay audience.
- more focus on verification
- add a validation discussion requirement. How are you going to know what you did is worth doing?
- add a "next things I would try/do"
The latter is going to be text heavy but no way to avoid this unless there is a follow on voice/personal interview.
There isn't any way you are going to auto score this stuff reliably, so that's probably ok. Consequence is your evaluators are going to actually have to be good at this.