Hacker News Reader

Comment by aabajian | Hacker News Reader

The author's pie chart showing data science to be 60% data manipulation is accurate. The biggest gap between good and bad data scientist is their comfort level with data wrangling. When interviewing candidates for data science positions, one of the simplest questions is to have them sort a 1 GB tab-delimited file.

1. Poor candidates will try to open the file in Excel.

2a. Marginal candidates will use R or Stata.

2b. Okay candidates will use a scripting language like Python.

3. Good candidates will use Unix sort.

To my knowledge, there are no university courses teaching the Unix toolchain and it remains very much a skill learned through practice.

View on HN

Not sure why you think a candidate who uses R is inferior to one who uses Python?

Also, a really good candidate should use the right tool for the job, so if you expect them to use Unix sort you should somehow imply a situation where that is the best approach.

I would also add: if it’s a one time thing, I would just do it in visual studio code or any other editor that doesn’t die on a 1GB file. And I have 12+ years experience with bash and unix tools, so it’s not about a lack of knowledge or experience. There isn’t anything magical regarding “sort” versus another tool, if there isn’t a need for automation they are equivalent.

> if you expect them to use Unix sort you should somehow imply a situation where that is the best approach.

I think the implied question is whether the interviewee is aware of the fact that trying to load a 1 GB text file could use up too much RAM space of the system. Unix sort is arguably the most memory efficient among the 4 choices there. It depends on the amount of available resources (which was not specified), and some companies might be willing to casually let people use 100 GB machines, though.

That is a fair point. A good script would be to move towards this scenario of more data than RAM and see what the candidate comes up with.

Demonstrating again that interviewing is mostly about confirming the prejudices of the interviewer.

In my experience the real meat of data manipulation/wrangling is in extracting statistical value from the data. If the data is noisy, what aggregations and what filters produce the best signal without smudging the underlying statistical properties?

While this is a good skill to have and is a good sign of efficiency, for me the deal breaker is how well the purpose of data transformations is understood and the care to which statistical value is extracted and tested.

This may doom me as a merely okay candidate but a simple paging strategy in python trivializes the problem and has the added benefit of probably being the "okay-est" tool for whatever the next transform is as well.

Why is chunking, with Pandas, a bad choice?

It's not. Welcome to okay-land.

Reply on news.ycombinator.com