> grunt work
Honestly, it is an insult to call good data preprocessing skills grunt work.
It may not be as attractive, but requires a lot of understanding of the nature of data and way in which patterns present themselves.
Honestly, it is an insult to call good data preprocessing skills grunt work.
It may not be as attractive, but requires a lot of understanding of the nature of data and way in which patterns present themselves.
For what it's worth, I don't really see this as a problem in and of itself. Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data. But it's good that tutorials segment these out, since not everyone needs to learn both at the same time. Anecdotally, I had plenty of real-world experience dealing with messy data by the time I started learning data science.
As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation. Granted, they're still really time-consuming as it is, but there's probably some room for optimization there.
True, although ideally the data is cleaned/structured before it hits a data warehouse and the data scientist starts working with it. It's an iterative process (DS finds messy data, flags the problem upstream, repeat).
I do wish there were more tutorials with messy data; the ones I make I deliberately try to use complex datasets and highly relational ones, although as a result the tutorials are more complicated and more imposing for beginners.
> As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation.
When I was interviewing for DS jobs, I got a lot of the implement-binary-search-on-a-whiteboard questions which annoyed the hell out of me. But for the takehome assignments, they required some data ETL, which I felt was more representative. (some assignments had deliberately flawed datapoints the user was expected to identify and remove while not explicitly being told to do so)
They're just calling it like it is.
It's literally what the phrase means.
(lightheartedly; I'm not like disgusted over here :-) )
Edit: trivial isn't the right word, it's obviously an important step in the process but quite different from what you are really paying a "Data Scientist/ML Engineer/etc" to do.
It's not that putting the sculptor on the task would be a waste, nearly as much as that the sculptor doesn't possess any of the necessary skills.
How many people know how to find a source of marble, run a mine, and sculpt a great work of art? Roughly nil, but it's how many organizations seem to hire.