Statistical Techniques Data Scientists Need to Master

Although I didn't downvote, I often downvote comments that are negative on knowledge. You know the type. If we're talking about CS they will argue that you don't have to know algorithms and data structures to be a good developer, because you can just go on stackexchange. If we're talking about data, they will argue that knowing what a p-value is useless because the methodology is flawed anyway. If we're talking about knowledge itself they will argue that going to university is a waste of time. And of course, no matter the subject matter, you never need "all that maths".

I was a "business analyst" rather than "data scientist", but this was before the term was a thing (late 90s) - I was an industrial engineering grad student hired to analyze increasing amounts of operational data to improve processes. Pretty similar thing.

One thing I learned from the experience (like you, I believe) is that collecting, managing, and disseminating data is very complicated. I'm glad you put "creative work" in quotations when talking about building ground up models, because I do believe there's a lot of actual creativity and decision making that goes into producing this data. By the time it gets to the model, often the most important assumptions and transformations have already been made.

Here's a story I like to tell to illustrate this - we were looking for ways to improve inventory management, and we noticed a huge spike in orders toward the end of each fiscal quarter. This was a supply chain management issue, a big deal in the early 00s (still a big deal now), and wide variations in demand make it more difficult and expensive to store the right inventory. We were all about creating a model to balance expensive of carrying extra inventory vs shortening lead times.

Except... it turned out here was no spike in orders! The system for adding to an order was irritating to use, so the sales people were just deleting orders and creating new ones. So if they'd had 10 orders over the last 10 weeks, and got 2 more, they'd remove the entire 10 orders from the database and enter a new one for 12 orders at the end of the fiscal quarter! We realized this only by meeting with people on the factory floor, tapping into different databases (some of which were not on the network yet), and finding a different way to track orders.

Experiences like this really do drive home a few things - first, how little you actually do know coming out of college/grad school (it's awesome that you know the mathematics around optimizing stochastic processes, but honestly, it probably isn't worth as much as being aware of a shadow database system on the factory floor that gives a much better picture of what's going on that the sales database that you were all poised to use as input to your optimization model). Second, it shows the amount of creativity, insight, and skills needed. Hell, an investigative journalist would have been more useful to our company than me, an engineer type who preferred to sit at his desk and code up nifty, "Creative" mathematical solutions. There are a lot, and I mean a lot of skills that go into being good at this. Math is important, and engineering degrees are great. But there is an 80/20 rule here. There are companies where knowing the outer orbits of integer optimization or stochastic gradient descent are helpful (my grad program, at Berkeley, seemed mainly interested with putting me through mathematical proofs about convex sets). But in most situations, I think competence in these areas combined with some qualities of an investigative journalist will be considerably more effective.

Something as simple as a text field for entering a date can produce interesting effects. Even if your data looks good on the surface, when you dig in, you might find interesting things. Like data entered when the system was definitely shut down for maintenance.

Data engineering cleanse data, manage workflows for ETL, build views, encapsulate rules and logic.

Data science is taking that cleaned up data and testing hypotheses.

Plenty of people can do both, and frankly it's very condescendimg when I run into a data scientist who prefers to outsource the engineering (because they're bad at it) so they can do "real work."

Source: I hire data scientists for Fortune 100 companies.