Data Science has a tooling problem | Hacker News Reader

Data Science has a tooling problem

counting.substack.com

46 pointskongxu10 comments

I think a lot of companies hopped on the "data science" bandwagon when they were really looking for data monkeys. A great data scientist may have more questions than answers, challenging assumptions and sharpening domain knowledge. But leadership/PMs aren't used to having to really think hard about problems, they're used to some person fixing the spreadsheet or "go get me number... plz ".

I hope as the field builds more hierarchy and data scientists have more organization decision making power, things could improve.

I once worked at a firm that ended up spending > 10 MM dollars because a very senior person did not understand statistical noise. There was a metric composed of ~600 human created data points with an error rate of 2%. Every week if the errors doubled very expensive ICs/managers would need to investigate. So the firm started spending more on trying to stabilize the metric - multiple applied scientists were hired to develop new statistical models. Engineering teams were funded to improve data collection. At its peak, there were probably 15+ engineers and scientists working on making sure that 12 errors out of 600 didn't become 24 errors out of 600 a couple weeks out of the year. This problem was compounded by trying to measure ~30 different segments with 600 measures per segment per week.

To this day I'm shocked at how much time that ate up. Expecting leaders to understand statistics was unlikely to work, and we shouldn't be surprised that 5 statisticians paid handsomely to solve the problem came up with ways to occupy their time.

I don’t have full context but why it does not make sense to invest in bringing down the error in measuring a very important metric ?

I used to barbecue using my experience(eyeballing it), and many times I was getting food cooked pretty well, but it was very unpredictable.

After I bought remote thermometer I consistently cook my food to perfect temperature.

Because the measurement itself was noisy, and unless the operational metric was updated, it would never be stable. I’ve experienced this hell, where a client is “red” or “yellow” based on an a phone call with an account manager. That assessment is one of many inputs to a flawed operational metric that swings wildly.

To extend your metaphor, imagine if you were constrained to only visually asses the food. Your fancy new remote monitoring system is simply a camera that allows a stupidly expensive pitmaster view your food and make an expert assessment. The result might get a little better, but it’s (probably) never going to be as good as using an actual thermometer.

Your story sounds outrageous initially (10 million dollars!) but if this is a 5, 50, maybe 500 billion company, that number isn't so crazy. If it's related to quality control of a manufactured product, large changes in errors would seem to be a significant problem. Moreover, if 12 errors can become 24 in a week and nobody did anything about it, what would prevent 24 from becoming 48, or even more, until hundreds of your metrics were out of line and in total chaos?

I'll also note you don't really explain "error rate of 2%." Do you mean, all 600 data measures are continuous variables, of which, each measured result may be off by 2% (or 2% is the std dev?), or are these discrete or binary measures, of which 2% are entirely wrong?

An Indian chief was asked by his tribe, "Winter is coming, what should we do?"

He told them to collect firewood.

When they brought that back they asked, "Is this enough?"

The chief had forgotten the wisdom of his ancestors so he called the weather service.

"Is it going to be cold this winter?" he asked. "Yes!" they told him.

So he instructed his tribe to collect more firewood. When they asked again "Is this enough?" He called up the weather service.

"So when you say it is going to be cold, exactly how cold?" They told him "All indications are pointing to a very cold winter."

He told his tribe to collect even more firewood. When they asked yet again if it was enough, he called the weather service again.

"Can you tell me how you know this winter will be so cold?" They told him "It's got to be. The Indians are collecting firewood like crazy."

This reminds me of an actual story about book that was being priced at hundreds of thousands of dollars (I looked for a link to this story but couldn't find it). The gist was that a feedback loop from two online book sellers, neither of which had a copy of the book. They automated the pricing such it was based on the other's listing plus a small markup; the idea being that if someone wanted this book, they could buy it from the other seller, add the markup and pass it on the customer. But since both were doing this, one would see the price add a markup and set it as their price. After a while, the other would then see this price and do the same thing. Over time the price just grew and grew.

I understand this sentiment as obsessing over tools can really wreck your productivity. However the pipeline of "incredible ML breakthrough/idea" to open source package is strikingly fast. To that end, the tools often end up unlocking things that were completely blocked or very difficult beforehand.

Yes, there are of course other topics that are not just "tooling" related. But data science does have a tooling problem. Lots of unsolved tooling problems, in fact. Especially when you're dealing with larger data science orgs, you start running into a lot of inefficiencies and problems that do need to be solved in a more systematic way. Making data scientists more efficient allows them to actually work on the more interesting problems and ask the questions that deliver value.

Appreciate the call for more honest and open conversation about the practical challenges of data science, and I believe that we as a community can benefit greatly from sharing our experiences and lessons learned with each other.

Reply on news.ycombinator.com