'Comically bad' datasets used to train clinical models for stroke and diabetes | Hacker News Reader

'Comically bad' datasets used to train clinical models for stroke and diabetes

retractionwatch.com

63 pointsleephillips12 comments

A lot of researchers think their job is to build models. They don't want to collect their own data, so they go find whatever dataset they can on kaggle or from a previous paper or wherever.

This is backwards. The model is the easy part. Getting good data is 99% of the job, and nearly any clown can make a good model once you hand them a good dataset.

As a clown, I can confirm.

If you hand me a clean, well-labeled, representative dataset, I can make the model do a respectable little dance by lunch.

If you hand me a Kaggle CSV with duplicated rows, target leakage, mislabeled outcomes, and columns named final_final_v2_REAL, suddenly I’m not doing ML anymore. I’m doing archaeology with a red nose on.

The model is the balloon animal. The dataset is the elephant you had to drag into the tent.

For a lot of clinical decision support use cases you don't even need fancy AI models to get accurate results. If you have good quality cleansed data you can literally just import it into Excel and run a simple linear regression analysis. But unfortunately that won't get you a reputation as an "AI thought leader".

You just need to figure out a way to brand that as a new, resource-conserving AI model.

I think it needs a cool name.

We'll call it SSLRM. Spread Sheet Linear Regression Modeling, pronounced SLURM. Sounds fancy and business friendly.

Actually a simple flow-chart works for a large number of use cases. That said, there are a lot of use cases where we don't have a simple way to run a linear regression model to get reasonable results where "AI" does seem to work well.

Im shopping dishwashers and many are AI branded because they have a dirty water sensor in them.

This holds in software as well. I encounter people trying to build solutions for problems that might not even exist, even in the context of addressing a specific bug. The act of measuring and collecting data is hard work, pretty boring sometimes, and often prescriptive in ways that aren't appealing. It's like we'd rather guess and use the ambiguity to allow ourselves to explore solutions we're more interested in. The alternative is manually profiling and poring through logs, so, I kind of get it.

So true and it's been like that for ages. It's why I called these people rogue data scientists five years ago:

https://ianreppel.org/how-to-spot-a-rogue-data-scientist/

Dataset quality is a huge issue in ML in general. You can often list a few dozen random samples from any given dataset and you will find out something weird going on instantly.

Old problems are new again.

Reply on news.ycombinator.com