Can synthetic data help train your AI model?

However when other variables came into the mix (different tones, ethnicity, participant age etc), the model predictions were up for tosses.

Now, medical data is much more harder in ML. The variability is quite extreme since all our bodies are little different from one another. In my thesis, I very firmly proposed that generative models should never ever be used in synthetic medical information as part of the thesis proposal - and there were those field studies to back it up. (The thesis is under non-disclosure at University of Tokyo until September 2023 since an embargo is involved from participating hospitals & data sources. I would have been eager to share otherwise). I am glad people aren't wildly trusting if task A, B & C are doable, task AB, CA, ABC, BC etc should be similarly tractable.

Building a generative model allows you to sample observations from a known distribution and confirm that your approach to learn is even able to recover the correct parameters. It's used as a basic sanity checking tool.

Synthetic data is great for this purpose. Before setting out on a complex and expensive modeling task, at least verify that in the best case (i.e. your model is literally the data generating process) you could even learn the specific problem. It's surprisingly easy to define models that are fundamentally not learnable.

However the idea that you could learn from synthetic data doesn't make sense, as you could just literally use the model that generates the data in the first place. If you're just learning known parameters, why waste the time training when you already have the answer?

Using real data that has been heavily anonymized and stripped of PII (perturbing dates, hashing strings etc) makes some sense, assuming that PII contains no information (which is unlikely). But the more clever you're trying to be with synthetically generating features the more your are essentially just manually building a model of how the world works with obviates the need to learn a model anyway.

The issues that come with generative models are not unique to machine learning. Here's a human example: imagine you are a radiologist with 20 years of experience. Throughout your career you have specialized in MRI scans. You are able to identify a wide range of pathologies. Now one day a friend walks into your office with a brain MRI of a giraffe. They ask you to determine if the giraffe is healthy. You have _some_ prior from the tens of thousands of human scans you have analyzed. Despite that, you can't make a determination with any high degree of confidence. You simply don't have a good baseline for giraffes. Any analysis you do make is assuming that learnings from human scans extend to giraffes.

Generative models are useful if we are _certain_ that training data covers essentially all possible variations. Training a model on MNIST to produce numbers/digits is trivial, capturing 99.99% of the perceptible variation in handwriting is (relatively) easy. The issue with doing this for most medical applications is twofold: 1) medical training data is difficult to acquire and much less abundant than most other data types, 2) we don't have a good measure for how much of the possible variation we have captured and how much we have missed

> "I think that medical data is a really good example of a use case that we don't want to work on," says Elbaz. "In order to model medical diseases, you need real doctors to help you. "There's a lot of specialized knowledge that you would need in order to create this medical synthetic data. Even though medical data is extremely valuable. It's something that I think requires a separate company. It's just too hard. Anything that requires very, very, specialized knowledge is hard," he concluded.

One of the clear uses of synthetic data is areas where data is inaccessible and/or expensive to gather like medical imaging. Turns out, there may be "no free lunch".

I'm really glad to hear that this company acknowledges how hard that problem is and avoids it. The risk is so high that there is just no value in it.

[0] https://www.microsoft.com/en-us/research/wp-content/uploads/...

If I were to add synthetic lens flare to an existing image, would that be synthetic? Change the white balance? Add dirt, scratches, replace paint colors, etc? Change someones skin tone? Add blemishes?

Knowing nothing, all of these seem like they would be very beneficial, but I always see people saying any synthetic data is bad data. With the probability of intercept being so incredibly low for some of this stuff, I can't see how that could be true.

Am I missing something, or isn't this a tool that can be used to generate a highly biased dataset to train an ML system on? This is how you'd insert biases into your 'algorithm' to get the desired result, isn't it? And then you could use that system to say, select from a pool of loan applicants, and any biases would just be 'due to the algorithm'?

It sounds silly, but maybe it's easier than hand-encoding the rules yourself?