Synthetic data generation for tabular data | Hacker News Reader

Synthetic data generation for tabular data

github.com

58 pointsskadamat12 comments

If people are looking for a quickstart:

Colab notebook for generating single-table data: https://colab.research.google.com/drive/1F3WWduNjcX4oKck6Xkj...

Colab notebook for generating multi-table data: https://colab.research.google.com/drive/1L6i-JhJK9ROG-KFcyzT...

do you have some thoughts on how sdv-dev type projects can be used to start populating, say, a database (eg: mysql running in a container) i've looked into this space a bunch (eg: Gretel, Tonic, etc) and there doesn't seem to be a good solution that works end-to-end Privacy Dynamics is quite cool but ideally I'd like something super lightweight that can get pointed to a source db of some sort and then write to a sink (maybe applying a transformation layer in the middle)

Curious what a good end-to-end solution looks like for you? Is it more about ease-of-use (import/export with minimal effort) or is there a privacy layer that's missing?

I see it in 4 steps: 1. Connect to a source db to import your data 2. Train a Gen AI using the source data 3. Use it create synthetic data 3. Export synthetic data into a new db

The SDV team is working on business solutions to cover the full use case. You can use the public SDV to validate steps 2 and 3.

its not necessarily about the privacy layer per se. the workflow i was ideating over is as follows:

1. spin up a production-equivalent database (eg: mysql container instead of prod RDS)

2. point a process/binary (maybe a simple container) to:

-- source db (RDS)

-- sink db (mysql container)

-- transformation function (that may use gen AI, etc) to seed sink db with synthetic/anonymized data [there may be some parallel process to enable testing of this transformation function]

3. profit (use this for dev etc)

Key over here would be speed in step (2) if the entire pipeline were to run end-to-end on-demand. do you have some examples of using SDV to achieve this? highly possible that there's already something in the docs that I have missed

This is what I am trying to solve via building Data Catering (https://data.catering/). It gives you the ability to generate data into any database (along with maintaining any relationships between data) via metadata that can be retrieved via a source database or other types of metadata sources (i.e. Open metadata).

I remember a lib that was like this but would use GANs etc to gen data. I tried it with little success, reverting to SMOTE. Wonder how this would do? My impression is that tabular data is difficult to use synthetically.

SDV does offer a CTGANSynthesizer, which is a GAN-based generative approach. Could be worth a try, though CTGAN specifically may require customization (tweaking some parameters).

That being said, synthetic data definitely isn't a magic pill for all use cases. I have found it particularly useful for things like QA, performance testing, etc. -- where alternative tools for test data creation aren't sufficient.

For the use case of imbalanced classification: May be worth asking what is it about existing solutions (SMOTE) that doesn't work well?

What do people do with synthetically generated data? Use it to train models? I doubt if models will improve by appending redundant data since they use the training data to estimate distributions. Just curious.

The thing is you use synthetic data to when it is difficult to obtain real data. For example, in medical imaging, it is very expensive to collect MRI scans to build a large dataset. Not to mention the potential privacy issues and obtaining informed consent to publish the dataset. Synthetic datasets can help here to, for example, pretrain your model and fine tune on real data afterwards. I'm then assuming that collecting tabular data can face similar issues which prevent building large datasets.

Can someone help me understand the licensing of this?

https://github.com/sdv-dev/SDV/blob/main/LICENSE

It was MIT licensed up until 2022 where it was changed to what it is now, where they say that it will become MIT again 4 years after release... but is that from when the license was changed or the first release of the software in GitHub?

IANL but I've taken it to mean that releases acquired under the original license would continue to be governed by those terms.

I'm liking this new approach better than e.g. perpetual AGPL though, as it provides incentives for businesses to acquire commercial rights while avoiding any dead end agreements that outlive the startup entity.

Hey there, we wrote an in-depth post on the BSL license and why we switched to it: https://datacebo.com/blog/sdv-bsl-license/

BSL is an _eventual_ open source license. 4 years after every version release, that code now transitions to being MIT licensed. SDV's roots are in academic science (MIT), so we wanted to make sure researchers could still use the toolkit for their work.

Reply on news.ycombinator.com