Hacker News Reader

Comment by axegon_ | Hacker News Reader

That was very much evident even from back ehwn the first GPT's came out. The moment you started introducing synthetic data, the quality plummeted.

But there is another use case where LLM's can truly help with synthetic data: the more classical classification and regression problems - specifically gathering training data. I had this exact case at work two days ago: A large dataset with a small subset of labeled data. For a binary classifier, there was a huge imbalance in the data - the ratio was roughly 75-25%. I did not have the desire to do all this manually so I used an LLM to get a list that would even out the numbers(and get a 50-50 ratio). And using the data I had, plus the additional synthetic data, the accuracy of my small classifier ended up picture-perfect(given that my actual target was "85-90%" accuracy and the actual result was just shy of 99%).

View on HN

I'd argue that the case you give isn't an example of using a computer to generate data, it's a case of a human adding data (the data being the fact that the binary classifier should have a 50/50 balance).

This sort of massaging of data has its drawbacks as well--obviously this only works if the balance of that binary classifier actually is 50/50 in reality: I don't know enough about your case to say you were wrong, but I can imagine a lot of scenarios where a binary classifier should not be represented 50/50 in the data.

This is a question of definition. It is synthetic in that I just passed a prompt, asking for N amount of examples of X. And I did not go over the entire list I got and blindly trusted it. In this context, I needed an even(or nearly even distribution) of samples in the training data and it worked way better than I was hoping. Mind you, I have to face a similar issue next week and I'm not sure this approach would cut it - I need way more training data and way more classes to work with. 249 classes if I'm not mistaking.

I question what "it worked way better than I was hoping" means in this context. If you're saying that filtering the input data to create a uniform distribution created a uniform distribution in the output, I'm not sure why you'd hope for any less--that's exactly what I'd expect to happen. But that's a poor measure of success, because you don't know what side effects that had: the removed data ostensibly contained other variables besides your binary variable, and you don't know if those variables were sampled in any useful way, so I'd be hesitant to say this worked well without at least an attempt to measure those other variables.

Can you clarify what you mean by using an LLM to “get a list that would even out the numbers”? If you’re doing binary classification, you need datapoints for features as well as the target class so how does an LLM synthetically create that without causing problems?

Just curious, but did you compute that 99% using purely real test data, or does your test set also include artificial data?

Apart from using the results from the training/testing/validation sets? Several people manually went over several thousand random samples.

Reply on news.ycombinator.com