Hacker News Reader

Curious about how you generated the training data? Was it just asking an existing model to generate a bunch of examples?

I ask cause would this be a kind of model distillation?

I have a small model I'm looking to train on some data, and I have some real live data but I'd love to be able to extend it.

View on HN

All synthetic data. For this usecase, it was easier because all current generation LLMs, even the small models, are really good at bash commands (and SQL queries too)), so you can reasonably start batches of cheap subagents whose output is reviewed by a more capable model and merge into main training set. After 100k, I had to standing instructions to run the generation loops selectively, meaning only update samples in a given area where we see poor capability.

Do you have a write-up or git repo for this? Would love to learn more and/or dig into the guts

edit: others have asked any you have replied "soon (tm)", looking forward for that day

It would be awesome to share your training set on hugging face if it’s easy to de-personalize it. The largest I could find was only 800 rows.

I’ve previously fine tuned (SFT, currently trying to distill) models using this https://huggingface.co/datasets/westenfelder/NL2SH-ALFA

It is a form of distillation, as long as you're working a very narrow "trivial" topics it works perfectly.

Reply on news.ycombinator.com