I have done my share of fine tuning on open source LLMs (e.g. Llama). I'm surprised you have very poor generalization.

I assume you're using standard techniques, like lora/qlora, which might leave room for issues with your data. Can you share more details on what is the format of your data points? like, Q/A, free text,...