Hey OP, I found some issues with your code:

During SFT, it uses the full training dataset[1]:

df = pd.read_csv('data/extraction_training_data.csv')

And during the evaluation, it uses the middle part of the same dataset[2]:

df = pd.read_csv('data/extraction_training_data.csv')

df = df[100000:100000+NUM_TEST_SAMPLES]

Also, you split train/test/val by chunk and not by document[3]. Then, the model "has seen" the documents that you're using to evaluate it (even if you're not evaluating it on the same chunks).

[1]: https://github.com/herniqeu/extract0/blob/0f8696a6fb1b620658...

[2]: https://github.com/herniqeu/extract0/blob/0f8696a6fb1b620658...

[3]: https://github.com/herniqeu/extract0/blob/0f8696a6fb1b620658...