But training to update a single fact isn't slow and costly. finetuning for just a few facts could be done in a matter of seconds with just a few text examples of the correct answer.
Total time is mere seconds.
If you save the adam optimizer parameters from previous runs, you'll do even better at preventing forgetting.