> Finetuning seeks to extend the length of the context window itself.

Does it? I'd thought fine-tuning was more like transfer learning, adding highly specific data to the training set (e.g. your internal codebase) rather than actually modifying the architecture.