Just as you say, you do not need Llama's training data in order to build Llama from source, nor even to fine tune it. You don't need a copy of the training data.
(edit to spell Llama correctly)
(edit to spell Llama correctly)
The currently LLM status is more akin to opensource plugins to some propriety system, like game modding.