Maybe a mixed approach similar to what dvc [0] is doing. Fully integrated with git but tracked and stored elsewhere.

[0] https://dvc.org/