There is work going on about external object database support that could help in the long run:

https://github.com/git-lfs/git-lfs/issues/1702

(I am working on this for GitLab.)