That's just how plain git works anyway. Commits are just a tree of hashes which refer to the blobs in the objects store. Diffing is done on demand. If you can implement diff for binary files, then you've got git.

That's exactly what git-annex does: https://git-annex.branchable.com/

LFS is kinda close, but you don't store the files "somewhere convenient": you need support on every git remote to handle the files.