Git lfs is what you want.
But beyond that, what’s keeping the server from storing these deltas instead of full versions? Only argument I can think of is that server side corruption of a single file would break the chain of deltas and you’d lose every child.
But surely that can be solved using some sort of redundancy.
If you're using git-shell or git-daemon to host your git repositories on a server, then you can't use that alone to also manage transfer/access of stuff that is managed through git lfs.