This is also my biggest issue with Git. And integrated/streamlined support for large files.
LFS uses some sort of internal filtering and tracking to determine which binary files might have changed. It seems to have trouble deciding if there are actually dirty files that need changed. So you can't just say, "Okay, go find all the binary files that didn't actually get moved to LFS and correct them"
Instead you end up with random moments where you want to commit a single file and git instead detects 1000 png files that it absolutely could not go on without doing something about.
But then the diff is a disaster so good luck understanding that what it is actually mad about is that it wants to move the files into LFS. The only way I finally figured it out was to manually load the object blob and notice one of them was an LFS pointer file.
I personally think git annex handles things more elegantly, but lfs won that battle.
It is a pile of garbage, but it's better than nothing.
Also, deep learning training data often consists of large image files, and can also be considered "source code", and in any case it can be very useful to put these under version control.
And finally it can be useful to put external dependencies as tar-files into your source tree.
For writing tests in a deep learning code base, rather than simply including a native data file (image, CSV, whatever), I've taken to writing a fake data creator class. It always feels like overkill when an alternative solution is including a native data file or two that already exists.