This buys you incremental writes, commit hash pinning and diffs for free. In addition to having a git archive outputting a zip export as well?
Main wrinkle is you would need to gitignore the sqlite database as it doesn't store very well in git (binary, changes lots per insert). But it's easy to regenerate anyway as a rebuild-able cache.
Git is supported in the spec, though the tool doesn't yet handle it (soon! git is nice as you get a log "for free" which helps agents understand more about the memories).
As for sqlite in git: yes you can gitignore it, you could use git-lfs, you could use an out of band database. It would be great if there were another format more amenable to git to store vectors in. I looked at csv closely, but I was worried about float formatting/representational bugs. There is a gap in the market for a format here.
Well regarding how CSV didn't quite work for you. Have you checked out recutils?
Its a GNU Project and at least does basic relational database operations on plain text files. You can easily install it in most linux package managers.