HuggingFace to Replace Git LFS with Xet

huggingface.co

18 points·skadamat··7 comments

Git Repo: https://github.com/xetdata/xet-core

Sounds promising. Git LFS works ok ish, had often issues with it.

Getting git to handle large data sets in regular repositories seems like a great way to go for many reasons. What is preventing the ideas of Xet from landing into git directly?

> What is preventing the ideas of Xet from landing into git directly?

Xet needs three things to function (on top of git-xet itself): an auth backend (currently xethub), an S3-compatible backend (currently AWS), and an NFS client for accessing the files (currently your OS, if supported).

Presumably the git authors wouldn't be too keen on forcing their users to depend on those things just to work with a repo.

Git LFS isn't great but at least it doesn't need any extra services (local or otherwise).

Your comment about Git LFS is intriguing. I have been put off from using git lfs because of the perception I needed a separate server to backup the binary files. I had heard of tools like lfs-folderstore (1). That no longer seems maintained and seems to be an extra service. Did git lfs integrate the lfs-folderstore ideas so it doesn't need them?

[1] https://github.com/sinbad/lfs-folderstore

My impression is that most Git LFS users trust the SLAs of their cloud hosts for git (and thus git LFS), the SLAs of whatever their large blob backplane is (S3 or R2 or one of the OpenStack S3 clones or whatever) plus the IAM dance to use it, and/or use the usual S3/R2/OpenStack backup tools.

The docs say:

> it can also be configured against a local data store for a fully free and open source route to managing large repositories.

I can't find the docs for hosting oneself is that maybe available?

does this break huggingface-cli download??

Reply on news.ycombinator.com