Hacker News Reader

Comment by penciltwirler | Hacker News Reader

Hugging Face's internal storage system is more sophisticated now after they acquired XetHub. They have a clever way to chunk and content dedupe the data that makes it a lot more efficient to version control large files.

View on HN

Xet's tech is very interesting and I greatly enjoyed their blog posts and other technical writeups[0]

But in the era of frontier models and hardware designed for fast model loading (eg: LLM on a chip) I can't imagine that this is worth so much. There's *a lot* of competition to improve Git LFS.

So while I agree with you that this plays a role I also have to believe that it's to understand model usage by competition and thus stifle it.

[0] - https://huggingface.co/docs/hub/en/xet/overview

git lfs just dumps your bin on s3, and is abandonoware since ms bought github, since github historically hosts on aws s3 and improving it would immediately benefit amazon. gh, the maintainers of it, is the only forge that does not allow clean-up unreachable artifact from the server.

Surely you could now vibe code such system for under $1,000.

I highly doubt it, there's about a million things that only show up at scale that aren't lessons the bots can easily digest from Medium posts and bypass

You are assuming the operator of vibe coding tools is not experienced in the domain.

There's more to than just the website. You have network bandwidth costs, security costs, infrastructure costs.

You can't just spin up a VPS and throw a few brain jar's in; the network bandwidth must be a fortune. Hosting over 700GB+ for many of the same models has storage cost too.

That is fair point, but still you can build such system for under a grand and then pimp yourself out to VCs.

Reply on news.ycombinator.com