Hacker News Reader

Comment by minimaxir | Hacker News Reader

Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.

View on HN

Stay far away from HuggingFace if you can. Battle-hardened if you only do the absolute simplest and boring stuff. Look at the number of open issues and skim through a few source code files and you’ll understand.

Like LangChain, they were at the right place at the right time. That doesn’t make them good.

Good to hear that I'm not the only one thinking that. I read a lot of their code for one of my pet projects and I thought that maybe it's me that doesn't get it because I don't write that much python.

In the context of this blog post (reimplementing nanoGPT as-is without shenanigans, although training is a separate issue), transformers is straightforward enough. I do agree that going beyond the documentation demos does make things more complicated, but IMO still less so than other implementations speaking from experience. AI is still sometimes necessarily complex.

It's certainly an order of magnitude easier to use something like transformers or diffusers than the original implementations provided by the original model trainers, and has a few good optimizations out of the box.

That's different from LangChain which is complex for the sake of being complex.

We are talking about very big models which training requires an enormous amount of hardware. Not sure how scalable HuggingFace transformer for training such models is.

transformers outsources those types of tasks to accelerate, also developed by Hugging Face: https://huggingface.co/docs/accelerate/index

I can't speak to the efficacy of it at large scale, though.

HF is okay for off the shelf prototypes and very quick, multi-hour hacks, and I greatly appreciate their contributions to the community, but unfortunately in my (and sadly, many other people's experiences), their code is a continuum of devilish nightmares for anything beyond that.

To be honest, HF compat comes at the cost of abysmal performance compared to an inference solution such as vLLM.

New to the space — I have been using HF for both training and inference.

Do you have recommendations for a different framework for training? Accelerate seems fantastic for scaling up once I need to

HF has been pretty good for training in my experience. Both accelerate, deepspeed and other improvements are available and the ecosystem is vibrant. One thing you might want to look into is to write your own training loop rather than using Autotrainer, but that's if you want to get you hands dirty and get finer control over how you call layers, your loss function etc.

Inference is where it falls short really, and solutions like vLLM are much much faster.

Reply on news.ycombinator.com