Hacker News Reader

Comment by kkzz99 | Hacker News Reader

Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?

View on HN

What I'm most interested in with abstraction is how easy it is to change something that doesn't fit neatly inside of the abstraction framework. It looks like the model is pretty flexible as it's just plain pytorch, I couldn't immediately tell about other aspects of the training - for example they have their own optimizer, what if I want to change something?

There are lots of "just one line of python" type frameworks that are fine if you want to do the one thing in the demo but are more complicated than just writing it yourself if you have to change something.

We do have reference optimizers implemented for use on our system and available in the `cerebras_pytorch` package, but this isn't because those are the only ones supported; instead, no vanilla pytorch optimizer is currently `torch.compile()` compatible. The main difference is that we pre-initialize the optimizer state instead of doing it lazily in the first `step()`

Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.

To be honest, HF compat comes at the cost of abysmal performance compared to an inference solution such as vLLM.

New to the space — I have been using HF for both training and inference.

Do you have recommendations for a different framework for training? Accelerate seems fantastic for scaling up once I need to

HF has been pretty good for training in my experience. Both accelerate, deepspeed and other improvements are available and the ecosystem is vibrant. One thing you might want to look into is to write your own training loop rather than using Autotrainer, but that's if you want to get you hands dirty and get finer control over how you call layers, your loss function etc.

Inference is where it falls short really, and solutions like vLLM are much much faster.

Stay far away from HuggingFace if you can. Battle-hardened if you only do the absolute simplest and boring stuff. Look at the number of open issues and skim through a few source code files and you’ll understand.

Like LangChain, they were at the right place at the right time. That doesn’t make them good.

Good to hear that I'm not the only one thinking that. I read a lot of their code for one of my pet projects and I thought that maybe it's me that doesn't get it because I don't write that much python.

In the context of this blog post (reimplementing nanoGPT as-is without shenanigans, although training is a separate issue), transformers is straightforward enough. I do agree that going beyond the documentation demos does make things more complicated, but IMO still less so than other implementations speaking from experience. AI is still sometimes necessarily complex.

It's certainly an order of magnitude easier to use something like transformers or diffusers than the original implementations provided by the original model trainers, and has a few good optimizations out of the box.

That's different from LangChain which is complex for the sake of being complex.

We are talking about very big models which training requires an enormous amount of hardware. Not sure how scalable HuggingFace transformer for training such models is.

transformers outsources those types of tasks to accelerate, also developed by Hugging Face: https://huggingface.co/docs/accelerate/index

I can't speak to the efficacy of it at large scale, though.

HF is okay for off the shelf prototypes and very quick, multi-hour hacks, and I greatly appreciate their contributions to the community, but unfortunately in my (and sadly, many other people's experiences), their code is a continuum of devilish nightmares for anything beyond that.

I think you're missing a crucial difference here:

> ... write 20k LOC of complex code ...

Vs the all important:

> ... write _and maintain_ 20k LOC of complex code ...

The answer to the latter, for me, is a hard no.

Reply on news.ycombinator.com