Comment by bbor | Hacker News Reader

Comment by bbor | Hacker News Reader

Parent Full thread

Thanks for the corrections! I’m very much not an expert on LLM usage in the real world. But I’m a bit confused:

  does go through a pretraining phase using the redpajama dataset, and then an SFT phase with a different objective

Isn’t that equivalent to what I said, since “SFT” seems to stand for “supervised fine-tuning”? That it starts with a pre trained model, and then modifies that model according to your corpus?

Perhaps the confusion here is my ambiguity with “model”; I now see that there’s really two models-one for generating a draft + clues and one for constructing the final output—and this library only concerns/modifies the former. Maybe?

I should have quoted you more specifically, my apologies. I was responding to the comment that there was some training of the "model on your chosen book".

There is no fine-tuning done specific to the corpus you own. I noted this in a sibling comment, but both the pretraining and fine-tuning objective uses a generic dataset (redpajama) which "aims to maximize the generation probability of the next token given the KV cache of the previous memory tokens" (quote from section 2.2 of the paper).

This is why I noted you could really use any long-context model that also has good retrieval performance. They're training their own draft model in lieu of using an existing model, but you could get similar/better outcomes using something like claude sonnet 3.5.

Thanks for taking the time, that makes sense. This is not the first time I've misunderstood something by having opinions about what it should be doing in my opinion, haha. I absolutely agree with your last point, too.

Reply on news.ycombinator.com