Hacker News Reader

Comment by TarqDirtyToMe | Hacker News Reader

LLMLingua uses a well-trained small language model after alignment, such as GPT2-small or LLaMA-7B, to detect the unimportant tokens in the prompt and enable inference with the compressed prompt in black-box LLMs, achieving up to 20x compression with minimal performance loss.

View on HN

“Why waste time say lot word when few word do trick” -Kevin Malone

Perfection. Key insight. "Few Word [is] All Need" (with a robust enough foundation model)

Linked for the culture: https://www.youtube.com/watch?v=bctjSvn-OC8&t=4s

Sleep big last night

"Kevin, are you saying 'See the World' or Sea World?" -- Jim

What would happen if instead of the long prompt, you just sent the mean of the embeddings of the prompt tokens?

Came here to mention this. Whenever I hear "alignment" I immediately say "No way am I going to use that shit". Seriously, there's alignment and then there's censorship—the AI creators are using the former when they actually mean the latter. This needs to stop.

My understanding is that in an academic context you’ll hear alignment anytime a model is tuned to accomplish a certain task, not just to steer its political affiliation and idea of ethics

I don’t think this models use of alignment implies any sort of censorship, it’s just being tuned to accomplish the task of outputting only important tokens for the target llm

In my experience it means the AI will waste tokens apologizing for it's short comings and ignoring task prompts in favour of it's alignment.

This does not seem relevant to the alignment discussed in the paper. It seems to be explicitly out of scope:

> The potential harmful, false or biased responses using the compressed prompts would likely be unchanged. Thus using LLMLingua has no inherent benefits or risks when it comes to those types of responsible AI issues.

It amazes me that this amazing new technology comes out and there is a group of people who are like "NO, NOT IF IT CAN'T TELL RACIST JOKES!"

I agree that like "tone" alignment is silly and pointless for models in the public domain, but if I were a big company who wanted to keep customers I'd align my models this way. It isn't censorship, its marketing.

Reply on news.ycombinator.com