Comment by jarulraj | Hacker News Reader

Comment by jarulraj | Hacker News Reader

Parent Full thread

Interesting observation :)

I guess this is the SQL query you have in mind that uses the LIKE operator:

  SELECT ChatGPT("Respond to the review with a solution to address the reviewer's concern", review)  
    FROM postgres_data.review_table  
    WHERE ChatGPT("Is the review positive or negative?", review) LIKE "%positive%"
        AND location = “waffle house”;

From a query processing standpoint, both queries should have equivalent performance -- unless we build an index over the output of the ChatGPT query in EvaDB, in which case the former query would be faster than this one.

mm, no, not unless you're doing some LIKE-specific optimization (and even then, I think you'd want "positive%").

So like, at the end of all the decoders, the model gives you an output vector; you multiply this by your embeddings to get your token probabilities, then you sample from them to choose a token.

Instead of sampling, you could just look at the probabilities for the tokens "positive" and "negative" and return whichever of those two is highest.

Yes, it should be "positive%". That's an interesting LLM-level optimization.

Doesn't this token sampling optimization require using a locally-running model like Llama?

I am presuming that OpenAI doesn't provide direct access to token probabilities in its API.

Reply on news.ycombinator.com