github.com
You can do things like:
cat names.txt | llm sort -q "Which one of these names is best for a pet seagull?"
cat books.txt | llm sort -q "Which book is more related to basic vs. advanced CS topics?"
I see a lot of potential marrying LLMs with classic UNIX interfaces.
https://github.com/BishopFox/raink
https://bishopfox.com/blog/raink-llms-document-ranking
raink \
-f testdata/sentences.txt \
-r 10 \
-s 10 \
-p 'Rank each of these items according to their relevancy to the concept of "time".' |
jq -r '.[:10] | map(.value)[]' |
nl
1 The train arrived exactly on time.
2 The old clock chimed twelve times.
3 The clock ticked steadily on the wall.
4 The bell rang, signaling the end of class.
5 The rooster crowed at the break of dawn.
6 She climbed to the top of the hill to watch the sunset.
7 He watched as the leaves fell one by one.
8 The stars twinkled brightly in the clear night sky.
9 He spotted a shooting star while stargazing.
10 She opened the curtains to let in the morning light.To optimize performance and reduce LLM distraction, instead of aggregating answers two by two, it does batched aggregation. The key innovation is in the batching order - I implemented a [semantic_batching function](https://github.com/thiswillbeyourgithub/wdoc/blob/18bc52128f...) that uses hierarchical clustering on the embeddings and orders texts by leaf order.
The implementation was straightforward, runs very fast and produces great results. The function is designed to be usable as a standalone tool for others to experiment with.
The 'allpair' variant, which aggregates the score across all n*(n-1) comparisons will work, but the other two couldn't possibly have stable results - it would depend on the order of the items in the input.
Otherwise it might be more or less the same as using https://github.com/TheR1D/shell_gpt ?
Log probabilities of output tokens indicate the likelihood of each token occurring given the context.
It's astoundingly less efficient right? How many compares ( and LLM calls ) to rank 10 items in order? And is it actually stable? You could get a ranking with logprobs in one llm call for 10 items, or do it n=3 times, with a shuffled order and average them out. I'm not sure how to scale to larger sizes of items though.
I guess it depends on how many items you are sorting, but when I think about sorting I think about putting 100+ items in order.
cat spelling.txt | llm sort -q "how many times does the letter 'r' appear in the word 'strawberry'?"
cat geology.txt | llm sort -q "which kind of rock is the best to eat, according to Berkeley geologists?"
cat law-library.txt | llm sort -q "what legal precedents are best for my lawsuit vs Colombian airline Avianca?"
cat adhesives.txt | llm sort -q "what glue is best for holding the cheese on my pizza?"I know that’s always the complaint with LLMs. But at least with the paid ChatGPT (the product, not the API), it has access to a Python runtime and web search built in so it could at least attempt to get the some of the answers right.