Hacker News Reader

Comment by sixQuarks | Hacker News Reader

Has anyone looked through all the responses and chosen any winners?

I took at the look at the SVG smilies, only GPT gets it right (https://benchmarks.llmonitor.com/svg). You can preview the SVGs drawn by the LLMs by pasting this into the browser console

document.querySelectorAll("td pre").forEach((node) => { let code = node.textContent; node.insertAdjacentHTML('afterend', code) })

Or take a look at my screenshot: https://i.ibb.co/Kw0kp58/Screenshot-2023-09-09-at-17-15-20-h...

Clever use of the dev tools console. I think you’re being a little uncharitable, the prompt said “smiley” and not “smiling emoji” and “smiley” was once upon a time used for the whole slew of emoji. With that in mind, several others also pass.

Good point, but you may be too charitable. At least according to Wikipedia, a smiley is supposed to be smiling. https://en.wikipedia.org/wiki/Smiley

I don’t know about Wikipedia but I’m speaking from lived historical experience.

Here’s a throwback for you: https://www.phpbb.com/customise/db/styles/smilies-13

(All “emoji” images were just called smilies in a lot of software.)

Oh neat, thanks for sharing, wanted to add an interpreter to that test

I have not looked at all responses, but I have found interesting that at the question about the sisters of Sally, initially all LLMs have provided wrong answers.

Nevertheless, when the initial prompt has been augmented with "Let's think step by step.", that helped 2 LLMs to give the right answer: GPT 4 and ReMM SLERP L2 13B.

So at least for this kind of problem, there were 2 clear winners. The fact that one of them is GPT 4, is not a surprise.

GPT4 seems to me to be the best. Undi95/ReMM-SLERP-L2-13B the runner up.

Clearly the prompt construction and initial instructions are critically important here. Without that, the ReMM-SLERP-L2-13B model produces awful results. Blank answers about half the time. For lay users, ChatGPT remains the undisputed winner. Although I do see reasonably good results with the more recent llama-2 70B variations, which are plausibly useful a majority of the time.

What a name!!

Reply on news.ycombinator.com