Hacker News Reader

Comment by crazysim | Hacker News Reader

How reliable is this? I'm half joking too but I wouldn't mind reading a report comparing this on OpenAI and various other LLMs.

Somehow I doubt it'll be 100%... right?

View on HN

Might be more reliable to ask it to generate a JS function to compute is-even, then just js eval the returned code with the args.

What could go wrong

normally llms are pretty bad at math, but in this case it should just look at the final digit and map 10 values which I can't imagine going wrong

I tried with this on chatgpt.com (anonymous) and it was wrong:

>You are an AI assistant designed to answer questions about numbers. You will only answer with only the word true or false.

>Is 393330370227914821469106615363204944758938252979261537157082994586230072180858944545028761701928694832864623009988147774229437650643225379825905427239525512110359581021414640894111281701792224552922491447051506246553646282117414112976459608594044929244664050172002138933343230226871897567 an even number?

response:

>True

The prompt was the same used as the library: https://github.com/Calvin-LL/is-even-ai/blob/b00dbfcbb89a197...

To be fair, Javascript double point precision get tripped up by such large numbers as well.

The tokenizer might lump the last digit together with some preceding digits though. I know o200k_base (OpenAI -o models) tends to give groups of three (900001 for example is 900-001).

Anyway, I wouldn't be surprised if a non-finetuned model made some mistakes.

I'm just curious how _low_ can you go before it does go wrong.

I mean if this were a check for primes the answer would be 1.

is 0x2a even?

its as accurate as how Tiktokenizer deal with numeral characters

Reply on news.ycombinator.com