Jev Does Not Play Dice: 83% probability, 19% accuracy on a hidden fair die roll

kantahayashiai.github.io

4 points·kantahayashi··9 comments

Author here. I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%.

Code and data are here: https://github.com/KantaHayashiAI/jev-does-not-play-dice

Happy to answer questions!

Maybe it watched Rosencrantz & Guildenstern Are Dead[1] once too many.

[1]: https://www.youtube.com/watch?v=C_TfdNAXOwE

I found that in the play it's 92 heads in a row [1], and in my coin test Jev returned around 92%. Maybe Jev was actually trained on it.

[1]: https://en.wikipedia.org/wiki/Rosencrantz_and_Guildenstern_A...

why would you use ai for that though? how does that translate to other tasks?

I used a die as an example of problems whose answers can't be known at all, unlike problems with correct answers like MMLU questions. In a more practical test, a document said "30% risk" and Jev returned 5%. The point of the post is that Jev is weak at some kinds of tasks, so you should check the calibration for your own use case.

saw this yesterday (when someone else posted your post), thanks for sharing such easy to understand cases which show Jev does not produce real world likelihoods for the choices

Thanks! I'm really glad that you read my article.

it's the kind of posts I come to HN for!

you did a interesting little analysis of a hot topic, wrote it up nicely, makes an interesting facet of Jev et al very clear

– Will I see a crocodile today?

– Yes: 0.1% / No: 99.9%

– ...but I work at a crocodile farm!

Reply on news.ycombinator.com