Hacker News Reader

Author here. I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%.

Code and data are here: https://github.com/KantaHayashiAI/jev-does-not-play-dice

Happy to answer questions!

View on HN

Maybe it watched Rosencrantz & Guildenstern Are Dead[1] once too many.

[1]: https://www.youtube.com/watch?v=C_TfdNAXOwE

I found that in the play it's 92 heads in a row [1], and in my coin test Jev returned around 92%. Maybe Jev was actually trained on it.

[1]: https://en.wikipedia.org/wiki/Rosencrantz_and_Guildenstern_A...

why would you use ai for that though? how does that translate to other tasks?

I used a die as an example of problems whose answers can't be known at all, unlike problems with correct answers like MMLU questions. In a more practical test, a document said "30% risk" and Jev returned 5%. The point of the post is that Jev is weak at some kinds of tasks, so you should check the calibration for your own use case.

Reply on news.ycombinator.com