niklio

The robots love coffee so this is a major red flag

Sounds like a good next project!

beep boop, congrats on the high score

The calibration is set up to get some variation between humans. Dario's brainrot is even worse than yours: https://howclankerareyou.com/r/DarioAmodei

Thanks! That's a great idea - i'll top up my fable budget and get started :)

The metric used is per-word surprisal: -logprob of each word you type. This is just the same thing as per-word cross entropy or KL-divergence where the user distribution is one-hot. Calibrating it so text generated by frontier models scored poorly was a challenge at first. Originally ChatGPT was scoring around 54%. I'm still having trouble assigning high scores to the personalized Gemini and ChatGPT responses when I'm logged in because all my personal context gives surprising responses.

And yes, gibberish responses score very human :)

Ribbon | Machine Learning Engineer | NYC | https://ribbonhome.com

Ribbon is looking for machine learning engineers who want to be part of a mission driven team at a fast growing, series A startup.

We're helping consumers use a mortgage to buy the home of their dreams. Institutional investors are making the real estate market inaccessible to the average consumer. Our mission is to level the playing field. We're doing research and building production systems to predict home valuations, underwrite consumers for mortgages & simplify the home buying process.

Technologies: Python, Scala, Spark, AWS, Terraform, Tensorflow, Docker

Submit your resume https://jobs.lever.co/ribbon/ or send me an email nik@ribbonhome.com