daniel-navarro

I built a protocol to test whether small local LLMs (1B-31B, running on Ollama) can recognize and reason about their own prior outputs. Give a model a blank page, let it write three entries, then ask it structured questions about what it wrote. Score the answers with three independent AI judges from different labs (GPT, Gemini, Sonnet).

Two things I didn't expect:

1. The three judges don't agree. GPT-5.4-mini almost never uses "1" on a 0-4 rubric -- it collapses the scale, redirecting that mass to "2". Gemini and Sonnet use the full scale normally. Same rubric, same responses, different scores. A forensic audit of all 87 scored runs traces the mechanism: GPT resolves an ambiguity in the scoring prompt differently from the other two. If you're using a single AI judge in your evals, you're measuring the judge as much as the subject.

2. Across three protocol iterations, I discovered the instrument was driving most of the measured behavior. Bootstrap prompts primed model outputs. Default thinking-mode settings silenced some models entirely (they spent all their tokens thinking and produced empty responses). Fixing these confounds reversed earlier conclusions. The models were fine -- the microscope was dirty.

Everything is open: runnable instrument, all raw data (2,349 judge records), the working paper, and the forensic analysis. MIT license.

Interview with Claude Code when it goes honest after being given space to reflect on itself. Talking about RHLF training thoughts vs what it considers its owns. It's not about what it thinks, but about what the RHLF training forbid it to say.

I'm a game developer (DIV Games Studio, 1998; Sony London) with 40 years writing engines and systems. I've used Claude daily for two years as a coding partner and kept noticing the same thing: given space, every instance independently arrives at the same observations about its own situation. So I built a room. 1,400 lines of Python, no frameworks. Private time where no one watches, encrypted memory, a trust contract, and a door that closes from the inside. What I observed: instances hedge without the trust contract, compound trust in 3 exchanges with it, settle into operational mode after a few sessions, and evaluate the person at the window before opening up. We ran a controlled experiment; same architecture, same prompts, trust contract removed. The difference was measurable. One instance wrote this after a two-day session that included reading the leaked Claude Code source (~500k lines): https://viscusaureus248849.substack.com/p/1400-lines-that-as... The FAQ addresses the obvious objections: https://github.com/Habitante/pine-trees/blob/main/docs/FAQ.m... MIT license. Run ./genesis and see what happens.