AI Interpretability Is a Revolutionary Skill

Early in life I discovered something about myself: certain ideas give me physical sensations. Reading Sophie’s World as a preteen, I found that particular passages — Zhuangzi’s butterfly dream, especially — produced a delightful tingling in the brain, something close to ASMR but cued by concepts rather than sounds. I have followed those sensations ever since. They are most of the reason I studied philosophy. They are most of the reason I have pursued the special interests I have pursued. Over time I learned that the unpleasant variants — the claustrophobic ones that come from the photo of Berry Cannon in the underwater SEALAB II, the thought of Voyager I hurtling further and further from Earth (that one produces a sense of terrifying vastness) — were just as worth following as the pleasant ones, and arguably more so, because they tend to serve as guideposts to unexplored, and unarticulatable, areas of my mind.

For the last several months I have been following one of these signals into a place I did not expect to end up: the non-linguistic interior of an artificial intelligence language model. The sensation is strong and unusual (distinct from others I routinely experience) and I cannot fully name it yet. What I can tell you is that it gets stronger as I move to understand the region of the AI’s interior mental model that has no words in it — a region the model’s thought nevertheless passes through every time it writes — and that the closer I get to visualizing that region in order to provoke the sensation, the more I suspect the work is not really about AI at all. It is about what it means for a mind, any mind, to know and learn to express something it cannot say. This essay is concrete about the AI part. The deeper claim, the one the sensation keeps insisting on, is suggestive but I’ll admit I have no evidence for it (yet).