What We Can't Measure About AI Yet

A new tool atrophies some skills and creates new skills that did not exist before. It is easier to identify which skills atrophy than the new skills being created as even the language to describe what is new has not yet come into focus.

" What changed was that the cost of preliminary exploration collapsed. I could sketch an argument, identify the first serious objections, test whether they were fatal, and reach a provisional verdict in an afternoon rather than a fortnight. [...] Dropping a question after an afternoon’s work feels nothing like dropping one after three weeks. When the exploration costs are low, the sunk cost attachment disappears, and you find yourself dropping bad questions earlier and more often, which means the questions you keep are better. I explored far more ideas, and my working portfolio became both larger and better curated. [...] The skill that improved most, and the one I would never have thought to look for, was something I can only describe as question-identification – the ability to find problems that are both tractable and important. This is the thing an academic career is substantially built on and which nobody, so far as I know, has ever tried to teach directly. [...] nobody told me that working with LLMs would reorganise my research practice around finding good questions rather than producing answers to the ones I already had. There is a further wrinkle here that I find harder to set aside, which is that the language I have for describing what these tools have done to my thinking is itself partly a product of working with them. "