We're incapable of putting an accurate, standardized value on developer productivity, yet there often seems to be consensus between senior engineers on who are the high performers and the low performers. I certainly can tell this about the people I work with.

We are definitely not. Point at a problem, and measure the cost of solving it. That's developer productivity.

We only avoid doing it at scale because it's expensive. In particular if we want the measurement to generalise out of sample.

(In particular in this case, where once we're done, proponents will claim our data is too old to be a useful guide to tomorrow.)

Yes.

If we could even measure teams, against themselves, others and some kind of baseline, but we don't AFAIK.

Lines of code pushed ... obviously /s

Unironically, ai evaluating the impact of those lines might be getting close to a metric that would measure output better than having everyone print out their last 6 months of work for the new boss to look at.