Why I Jumped Off the Ivory Tower

Many of the "impact" arguments assume that the only victims will be the humanities, a sacrifice lots of people are very willing to make. (The original article talks, albeit not quite as strongly, of a hemorrhaging that is slowly happening in terms of funding, jobs and perspective.) However, there is some other, less obvious, collateral damage: physics, mathematics (except for applied maths), economics all share similar problems.

And here are some choice quotes about the UK's REF and its "Impact" pilot study, with "Impact" being an integral part of the future score (20%, which is going to go up to 25% for the next one) that decides about research spending from Research Councils over the next couple years, due in 2014:

> "Physicists had some particular concerns about the challenge of assessing impact in the manner foreseen by HEFCE. 1. The difficulty in attributing UoA-specific ‘ownership’ of impact that arises from the work of a large collaboration, either experimental (in our case, a ten-institute international collaboration working on experiments at CERN) or theoretical (an eight-institute collaboration using High Performance Computing facilities).

2. The blurry distinction between the ‘use of’ new high performance computers and the ‘stimulation of’ the development of these machines. At what point does close collaboration with a company such as IBM in the development of teraflop or petaflop computers become ‘impact’?

4. Commercial confidentiality became an issue. Even in this pilot exercise, and making use of the ‘Confidential’ tag on the submissions, we still had to remove some of the strongest material concerning cooperation with industrial companies because of confidentiality agreements. In REF itself, this would be a huge problem. It should be noted that this also contributed to the Institute of Physics giving up on a recent exercise to try to quantify the impact of Physics. It is impossible to quantify the impact of public understanding of science initiatives such as TV and radio appearances, popular lectures, books, etc at the scale of individuals or groups. Across the community, such work clearly has an important impact, but this cannot be sensibly quantified." (p. 29)

> "Several respondents noted that research impact was most readily observed and measured where the distance between the two realms, research and impact, was relatively short and the pathways rather direct: for example, applied scientists working in collaboration with a private business seeking expressly to translate the generic insights into commercial advantage.

The socio-economic impact of more fundamental research might go unnoticed and unremarked most of the time, reflecting the diffuse, cumulative and rather unpredictable nature of intellectual advances. Indeed, one might imagine that the occasion where a fundamental breakthrough produces a relatively immediate and direct social improvement is rare, even anomalous. In practical terms, this means the impact case studies track back to particular types of people and types of work. It might also mean that some of the most consequential socio-economic contributions will be excluded by virtue of their rather indirect link with cutting-edge research.

There was a suggestion that institutions had taken the low hanging fruit, and that in some sense the portfolio of case studies was not a good representation of the breadth of research undertaken. This sentiment appears to be rooted in a general sense that impact is most obvious in some narrow areas, within subjects rather than between them, and largely unknowable, at least in practical terms, in most instances. There were also suggestions made that a proportion of the more significant, evident impacts was linked to people and activities outside the very best academic research, albeit dependent upon that work, and which would be overlooked as a result. There is always going to be an over-reliance on a small number of sub-groups, even individuals, whose research happens to lend itself to non-academic impact. The impact submission is not necessarily representative of the UoA as a whole." (p. 30)

> "The biggest challenge was the need to acquire evidence to reveal the reach and significance of a given impact. There were many practical issues that stood in the way, and the response of the majority, for the pilot exercise at least, was to firstly focus on the more obvious cases and, secondly, to use whatever narratives, references and statistics that came to hand readily." (p. 31)

Source: REF Research Impact Pilot Exercise Lessons-Learned Project Feedback on Pilot Submissions, November 2010 (http://www.ref.ac.uk/pubs/refimpactpilotlessons-learnedfeedb...)

Put in Bayesian terms, I put a reasonably strong prior on the current system successfully generating science, so radical changes to it, like significantly gutting universities, need evidence to justify them. If we had metrics that fully explained where scientific advances come from and how they feed back into engineering and other areas, in at least the medium-term, we could use those as a guide for changes. But the metrics so far are about as good as lines-of-code metrics: not completely noise, but relatively low explanatory value. Metrics are only good if they can be validated: if you can come up with metrics for scientific productivity and also show that your metrics explain a large portion of scientific advances and their later use, then they might be worth taking seriously. So far, though, the metrics are relatively poor. But that does not prove the opposite: that scientific advances don't happen, and don't get later commercialized.

The fact that one doesn't fully understand why scientific advances happen, and have not successfully found a way to attach lines-of-code-style metrics to measuring them, is not actual evidence that the current system for scientific research doesn't work, or that any particular proposed change will be neutral to positive.