Here's the part that really seems to matter the most.
This does create some danger of circular reasoning
(perhaps we're just carefully describing our own biases).
But we have to start somewhere, and basing our
evaluations on how people write actual code seems like a
good place.The really exciting point comes when we can
re-run all this analysis, basing it on actual job
performance, rather than interview results.
Absolutely. Results on the earlier screens and results on the later interview aren't exactly independent variables, and neither is the one that really seems to matter - subsequent on-the-job success. There are all sorts of biases and confounding factors likely to be shared between them, especially since there's no indication that the later interviews were even done blind w.r.t. the earlier screens. Until then, we're just measuring correlations between different interview techniques, and it should be no surprise that two different kinds of code-focused interviews show the highest correlation.