Six principles for evaluating cognitive capabilities in AI models

Worth reading the substance behind each of these:

Principle 1: Be aware of your own anthropomorphic cognitive biases.

Principle 2: Be skeptical of others' (and your own) hypotheses. Design control experiments for possible alternate strategies that could produce the observed behavior.

Principle 3: Design novel variations of stimuli or benchmark items to test robustness and generalization.

Principle 4: Be curious about mechanisms underlying performance.

Principle 5: Consider performance versus competence.

Principle 6: Analyze failure types, and embrace “negative” results.