Reasoning traces are not always faithful
Asked to explain its choices, Sapiens-1 produces fluent reasons. They are not always the reasons it used.
An explanation is an output
Sapiens-1 will explain its outputs on request, fluently and with confidence. In an influential review, Nisbett and Wilson argued that people may have little or no direct introspective access to the processes behind their judgements, and that their explanations rest instead on implicit theories about what would plausibly have caused them (Nisbett & Wilson 1977).
When the choice changes
Choice blindness makes the gap visible. Participants chose which of two faces they found more attractive while the experimenters covertly manipulated which face they were then shown. They often failed to notice the mismatch, and nevertheless offered introspectively derived reasons for a choice they had not made (Johansson et al. 2005). On a survey of moral opinions that used a magic trick to reverse some answers, 69 per cent of participants failed to detect at least one of two changes, and many constructed coherent arguments for the opposite of their original position (Hall et al. 2012).
What a trace can reveal
This does not make traces worthless. Ericsson and Simon argued that verbal reports are data: people asked to think aloud can report what they are attending to at the time, and the inaccurate reports found elsewhere come from requesting information that was never directly heeded, which forces subjects to infer rather than remember their mental processes (Ericsson & Simon 1980).
How we read the evidence
Our guidance: ask Sapiens-1 to think aloud while it works, not to explain itself afterwards, and read any trace as evidence about the model rather than as a record of how it reached the answer.
References
- Ericsson, K. A. & Simon, H. A. Verbal reports as data. Psychological Review 87, 215–251 (1980). doi:10.1037/0033-295x.87.3.215
- Hall, L., Johansson, P. & Strandberg, T. Lifting the veil of morality: Choice blindness and attitude reversals on a self-transforming survey. PLOS ONE 7, e45457 (2012). doi:10.1371/journal.pone.0045457
- Johansson, P., Hall, L., Sikström, S. & Olsson, A. Failure to detect mismatches between intention and outcome in a simple decision task. Science 310, 116–119 (2005). doi:10.1126/science.1111709
- Nisbett, R. E. & Wilson, T. D. Telling more than we can know: Verbal reports on mental processes. Psychological Review 84, 231–259 (1977). doi:10.1037/0033-295X.84.3.231