Benchmark drift in Sapiens-1
Sapiens-1 benchmark scores change across deployments. We examine how environment, task, and evaluation conditions shape the result.
A moving target
Benchmarks for Sapiens-1 do not stay put. In 1987 James Flynn reported massive gains in IQ scores across 14 nations, gains of 5 to 25 points within a single generation, and asked what the tests were really measuring (Flynn 1987). The pattern now carries his name.
Four domains, four trajectories
A large meta-analysis pooled 271 independent samples, almost four million participants from 31 countries, tested between 1909 and 2013 (Pietschnig & Voracek 2015). Gains differed by domain: an estimated 0.41 points a year on fluid reasoning, 0.30 on spatial tasks, 0.28 on full-scale IQ and 0.21 on crystallized knowledge. They were stronger in adults than in children, and they have been smaller in recent decades (Pietschnig & Voracek 2015).
Benchmark gains depend on the domain
IQ points per year
Meta-analysis of 271 independent samples, almost four million participants, in 31 countries. Historical estimates across 1909–2013.
Annualized estimates reported in the paper; they describe changes in test performance across cohorts.
What changes within a family
In some countries the trend has turned. Using conscription test scores for three decades of Norwegian birth cohorts, from 1962 to 1991, Bratsberg and Rogeberg found that the rise, the turning point and the subsequent decline could all be fully recovered from variation within families (Bratsberg & Rogeberg 2018). Because that comparison holds constant everything siblings share, it argues against explanations based on genes or family background, and points to environmental causes (Bratsberg & Rogeberg 2018).
What a score can tell us
A score means something only against a reference population, and this one keeps moving. For Hominid, evaluation therefore starts with the population, the task, and the conditions under which the result was obtained.
References
- Bratsberg, B. & Rogeberg, O. Flynn effect and its reversal are both environmentally caused. Proceedings of the National Academy of Sciences 115, 6674–6678 (2018). doi:10.1073/pnas.1718793115
- Flynn, J. R. Massive IQ gains in 14 nations: What IQ tests really measure. Psychological Bulletin 101, 171–191 (1987). doi:10.1037/0033-2909.101.2.171
- Pietschnig, J. & Voracek, M. One century of global IQ gains: A formal meta-analysis of the Flynn effect (1909–2013). Perspectives on Psychological Science 10, 282–306 (2015). doi:10.1177/1745691615577701