WYSIATI and the Assessment of School Systems

For me, it seems like Amanda Ripley is everyplace right now. This morning, when I went out for breakfast, I carried a copy of The Atlantic with me into Panera and consumed her cover story "The Case Against High-School Sports" along with a pastry and a latte. In the afternoon, I listened to a radio interview she recorded with Bloomberg Radio. And tonight I finished chapter 3 of her book The Smartest Kids in the World. I'm finding this increased exposure of comparative education, a passion of mine, stimulating, and in spite of some critical remarks I have been making here and there (most notably in a review that was published in August by Whitney Tilson, in one of his periodical newsletters and on his blog, of her Wall Street Journal article "The $4 Million Teacher"), I am enjoying all this Ripley, believe it or not.

Nonetheless, her dismissive evaluation of Switzerland's education system, which I think the best in the world, in the Bloomberg interview and her valorizing of the Korean and Polish school systems on account of their improvement shown in the results of PISA (the Programme for International Student Assessment, the best known international tests that allow for comparison between different countries' educational systems) is indicative of an especially problematic instance of WYSIATI, a cognitive malady diagnosed by Daniel Kahneman and especially prevalent in educational evaluation, to the great harm of us all.   

A couple of years ago I read Thinking Fast and Slow by Daniel Kahneman, emeritus professor of psychology at Princeton and, most unusual for a psychologist, winner of the Nobel Prize in Economics. Among the fascinating chapters in that book is chapter 7, "A Machine for Jumping to Conclusions", which introduces readers to WYSIATI, which stands for "What You See Is All There Is", the cognitive fallacy that is the "machine" referred to in the chapter title. The fallacy is well illustrated by a cartoon I saw once, which shows a man searching desperately under a streetlight at night for his glasses, which have fallen off. When asked why he has been searching in the same place for so long rather than looking elsewhere, the man responds, "Because under the light is the only place I can see." Similarly, when asked, "What's the best education system?" even well-informed researchers in comparative education often engage in question-substitution, another habit of our inherently lazy minds, and convert that question into the much more easily answered "Which country [or school, or teacher, or student] has the best test scores?" and then provide the answer to that in substitution for the answer to the original question, whose answer would necessarily require the human use of qualitative judgement in addition to the machine-like use of memory for relevant quantitative data.

Too many pundits convert the hard question, "Which countries, and which aspects of their systems, might serve as useful models for the United States [or another country] in improving our educational [or health care, or other] system?" into the more easily answered, "Which countries [or states, districts, schools, teachers, students] have better scores than ours?" Those of us with first-hand experience in such jurisdictions are well aware of the tradeoffs required for higher test scores, and avoid publishing such simplistic evaluations, even if experts with power, influence, and overconfidence (most prominently the PISA datameister at the OECD, Andreas Schleicher) are regularly provided with audiences for broadcasts of similar spurious claims and arguments.