Jan 1, 2019
John Hattie is an education
researcher from New Zealand with a very ambitious goal: to
synthesise the myriad quantitative research studies on education in
a single publication. The number of articles affecting his book
Visible Learning numbers in the region of 80 thousand (!).
The results of his analysis have
been hailed as the "Holy Grail" of education by such prestigious
authorities as the Times Education Supplement. So, how did he and
his team do it?
Hattie uses an approach known as meta-analysis. Meta-analyses take numerous research articles trying to measure an effect and compare them in order to ultimately determine the size of the effect. They are common in medicine, where they are often used to elucidate whether a drug is truly effective or not, as a single study may incorrectly show a drug to be effective simply by chance.
However, Hattie goes one step further and carries out a meta-analysis on other meta-analyses, forming a sort of "meta-meta-analysis". With this approach, his team only directly work with 400 articles, as each of these is a meta-analysis of tens or hundreds of other articles, which is how we reach the gargantuan number 80 thousand.
You would have thought that such an ambitious, influential, and widely praised work would have come under much careful scrutiny. And you would have thought that since it is so statistical, numerous other researchers in the field of education would have performed at least a surface-level plausibility check.
However, you may be disappointed.
It took two years for anybody to even begin to notice the glaring
statistical errors behind this work, and even when they were
noticed, Hattie's team didn't treat them with the gravity they
deserved. Methodological criticism gradually increased in number,
and by now it is clear that the "Holy Grail" has numerous leaky
holes.
In this episode, after
introducing Visible Learning, I go on to take some
highlights from one such criticism, entitled How to engage in pseudoscience with real
data: A criticism of John Hattie’s arguments in Visible Learning
from the perspective of a statistician, written by Canadian
statistician Pierre-Jerome Bergeron. I aim to explain the most
accessible points, and leave the more complex parts of the article
for those with the interest and mathematical acumen to look
up.
This sort of thing seems to happen a lot in education. It's one of the reasons why it's so hard to figure out how things work and what's actually true in the field. At least I can warn people about the problems with this still widely cited work.
Enjoy the episode.