Welcome!

Improving Education: A triumph of hope over experience
Inaugural Lecture of Professor Robert Coe, Durham University, 18 June 2013

Despite the apparently plausible and widespread belief to the contrary, the evidence that levels of attainment in
schools in England have systematically improved over the last 30 years is unconvincing. Much of what is claimed
as school improvement is illusory, and many of the most commonly advocated strategies for improvement are
not robustly proven to work. Even the claims of school effectiveness research – that we can identify good schools
and teachers, and the practices that make them good – seem not to stand up to critical scrutiny. Recent growth of
interest in evidence-based practice and policy appears to offer a way forward; but the evidence from past
attempts to implement evidence-based approaches is rather disappointing. Overall, an honest and critical
appraisal of our experience of trying to improve education is that, despite the best intentions and huge
investment, we have failed – so far – to achieve it.
Nevertheless, we have reason to be hopeful, provided we are willing to learn from our experience. Specifically, I
will argue that we need to do four things: to be clear what kinds of learning we value; to evaluate, and measure
properly, teaching quality; to invest in high-quality professional development; and to evaluate robustly the impact
of changes we make.

Introduction
I am a school teacher who became interested in educational research, and ended up doing that instead. Most of
my research has looked at questions that are directly relevant to practice or policy: I want to make education
better for children and young people. I know there is nothing special about that; the hard question is: How?
On one level, my analysis is quite bleak: standards have not risen; teaching has not improved; research that has
tried to support improvement has generally not succeeded; even identifying which schools and teachers are good
is more difficult than we thought. That is our experience, so far. Recognising this is important, not because I enjoy
puncturing inflated assertions of success and what seem to me to be complacent and uncritical claims of ‘fools

gold’ improvement (although, I confess, I do), but because I think it is time we stopped repeating the same
mistakes.
I am optimistic about our capacity to learn from what has not worked, not to keep on repeating the same
mistakes, and to use what knowledge we have about what seems most likely to make a difference. Most of all, I
believe in the power of evaluation to tell us what is working and of feedback loops to allow that evaluation to
influence practice. Despite our experience, there are strong grounds for hope that we can do better.

Experience
First I will talk about the past, and ask whether what we have done so far has worked.
Have educational standards really risen?
This is inevitably controversial; my answer will surely offend many people who have invested energy and
resources into trying to raise standards. Those who have invested huge energy and resources may be hugely
offended. So why do I feel the need to say things that I know will offend them?
I want to be clear that I do not mean to imply any criticism of teachers or anyone else working in education. I was
a teacher myself and I know how hard teachers work, how committed they are to doing the best for, and getting
the best from, their students, no matter what challenges they face.
However, if it is true that despite the huge efforts we have made to improve education not much has changed,
there are important lessons for us to learn. One would be that effort and good intentions are not enough; we
have to work smarter, not just harder. Another would be that we must look carefully at the strategies we have
been using to improve, and replace them with some different ones. A third lesson is that a more critical and
realistic approach to evaluation may be required. An uncritical belief that things are improving may be
comforting, but is ultimately self-deceiving and unproductive.
In short, I find it hard to see how we can make real improvement until we accept the unpalatable truth that we
have so far failed to achieve it.
Unfortunately, a clear and definitive answer to the question of whether standards have risen is not possible. The
best I think we can say is that overall there probably has not been much change. However, we are limited by the
fact that in England there has been no systematic, rigorous collection of high-quality data on attainment that
could answer the question about systemic changes in standards. And we might well disagree about what we
mean by ‘standards’ or how they should be measured. The evidence we have is patchy and inadequate, but it is
the best we have. There are three main types of evidence: international surveys, independent studies and
national examinations.
Evidence from international surveys
International surveys of attainment, such as PISA, PIRLS and TIMSS, have become increasingly prominent as
indicators of our standing in the international league table of performance. These studies are not designed to
evaluate change over time and their interpretation and use is inevitably problematic (see e.g. Brown, 1998;
Jerrim, 2011). Nevertheless, they provide a source of evidence, based on a high-quality assessment development
process, robust attempts to equate scores across testing occasions and a rigorous national sampling process.
Figure 1 shows the scaled average performance for England in PISA, PIRLS and TIMSS between 1995 and 2011. It
is a mixed picture. In some curriculum areas, for some ages, performance has improved (e.g. maths age 10 in

TIMSS). For others, it has declined (e.g. PISA, age 15, all subjects). There isn’t really a coherent story here. There
might be a case for saying that gains at age 10 are of little value if they are lost by age 14 or 15, hence focusing on
the results for older children. For them, TIMSS suggests little change, PISA seems to show a fall.
Just to put these results in context, a change of 25 points in PISA has been estimated to be worth £4 trillion to
England’s GDP (Hanushek and Woessmann, 2010) and would typically be the difference between an average
performing country (such as England) and one ranked 5-10 internationally (such as Canada or New Zealand)
(OECD, 2010). The gap between the average and the top performing country is typically 50-60 points.1
However, when official sources (a DfE press release and comments from Sir Michael Willshaw, Chief Inspector)
cited the PISA decline as evidence of falling standards, they were censured by the UK Statistics Agency for making
comparisons that were ‘statistically problematic’ (Stewart, 2012), partly because the response rate in England in
2000 and 2003 had been low. An analysis by Jerrim (2011) tried to estimate the effects of higher non-response for
England on PISA in 2000 and 2003, as well as for the change in test month and – an important but little known
change – the fact that in 2006 & 2009 it was only Y11 15-year olds who were included (in 2000 & 2003 all 15-year
olds, whether in Y10 or Y11, were sampled). Overall, these corrections suggested there had probably not been
much change between 2000 and 2009.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------