What is evidence?
Author: Dr Philip Davies
Evidence is information or data that helps us understand a problem, or support/reject a proposition or hypothesis. Evidence can be in the form of quantitative or qualitative data, and can be collected using experimental or naturalistic methods of inquiry. If one wants to know the nature and magnitude of a problem in a population, and what is driving the problem, this can usually be found using statistical data from population surveys collected by national and international agencies, or location-specific surveys collected by researchers in the field. Additionally, it will require both administrative data, such as admissions to clinics and hospitals, and qualitative data collected by interviews with personnel working at the ‘coal face’ of the problem and its manifestations.
Evidence can also help us know whether a particular intervention is more effective than an alternative intervention, or indeed not intervening at all, in terms of achieving a desired outcome. Such evidence requires data collected using experimental or quasi-experimental methods in which some people will be exposed to intervention x and others will be exposed to intervention y. This type of evidence usually needs to be supplemented by qualitative evidence to understand why, how, and under what conditions the desired outcome(s) will be achieved.
Evidence is seldom definitive but is usually probabilistic, tentative and contestable. Evidence is seldom, if ever, self-evident. It requires the expertise, experience and judgement of its users to interpret it and decide how to use it to address the problem in hand. Making these judgements about evidence is confounded by the fact that not all evidence is of equal quality in terms of validity, reliability, credibility or defensibility. The evidence provided in a tabloid newspaper is normally not as valid, reliable, credible or defensible as that reported in a scientific journal. Evidence that is reported in scientific journals, however, may not always be of the highest quality, largely due to different standards of quality across different journals, and different levels of rigor in terms of the editorial and peer review processes.
Probably the highest quality evidence that is available for decision making comes from systematic reviews and other types of research synthesis. These reviews do not consider evidence from a single study only, or even a few studies, but from a comprehensive search of all studies on a topic over a ten to twenty-year period. Systematic reviews then subject each identified study to rigorous critical appraisal using agreed and transparent criteria of quality. This separates higher quality studies (the wheat) from lower quality studies (the chaff) and establishes the overall balance of evidence on a topic. This includes a balanced appraisal of the positive findings as well as the negative findings from the available evidence, as well as the intended and unintended consequences of a policy or intervention. Statistical and qualitative analysis of this resultant evidence provides greater precision about the size of the problem in hand, the likely outcomes of different policy options, and details of what needs to be done in terms of implementation in order to achieve the desired outcomes.
Evidence, then, is not a panacea for the many problems that decision makers face. When properly gathered and scrutinised, however, evidence provides decision makers with a powerful tool with which to get things right more often than getting things wrong. Evidence always requires thought and careful reflection. Evidence-based policy is not substitute for thinking-based policy.
Philip Davies PhD
Oxford Evidentia Ltd
February 2016
Dr Davies is Executive Director of Oxford Evidentia, a UK-based professional development and consulting company that undertakes capacity building in impact evaluation, research synthesis and the use of high quality evidence for policy making, good governance and public service delivery.
From 2012-2015 Dr Davies was Head of the London office of 3ie, the International Initiative for Impact Evaluation and had responsibilities for 3ie’s Systematic Reviews and Research Synthesis programme.
From 2000-2007 Dr Davies was a senior civil servant in the UK Cabinet Office and HM Treasury, responsible for policy evaluation and analysis. Before joining the Cabinet Office Dr Davies was a University Lecturer in Social and Political Science at the University of Oxford, and he has held academic positions at the University of Aberdeen and the University of California, San Diego.
Dr Davies has substantive expertise in health and health care, education, social welfare, crime and justice, and international development. He has taught courses on policy evaluation, evidence-based policy making, and the analysis and use of evidence across the UK and in the USA, Canada, and various countries of Europe, Africa, Australasia.
Dr Davies is a member of the Evidence Aid board of trustees.
Views expressed through our blog are those of the author(s) alone and do not necessarily reflect the official position of Evidence Aid.