Where there is smoke, there is (the potential for) fire: Soft indicators of research and policy impact

Citation

Gardner J, Holmes B & Leitch R (2008) Where there is smoke, there is (the potential for) fire: Soft indicators of research and policy impact. Cambridge Journal of Education, 38 (1), pp. 89-104. https://doi.org/10.1080/03057640801890004

Abstract
There is a growing literature examining the impact of research on informing policy, and of research and policy on practice. Research and policy do not have the same types of impact on practice but can be evaluated using similar approaches. Sometimes the literature provides a platform for methodological debate but mostly it is concerned with how research can link to improvements in the process and outcomes of education, how it can promote innovative policies and practice, and how it may be successfully disseminated. Whether research‐informed or research‐based, policy and its implementation is often assessed on such ‘hard' indicators of impact as changes in the number of students gaining five or more A to C grades in national examinations or a percentage fall in the number of exclusions in inner city schools. Such measures are necessarily crude, with large samples smoothing out errors and disguising instances of significant success or failure. Even when ‘measurable' in such a fashion, however, the impact of any educational change or intervention may require a period of years to become observable. This paper considers circumstances in which short‐term change may be implausible or difficult to observe. It explores how impact is currently theorized and researched and promotes the concept of ‘soft' indicators of impact in circumstances in which the pursuit of conventional quantitative and qualitative evidence is rendered impractical within a reasonable cost and timeframe. Such indicators are characterized by their avowedly subjective, anecdotal and impressionistic provenance and have particular importance in the context of complex community education issues where the assessment of any impact often faces considerable problems of access. These indicators include the testimonies of those on whom the research intervention or policy focuses (for example, students, adult learners), the formative effects that are often reported (for example, by head teachers, community leaders) and media coverage. The collation and convergence of a wide variety of soft indicators (Where there is smoke ...) is argued to offer a credible means of identifying subtle processes that are often neglected as evidence of potential and actual impact (... there is fire).

Keywords
evaluation; impact of research; soft indicators; anecdotal and impressionistic evidence

Journal
Cambridge Journal of Education: Volume 38, Issue 1