Chapter 2 Evidence synthesis overview

There are a variety of different ways that the term “Evidence synthesis” has been defined. Here we try and come up with a single sentence definition and then expand on this to talk about why evidence synthesis is important. In this book we will highlight different tools and approaches that can be used to carry out evidence synthesis in R and also highlight ways to identify where bias may influence the evidence synthesis presented to a decision maker. We hope to develop this book as a source of information and discussion on all aspects of evidence synthesis. Evidence synthesis researchers are generally very interdisciplinary because the problems that we seek to address are found across different areas of society and science. Often advances in evidence synthesis tools come from the medical field but are then adapted and adopted for use in other fields. There is now a strong history of evidence synthesis in other disciplines such as education, welfare, conservation, policing, international development, and so on.

A single sentence definition of “Evidence Synthesis” “Bringing together evidence from a variety of sources in a robust and transparent way to help inform decisions about specific issues or interventions”

2.1 What is evidence?

Evidence in “Evidence synthesis” is often (mistakenly) thought to only encompass peer-reviewed scientific literature. This, of course, does make up a large and important evidence resource but other sources of evidence are also seen as important. For example, reports from conservation organisations, governments, local managers, local societies (e.g. bird clubs) could all be considered evidence. We often refer to these types of reports as “grey literature.” Other sources of evidence include expert opinions - this is particular important for new emerging issues, but can be used to supplement other sources of evidence. Stakeholder opinions can also be considered evidence - in medicine this might be the patients who benefit from a particular drug, in conservation these could be local and/or indigenous people. As you can see the evidence that we use in evidence synthesis can be a lot wider than just the published scientific literature. One source of evidence in conservation which might not be considered in other fields is the results of simulation models where we do not have enough information about a species but using our theoretical knowledge or information from other closely related species we can simulate the patterns and processes associated with the species even if we do not have much chance to observe it in the wild (Critically Endangered species for example).

2.2 Why do we need evidence synthesis?

There are many reasons why we should not just use a single study or a single piece of evidence to base a decision on. One of the main reasons is that single studies are like a snapshot and as such may not give us a full picture of the size or direction (positive or negative) of an effect of an intervention. Replication is a cornerstone of science and without replication we can not be certain of our findings. Replication allows us to know how general the effect we observed is and how generalisable it is to other conditions (e.g. in other countries, climates, habitats, etc.). Evidence synthesis brings together these replications and determines the pooled effect (either qualitatively or quantitatively).

Single studies may be plagued by different biases that we as researchers may not be aware of. These biases are not necessarily the result of nefarious actions by the researchers (e.g. data fabrication) but there has been some recent well-publicised examples of this. A lot of “questionable research practices” come from actions that are mostly “bad practice” or result from bad training. One common action (which is how I was trained as a student) is known as “HARKing” - Hypothesizing after the results are known. This is where we start a study with one hypothesis in mind but after we see patterns in the data we omit that hypothesis and choose one that fits the data better (and tells a neater or more exciting story). Evidence synthesis assesses the quality of studies to try and reduce some of these biases.