When did you first introduce the innovation?
Between 12 and 24 months ago
Please describe the innovation you have developed
This innovation is the development of a teaching package and associated documentation about how to undertake a meta-analyses in R. R is an open-source, freely available, and platform-independent programming language in which many thousands of different packages have been produced to carry out virtually any type of analysis one can imagine. This innovation uses four of these packages in addition to the base programme; three meta-analysis packages (in the title) and compute.es (an effect size-calculator), to teach the process of meta-analysis and to allow students to carry out their own analysis.
What prompted you to develop this innovation?
The Higher Education Funding Council for England (HEFC) have stated that for research to be eligible for submission to the post-2014 Research Excellence Framework (REF) outputs must be deposited in a repository and made open-access within a specified period (HEFCE, 2014), the primary reason for this being the prompt and widespread dissemination of research findings. Alongside concern to make research more widely available, there is also increasing awareness of the importance of the quality of primary research; and the need for rapid and high quality research synthesis. However, within the wider research community there is also concern about an even more significant aspect of research; that is whether research findings are correct (Ioannidis, 2005)
One way of increasing the probability that research findings are indeed correct is to encourage reproducible research; allowing verification of results and alternative analyses to be undertaken. This is not necessarily the same as replication; for which resources may not be available, but it would allow for replication if this was thought important. A key requirement for this is that the data set and full details of the analysis are made available, and although the former may not be possible, the latter is in most cases (Diggle and Zeger, 2009). However, nursing students are often not educated in an environment where this is possible, due to the primacy of menu-driven and proprietary statistical programmes which does not result in reproducible code.
Diggle, PJ., Zeger, SL. (2009) Reproducible research and Biostatistics. 10 (3): 405-408. doi: 10.1093/biostatistics/kxp014.
HEFCE (2014) Open access in the post-2014 Research Excellence Framework. Circular letter 07/2014. London, Higher Education Funding Council for England.
Ioannidis, JPA. (2005) Why most published research findings are false. PLoS Med 2(8): e124.
In your view, what is it about this innovation that makes it different/important?
The innovative aspect of this package, and the associated learning materials is that it encourages students to engage not just in the ‘what to do’ aspect of analysis that is often taught with menu-driven programmes; but the logical process that underlies this process which is required in the direct coding environment used by R. As will be described later, current Government policy is encouraging the move from computer literacy to computer science; using existing menu-driven software encourages the former, while this innovation emphasises the latter.
The innovation is different because it departs from the common approach to teaching statistical methods, which often focus on ease of process as typified by menu-driven approaches towards teaching the logic that is required for coding. This innovation was to develop a teaching package for conducing meta-analyses primarily using metafor, a meta-analysis package that runs in R, an open-source and freely available statistical programming language that can be run on all major computer operating systems. The advantages of using this approach are:
- Publishing code allows for complete replication.
- The process of coding requires students to think in a systematic way and to make explicit decisions about the processes they are using rather than just selecting items from menus.
- The use of freely-available, platform-independent software opens access to those without the resources to purchase expensive software or particular hardware.
- Use of code allows for remote review of the process by course tutors. This is particularly useful when students are not in University as tutorial support can be given by email.
Viechtbauer, W. (2010) Conducting meta-analyses in R with the metafor package. Journal of Statistical Software 36, 3, 2-48. http://www.jstatsoft.org/
To what extent does your innovation make use of existing approaches, resources or technologies?
The innovation requires no extra resources beyond the development of the skills of coding by those running the course or session, as all of the software is freely available and it runs on all major operating platforms and on relatively old hardware. It can therefore be adopted by anyone with even the most basic computer equipment. Having downloaded the programme and packages an internet connection is not required, and because it accepts data in a range of formats (including .csv and direct entry) no particular programme is required for data preparation. Because this innovation does not require major financial investment its importance is not reflected in many institutional metrics which are based on levels of funding; however it is this very characteristic that makes it important.
To what degree has this innovation led to changes in education or clinical practice?
Use of this in a number of programmes has allowed many students to undertake meta-analyses and both understand the process and the output; as well as to be aware of the importance of making the process replicable by incorporating the code into their work. This fits in with Government attempts to shift the focus of technology teaching from computer literacy (being able to use programmes) to computer science (how to code and create programmes) (Gove, 2014); in this case from being able to use programmes such as SPSS and Excel, to writing code and functions in R.
The nature of the innovation and the packages used also mean that students can undertake analyses that can become increasingly sophisticated as their needs develop. Most students have been able to conduct basic meta-analyses as part of the course, but some have used aggregated data and calculated their own-effect size estimates using the advanced methods contained within the various packages. This would not have been possible using existing methods. After the basic teaching of the packages, tutorial support can be entirely remote; indeed with some overseas users support has been entirely remote.
Gove, M. (2014) Michael Gove speaks about computing and education technology. https://www.gov.uk/government/speeches/michael-gove-speaks-about-computing-and-education-technology (accessed 6 August 2015)
What evidence do you have of the impact of the innovation?
The innovation has successfully been used with pre and post-registration post-graduate students. Initial evaluations showed that students found the process challenging but possible, and although this was a self-selecting group who chose to come to the session, subsequently many students chose to do a meta-analysis using this programme. It has also been used with staff in a staff-development programme. Separate evaluation has not yet been carried out; although it was evaluated positively in the module evaluation.
I regularly receive complementary emails about the manual (described below) from researchers around the world; particularly in developing countries.
To what degree has the innovation been disseminated in your organisation or elsewhere?
Publication of an associated manual on ResearchGate has made the innovation more widely available; the first edition was downloaded 1 875 times between July 2014 to April 2015 (Purssell, 2014). A new edition has subsequently been published in May 2015 (Purssell, 2015).
Purssell, E. (2014) Meta-analysis using metafor in R. http://www.researchgate.net/publication/262923251_Meta-analysis_using_metafor_in_R_%28Note_this_has_been_updated_and_this_is_the_older_version_-_see_Meta-analysis_in_R_using_metafor_meta_and_MAd%29
Purssell, E. (2015) Meta-analysis in R using metafor, meta and MAd. http://www.researchgate.net/publication/276271680_Meta-analysis_in_R_using_metafor_meta_and_MAd
Please provide details of any plans you have to disseminate the innovation in the future.
In addition to continuing to teach this to students and staff within the Faculty; it is hoped to extend this innovation and to develop it with other interested people. A freely available video is also being developed as part of a Faculty initiative to develop an International MSc Programme; this will be uploaded to you-tube in addition to being hosted by the Faculty.
One benefit of being recognised by the Council of Deans would be an increased profile for this and the ability to bring it to a wider audience within the UK Health and Education Communities. In particular the innovation is not widely recognised because of its lack of need for funding; internal metrics being largely based around financial return, yet it is this very feature that makes this innovation attractive to those with limited resources.