This blog post is a cross-post from EdTech Hub‘s blog https://edtechhub.org/2019/12/18/reviewing_the_research_literature_in_educational_technology_for_development_balancing_rigour_and_inclusivity/ (Creative Commons Attribution 4.0). The blog post was written by Katy Jordan on the 18th of December of 2019.
Within the Research Sphere of the EdTech Hub, one of the main activities within the inception phase has been to conduct a literature review. Undertaking a literature review is established practice as a typical first step in any research project, to establish an informed foundation upon which to conduct further research.
The scope of the EdTech Hub is unusually wide for conducting a literature review. EdTech itself is a term which can be applied to a wide range of technologies used in educational settings. Similarly, the focus on low- and middle-income countries (LMICs) encompasses a diverse range of countries, territories and regions.
Given the wide scope of the programme, a two-stage approach has been used: initially, a large-scale scoping review providing a breadth of understanding of the topic, followed by systematic reviews focused on particular themes. Systematic reviews consider papers published on a very specific topic and compare the findings across studies in detail. Although historically systematic reviews have their roots within health and biomedical sciences, systematic reviews are increasing in popularity as a research methodology within the social sciences (Figure 1).
Figure 1: Number of records returned via Scopus based on the query “a systematic review”, for journal papers within the Social Sciences (orange markers, left-hand axis). 2019 is incomplete (search undertaken 10/10/2019). The trend in number of articles indexed in Scopus for Social Sciences as a whole is included for comparison (grey markers, right-hand axis).
Before conducting a literature review, it is necessary to set certain bounds to define how literature will be found and selected for inclusion. Transparency in this respect is necessary to ensure that others reading the findings can judge to what extent there are limitations or strengths to the evidence base, and hence how reliable the findings are. Setting and clearly articulating the sources of literature and inclusion criteria is also central to the rigour of systematic reviews as a methodology as it facilitates the replicability of approaches and results. Details such as this are published in search protocol documents.
However, there are no established protocols associated with this topic, so there was an immediate question as to how we should tailor our approach. To learn from the field and develop a protocol, we drew upon 22 existing literature review-based papers with foci related to those of the EdTech Hub. Further details will be available shortly in our forthcoming working paper (docs.edtechhub.org/lib/NM6CPLE9, DOI:10.5281/zenodo.3523943), but here we will focus on an issue which was surfaced in the analysis and required particular attention in our context: the issue of balancing academic rigour with inclusivity.
Two of the main components of a literature review protocol are the sources of information, and setting the criteria which found literature will be judged against in order to be included. In terms of finding literature, databases and other data sources, such as academic journals and institutional repositories, were listed in the papers dealing specifically with literature reviews (19 papers). The databases and their frequency, grouped according to subject area, are shown in Figure 2.
Figure 2: Research databases and other data sources used within the sampled documents, arranged according to broader subject area. Number of occurrences in brackets.
In the papers reviewed, a range of criteria was recorded for the inclusion and exclusion of literature. The most frequently used criteria included whether articles were peer-reviewed; being published in an academic journal; presence of particular keywords in title and/or abstract; type of research (e.g. empirical); and language.
Underpinning several of these factors is an assumption that being published in academic journals is a sign of quality, and academic journal articles are readily indexed within the main databases used for literature searches. However, it is important to caution against relying exclusively on academic journals as a source for the EdTech Hub’s review, as publishing in academic journals demonstrates biases according to location (see Figure 3, for example).
Figure 3: Global map, where territory size is depicted proportional to the number of scientific journal articles published in 2016. Worldmapper website: https://worldmapper.org/maps/science-paperspublished-2016/?sf_action=get_data&sf_data=results&_sft_product_cat=science&sf_paged=2 (CC BY-NC-SA 4.0)
The EdTech Hub’s approach needs to pay particular attention to setting the bounds of its literature review in a way which balances rigour with inclusivity, which is also important for anyone considering a literature review around LMICs. Database searches will be supplemented by opportunistic searches through experts and informal networks. Grey literature is at risk of being excluded by database searches yet blogs, presentations, informal publications and other communications may play an important role.
Searching for literature is just the first step of the process, and we will be writing further blog posts along the way. Following on from highlighting the need to be aware of biases in academic publishing, in the next blog post from the Research Sphere, we will discuss how we have applied the need for inclusivity to our inclusion criteria.
