An inclusive approach to searching for evidence on EdTech in low- and middle-income countries

This blog post is a cross-post from EdTech Hub‘s blog https://edtechhub.org/2020/02/18/an-inclusive-approach-to-searching-for-evidence-on-edtech-in-low-and-middle-income-countries/ (Creative Commons Attribution 4.0). The blog post was written by Meaghan Brugha and Katy Jordan on the 18th of February 2020.

A searchable database

The EdTech Hub has undertaken a large-scale search for publications on technology use in education in low- and middle- income countries. During this process, we created an internal research database. This is searchable through the use of a variety of filters, such as country or intervention of focus. Analysis of the database helps us to ground our wider research, innovation and engagement activities as a Hub within the scope and quality of the evidence base.

An example search to find all relevant evidence on educational technology initiatives in Gambia

Before starting to unpack the literature, part of our learning has been the process of developing an inclusive methodology and set of inclusion criteria for the evidence that we will include in the internal database. 

Our recent blog post by Katy explains how we developed an inclusive, systematic and replicable approach for the types of literature we included. We did this by broadening the search to include typically under-represented literature, while still ensuring it was rigorous. Having established the scope of the literature search, the next step was to identify a set of inclusion criteria.   

It was our experience that — while most types of reviews do report their search terms and search strings — very few provide substantive details on their process for selecting those specific terms. That’s why we decided to share our process in this blog post, where we offer insight into how we developed our keywords (our full keyword inventory can be accessed here) and inclusion criteria. We hope this may prove to be a helpful resource for those undertaking similar reviews, and of course, we would appreciate the well-informed and always welcomed feedback from our community. 

Identifying our inclusion criteria

Our inclusion criteria were selected in careful consideration of the research questions of the EdTech Hub. Decisions on the first three inclusion criteria were fairly standard and straightforward:

1. Publication date: Approximately the last ten years; more precisely, we chose 2007 as a starting date due to the advent of more advanced mobile operating systems (such as Android).
2. Type of publication: As inclusive as possible: not only peer-reviewed academic journal articles but also grey literature, such as project reports.
3. Research design: Again, broad and inclusive. The EdTech research is diverse, and we chose to include both empirical (various designs) and non-empirical publications. 

The remaining four inclusion criteria, however, each came with their own interesting challenges, which we expand on below:

1. Language of the evidence: Including multiple search languages broadens and diversifies the search results, and may also serve as a way to access previously underrepresented literature. But how do we know that we are searching in the right languages? Our initial starting point for choosing publication languages was the set of official UN languages — it simplified the approach and allowed the Hub to avoid the politics of language inclusion. However, this excludes Portuguese, when earlier work highlighted the importance of searching in Portuguese for relevant publications in sub-Saharan Africa. During our search trials, it further became clear that the inclusion of some languages would be redundant while the oversight of others seems unjustified. Our methodology report describes this challenge in further detail, and we will discuss strategies for mitigating this through sub-studies in a post coming soon!
2. Geographic location: The phrase ‘low- and middle-income countries’ is widely used, but the definition is contested. Following extensive conversations with internal and external colleagues, alongside multiple iterations and trials, we agreed on a list of criteria that countries (and disputed areas) had to meet for inclusion in our review. If a country or disputed area met one of the following criteria, they were included: 
   • Human Development Index (HDI) ranking of ‘low’ or ‘medium’ 
   • Inequality-adjusted HDI ranking of less than 0.69 
   • Multidimensional Poverty Index greater than 0.31 
   • Gini coefficient index greater than 40
   • a disputed territory (recognised by the UN) that borders a country that qualifies for inclusion (via the above criteria)
   • countries were excluded if they had ‘very high’ HDI
3. Intervention: Any form of technology that is used for educational purposes was included. All hardware, software, content (digital and non-digital) and technology-related regulations (e.g. licences) were included if they were used for educational purposes. In order to avoid significant bias in terminology, we had to provide a clear rationale for the inclusion of organisation names or specific technologies, such as an iPhone or iPad.
4. Population: Population specifiers included learners (early childhood and basic education) and pre- and in-service teachers. It was important for our review to include the most marginalised: students accessing informal and non-formal education, street children, students with special educational needs and disabilities, refugee education, and girls. Search terms were developed that were reflective of these populations.

Our inclusion criteria served as a foundation for the development of relevant and inclusive search terms. We aimed to make it as systematic and unbiased as possible. We recognise that in low- and middle- income countries there is a significant amount of evidence that is not always searchable in peer-reviewed journals or databases, yet may be highly relevant for our review. 

We also note that publications on barriers to adopting and scaling EdTech interventions could be highly relevant, but may not make any explicit reference to EdTech interventions. Findings from trials we conducted suggest that it is likely that these publications would not be picked up using our preliminary search terms. An important conclusion of this process was the decision to undertake a dedicated study on barriers to adopting and scaling EdTech interventions, rather than relying on inclusion in the wider literature search.

Interested in the findings of our review? Watch this space!