This post originally appeared here on 16th June 2016 under Creative Commons Attribution 4.0.

Back in April, I was asked the following question: What is the evidence that online education programmes (including the use of digital tools or apps) lead to measurable learning gains? The question was asked especially with regard to low- and middle-income countries.

What evidence is there?

In high-income countries there is extensive work on use of digital technology (covering a range of uses and tools). For example, in the UK, the EEF (https://educationendowmentfoundation.org.uk/evidence/teaching-learning-toolkit) ranks digital technology as “moderate impact for moderate cost, based on extensive evidence”. This indicates that there is evidence for impact (in specific settings, in developed countries), but that the impact is moderate, and carries a moderate cost. So if costs are not a major concern, digital technology can be a useful addition to support learning. However, there is a lot of hype around this, and there is no evidence that, for example, tablets, or other technologies, are a magic bullet (see our literature review on tablets), in low- or high-income countries.

What can we say about low- and middle-income countries? Importantly, there simply isn’t the same extensive evidence as we have in high-income countries (see our literature review and references therein). However, there is plenty of evidence about infrastructural constraints (in many low-income countries) on technology use for learning, and so costs need to be assumed to be higher (than in high-income countries). What about educational impact? Is the effect of digital technology lower, the same, or higher, compared to high-income countries? This can be argued in different ways, and a generic answer is impossible. However, one might suggest that the impact will be lower, perhaps at best the same, simply due to infrastructural constraints. So if you were to argue that the impact is much higher (‘magic bullet’) you would need to be able to back this up quite strongly (through new research).

Pitchford’s work deserves a mention (as an RCT using tablets in Malawi). However, one could criticise the study on the basis of the control (as ‘no intervention’). This means that the RCT cannot determine the added value of ICT (in comparison with a time/effort-equivalent intervention without ICT). Overall, the lack of robust (quasi-)experimental designs/mixed methods is a significant obstacle to understanding cause and effect (and guide future investment), even in some multi-million-pound interventions.

In the literature review, we do point out that often evaluations talk about short term motivational gains (say, after a few months), and clearly any kind of introduction of technology in low-resource settings is bound to lead to excitement. However, such evaluations are often short-term, and evaluations looking at the longer term (and learning gains, rather than just motivational gains) are lacking. (The literature review can be seen as complementary to the Educational Technology Topic Guide, providing additional evidence especially on tablets.)

What does the broader evidence suggest?

Focus on teacher development. Let’s rephrase the question, to get more definite answers. It is clear that effective use of digital technology depends on teachers, and teacher professional development, which is often overlooked. If you framed the question as ’Does the use of digital technology in schools lead to measurable learning gains without appropriate teacher professional development?’, the answer would be no. Some technology-focussed camps might argue with that (or might have argued with that in the past), but few education researchers would. Quantitative research evidence demonstrates that interventions involving TPD can have an impact on primary school student learning (McEwan, 2014), and the author comments that “it is telling that almost all successful instructional interventions in our sample include at least a minimal attempt to develop teachers’ capacity to deliver effective classroom instruction” (ibid., p. 27-28). Similarly Hattie’s (2009) renowned meta-analysis shows an overall large effect size (0.62) for professional development.

Focus on what works in the classroom. Another way to phrase the question (given the reference to basic education/skills, particularly literacy and numeracy) is ‘Is digital technology the solution to SDG4, quality Education for All, including the poorest and most marginalised?’ To answer this, let’s go back to the EEF toolkit, and compare digital technology (moderate impact, moderate cost, extensive evidence) with other interventions, such as meta-cognition and self-regulation (high impact for very low cost, extensive evidence), collaborative learning (moderate impact for very low cost, extensive evidence) etc. (interactive pedagogy). So if you want Value for Money – maximising each unit of learning impact per unit of cost -, you would start with interactive pedagogy (meta-cognition, collaborative learning, feedback etc.), rather than with digital technology.

And, unlike digital technology, there is a degree of evidence from low-/middle-income countries that confirms that such interactive pedagogy approaches do have an impact on pupil learning (even given the overall settings, such as large or multi-grade classes; Westbrook et al., 2013; Nag et. al., 2014). Moreover, gender-responsive pedagogy is highly relevant to addressing gender issues (Unterhalter et al., 2014). There is also evidence that it’s possible to design effective teacher development programmes, that empower teachers to use such approaches (e.g. Orr et al., 2013, as well as our own experience with www.oer4schools.org), and tentative evidence that such (school-based, peer facilitated) programmes can be sustainable and scalable.

It seems that programmes focussing on interactive pedagogy (incorporating feedback, collaboration, questioning, etc), aimed at those teachers providing learning opportunities for the poorest) do have a chance to work (based on reasonable evidence), while it is hard to see how programmes focussing on digital technology could reach the poorest at scale (given the limited finances available). So while digital technology does have moderate impact on learning, and does ultimately have a place in holistic approaches to learning, it is not the place to start an SDG4 programme.

If you were to implement a technology programme, where would you start?

However, the above does not suggest that experimentation with technology should be totally out of the question. What are the useful starting points?

Teachers and teacher development. In the question “learning gains” was probably intended as ‘learning gains for pupils in school’. However, technology can play a role in teacher development. Again, there is no extensive hard evidence, but there are encouraging stories, and it is easier to see how one might support teachers’ own learning with technology (at scale), rather than supporting pupils directly.

To support the teachers teaching the poorest and most marginalised pupils, one would suggest a dual strategy:

1. Start with teacher development (not technology-based), along the lines of what OER4Schools / T-TEL is doing (see our publications). A 20-school-week programme (e.g with two hours of teacher group meetings, supported by the right materials and teacher incentives) could have a significant impact. There is good evidence that this would work (c.f. DfID literature reviews cited above).
2. Consider how such programmes could be enhanced with a small amount of technology for teachers. For the technology-enhanced programme, (1) serves as a fallback: the programme is not dependent on technology, but enhanced by it.

The enhancements in (2) might mean:

• Giving a tablet to a school, for use during teacher development (e.g. to play video, or to access additional supporting materials for teachers).
• Give an SD card/memory stick to a school (used by teachers’ own devices, ‘BYOD’), or provide low-cost content servers (e.g. Raspberry Pi-based, such as Open Learning Exchange Ghana) to schools or teacher centres.
• Make access to educational resources for teachers (e.g. on a gov’t portal) free, e.g. the SA government negotiates with SA telecoms providers to ensure that access to http://oer.gov.za is free (i.e. does not incur mobile data charges). This has a precedent in ‘free basics’ and Wikipedia Zero.

If we had funding to allocate (with the remit of including technology for learning), we would propose mixed-methods research (including experimental design) that explores options 1 and 2 (e.g. as arms of a randomised control trial).

Resource development. Apart from better teachers, of course resources for the classroom are needed. We suggest that a way of using technology (outside the classroom) is to draw on open textbooks, such as the open textbooks being developed by the US government (K12 OER collaborative), because research convinced the US government that these are more cost-effective than commercial textbooks. The use of technology here is not in the classroom itself, but in the development process of the books. Significant cost-savings could be had by adapting these books for low-/middle-income countries, and printing them on demand for use in schools. This has the added benefit that for those with their own devices, that the books are available digitally as well.

If we had funding to allocate, we would propose setting up a think-tank about the use of open content (for teacher development and in the classroom) in low-/middle-income countries, with the remit to develop recommendations for funders and governments.

Technology in the classroom. To look at technology use in classrooms in low-/middle-income countries, we suggest that where technology is used, it should be used in conjunction with effective pedagogical strategies, such as meta-cognition, collaborative learning, etc. There seems to be little benefit in using technology purely as a means to deliver content.

A potential avenue, with promising added benefit, is to use technology (very part-time, i.e. with high student-device ratios) for difficult to understand topics. This would include, for example, common misconceptions in mathematics, and in practice could draw on software like the widely-used GeoGebra software. Note that this is very different from a push for e-learning: this is not about transmitting content, but the -meta-cognitive, collaborative- engagement of students with key misconceptions. More traditional e-learning responds to a different set of constraints (e.g. adult/young adult part-time learning with pre-existing moderate connectivity), rather than in the poorest and most marginalised school environments.

If we had funding to allocate, we would suggest research on targeted use of technology: in the context of interactive pedagogy, high student-device ratios, addressing specific learning areas, supported by extensive school-based, peer facilitated teacher development.

References

• Haßler, B., Major, L., & Hennessy, S. (2015). Tablets in schools: A critical review of the evidence for learning gains. Journal of Computer Assisted Learning http://onlinelibrary.wiley.com/doi/10.1111/jcal.12123/full
• Haßler, B., & Mays, T. (2015). Open Content. In P. Hwa Ang & R. Mansell (Eds.), International Encyclopedia of Digital Communication and Society. Wiley-Blackwell. Retrieved from http://bjohas.de/Publications/Hassler_Mays_OpenContent
• Haßler, B., Hennessy, S., & Hofmann, R., (preprint available). Experiences of developing and trialling the OER4Schools professional development programme: Implications for sustaining and scaling pedagogic innovation in sub-Saharan Africa.
• Haßler, B., Hennessy, S., & Hofmann, R., with Makonga, A. (preprint available). The OER4Schools professional development programme: Outcomes of a sustained trial in sub-Saharan Africa.
• Haßler, B., Major, L., Warwick, P., Watson, S., Hennessy, S., & Nichol, B. (2016). Perspectives on Technology, Resources and Learning – Productive Classroom Practices, Effective Teacher Professional Development. Faculty of Education, University of Cambridge.
• Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge.
• McEwan, P. J. (2013). Improving learning in primary schools of developing countries: A meta-analysis of randomized experiments. Wellesley, United States: Wellesley College.
• Nag, S., Chiat, S., Torgerson, C., & Snowling, M. J. (2014). Literacy, Foundation Learning and Assessment in Developing Countries. Retrieved from https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/305150/Literacy-foundation-learning-assessment.pdf
• Orr, D., Westbrook, J., Pryor, J., Durrani, N., Sebba, J., Adu-Yeboah, C., & others. (2013). What are the impacts and cost-effectiveness of strategies to improve performance of untrained and under-trained teachers in the classroom in developing countries?: systematic review. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. Retrieved from http://sro.sussex.ac.uk/43901/1/Undertrained_teachers_2013_Orr.pdf
• Pitchford, N. J. (2015). Development of early mathematical skills with a tablet intervention: a randomized control trial in Malawi. Frontiers in Psychology, 6. http://doi.org/10.3389/fpsyg.2015.00485
• Power, T. (2014). Educational Technology Topic Guide. Retrieved from http://www.heart-resources.org/topic/educational-technology/
• Unterhalter, E., North, A., Arnot, M., Lloyd, C., Moletsane, L., Murphy-Graham, E., … Saito, M. (2014). Girls’ education and gender equality. Retrieved from http://r4d.dfid.gov.uk/pdf/outputs/HumanDev_evidence/Girls_Education_Literature_Review_2014_Unterhalter.pdf
• Westbrook, J., Durrani, N., Brown, R., Orr, D., Pryor, J., Boddy, J., & Salvi, F. (2013). Pedagogy, curriculum, teaching practices and teacher education in developing countries: final report. (No. 2110). Retrieved from https://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3433