Summative Evaluation of Partnership for Enhanced and Blended Learning project

The challenge

Universities in East Africa are experiencing increasing teaching burdens due to rapidly growing student populations and academic staff shortages, resulting in adverse impacts on teaching quality and the student experience. Led by the Association of Commonwealth Universities (ACU), the  Partnership for Enhanced and Blended Learning (PEBL) project has worked with 23 universities across East Africa to enhance the quality of teaching and student outcomes by scaling up capacity for blended learning design and delivery, in order to address the growing shortage of academic staff. 

PEBL is supported by UK FCDO's SPHEIR programme.

INASP was commissioned by ACU to carry out a summative evaluation of PEBL, in order to identify the extent to which the project has achieved it's intended outcome of  “Increased flexibility in East African Higher Education systems to expand capacity to meet increasing graduate learning demands without eroding quality”.

The evaluation covered the period comprising September 2017 to April 2021, and the lessons from the evaluation will help inform similar projects and/or scaling up in the future. It focused on Kenya, Rwanda, Tanzania and Uganda, and considers project effectiveness, changes on institutionalisation of policies and practices, changes to capacity building, quality of blended learning, sustainability, vale-for-money, and overall lessons learned. 

Our approach

INASP’s approach was to facilitate a rigorous self-evaluation using a collaborative outcomes reporting (COR) approach to develop a performance story. This was a theory-based (starting with the Theory of Change), realist (taking account of the context) approach to assess whether the intervention achieved the intended outcomes. In addition, our approach was :

• Highly participatory, involving a wide range of project stakeholders to co-analyse the evidence and co-produce the final conclusions and recommendations. Involving all stakeholders makes it much more likely that the results will be used.  
• Both summative and utilisation-focused in order to identify results alongside what worked well and should be scaled up, what didn’t work well and should be avoided, and how projects like this can be implemented most effectively.
• COVID-19 resilient: it builds on INASP’s extensive experience of online learning, uses tools which can be applied by email or voice communication, and replaces the normal face-to-face analytical and consensus-building workshops with well-developed online workshop 

The Outcome

The approach worked well. The joint INASP-PEBL evaluation team identified and reviewed over 40 programme documents and 20 documents about other initiatives, interviewed 34 stakeholders, collected 10 stories of change and received 30 responses to an email survey of students. The validity and quality of the data were reviewed by contributors in four data validation workshops and the results were analysed by the INASP-PEBL evaluation team in two co-analysis workshops. The results of that were reviewed and the final recommendations co-produced in a final summit workshop including 25 stakeholders from across the partnership, who scored the quality and rigour of the evaluation as 4.5 out of 5, the accuracy of the results as 4.3 out of 5 and the usefulness of the recommendations as 4.4 out of 5.

PEBL has made substantial progress towards the planned outcome. All quantitative targets in the results framework have been exceeded substantially and this is supported by strong qualitative evidence. Though participants in the co-analysis workshop estimated about 90% achievement, indicating the quantitative indicators only tell part of the story. The programme has achieved or substantially exceeded them in three of its five outputs, and nearly met them in two.

The assumptions underpinning the programme were all correct and the theory of change worked broadly as expected. While the COVID-19 pandemic had undoubtedly accelerated the uptake of blended and online learning, contribution analysis indicated that project activities had contributed more than 50% to the overall outcome, and between 50% and 75% to delivery of the programme outputs. 

Full details of the results and the approach can be found in the final report.