Multidimensional EdTech Impact Index
Towards standardised impact metrics and holistic measures in K12 EdTech
About this project
The project follows the Delphi study research design with two rounds of expert consultation and is built upon an extensive literature review conducted as three research projects, culminating in three published reports.
For years, EdTech impact has been overlooked, with minimal consideration for educational outcomes and the true potential of technology in children's learning. Current measurements often focus narrowly on experimental or teacher evaluation studies, failing to capture the multifaceted nature of technology's impact. Developing accurate models for impact measurement is crucial for a more comprehensive understanding of the added value of technology for children’s learning.
Our project aims to develop a holistic index for measuring impact as a combined set across multiple dimensions. The goal is for the index to be used as a tool for monitoring and evaluating impact by policymakers and funders, but also as a tool for researchers to systematically compare EdTech tools’ performance over time and to use it for capacity building within the EdTech industry.
Four key principles
Impact is multidimensional
There are five dimensions of impact that need to be all addressed if an EdTech is claiming positive impact on education. These are: efficacy, effectivemess, ethics, equity and environment.
Impact is in products, people and processes
Making an impact in EdTech requires not only effective products but also attention to the learning process and the people involved, particularly during the technology's implementation.
Impact matters at all stages
Attention to EdTech Impact measurement is essential at every stage of an EdTech growth, from product conception, through its implementation and scaling across various schools and districts.
Impact is is weighted and aggregatable
Balancing evidence weight ensures fair impact scoring and acknowledges varying evidential weights within each aspect, essential for establishing minimal standards and achieving higher impact levels.
Impact is in products, people and processes
Making an impact in EdTech requires not only effective products but also attention to the learning process and the people involved, particularly during the technology's implementation in learning environments (classrooms, homes).
Implement
Scale
IMPACT
STAGES
impact measures
Efficacy
Effectiveness
Equity
Ethics
Environment
impact DIMENSIONS
Kucirkova, N. (2024) Towards holistic impact measures in EdTech, Aliance Magazine, Available online from:
https://www.alliancemagazine.org/blog/towards-holistic-impact-measures-in-edtech/
The suitability of the technology during design, implementation and replication, especially as it scales, is crucial in assessing the overall impact of an EdTech solution.
Impact matters at all stages
IMPACT
STAGES
Scale
Implement
impact measures
Efficacy
Effectiveness
Equity
Ethics
Environment
Design
Quality Standards for Design and Use
impact DIMENSIONS
Impact is is weighted and aggregatable
Level 1: Good
Level 2: Desirable
Three levels based on weight of evidence (Gold, Silver, Bronze)
Level 3: Exceptional
International standards of impact can be categorized into three levels of rigour, contingent upon the strength of evidence.
Kucirkova, N., Campbell, J., Lindroos Cermakova, A. (2023). EdTech Impact Evaluation Frameworks: Summary 2023, Report for WiKIT AS, DOI:10.13140/RG.2.2.21563.59681
Elements to determine weight and aggregation
Plurality
in methods
- Both qualitative and quantitative methods of study are important to understand, produce and evaluate evidence
Methodological plurality
Pragmatism
in alignment of incentives
- The market incentives for EdTech industry need to align with those of investors and EdTech users and policy-makers
Coordinated Incentivization
Proximity
to local stakeholders
- Locally produced evidence of what works and doesn’t work is crucial for equitable approaches to evidence building
Local evidence
Participation
of all stakeholders
A robust framework proactively includes the voices of all stakeholders in the EdTech cycle, including children
Participatory approach
A multi-dimensional impact index
dimensions
indicators
measures
Total score
Efficacy
Product
people
processes
Three Stages
- Develop
- Implement
- Scale
- Plural
- Pragmatic
- Proximal
- Participatory
Effectiveness
Evidence of impact
Weight and rigour of evidence
Equity
Ethics
Environment
This project is coordinated by the International Centre for EdTech Impact (formely WiKIT) based in Stavanger, Norway and operating internationally.
The project research group is inviting input from researchers/academics, educational organisations, educators as well as business representatives on the emerging indicators for the Index. If you would like to offer feedback, please fill out this survey by 21. May 2024.
2024 All Rights Reserved