Monitoring and Evaluation
Patient Engagement Monitoring and Evaluation Framework
Background/Rationale for the document:
Capturing the ‘return on engagement’ is complex, given the many factors that influence the impact of patient engagement. This Patient Engagement Monitoring and Evaluation Framework, with metrics, was created to help partnerships between patients and/or patient organisations, bio-pharmaceutical companies, regulators and health technology assessment (HTA) bodies to self-evaluate the progress and impacts of patient engagement in the medicines development lifecycle for all stakeholders involved.
Objective of the tool:
The objective of this tool is to support learning to facilitate meaningful patient engagement, helping users to understand the pathway to impact of patient engagement and demonstrate better decision-making in medicines development.
Summary of the content (overview of what readers will find in the tool), with links to main content:
The tool provides a map for monitoring and evaluation of patient engagement across multiple decision-points in medicines R&D, including 87 metrics organized across four key evaluation components:
1. Input metrics (n=13) show whether or not the conditions for meaningful and sustainable patient engagement are in place.
2. Activities/process metrics (n=16) show how the implementation of patient engagement is progressing and can elucidate areas for improvement.
3. Learning and change metrics (n=13) show the short-term, direct results of patient engagement which give an indication of the progress made towards impacts.
4. Impact metrics (n=45) show the long-term impacts for medicines development and stakeholders.
The context component allows users to understand what contextual factors (n=15) may
facilitate or inhibit success.
Each metric is accompanied by a description and possible methods for monitoring and evaluating its progress.
There is no ‘one size fits all’ set of metrics appropriate for every initiative or organisation. Therefore, this tool allows users to select metrics in order to develop a tailored set that aligns with their specific objectives and provides meaningful information in their context. Co-creation of a tailored set of metrics with all stakeholders involved supports the alignment of expectations and the development of a shared purpose for patient engagement.
We used participatory action research to develop and refine the Monitoring & Evaluation (M&E) framework.
A multi-stakeholder working group consisting of representatives of four European patient organisations, 15 biopharmaceutical companies and two academic institutions was created This working group was tasked with:
- Reviewing the literature on M&E of patient engagement;
- Development and testing of a M&E framework;
- Identification and selection of appropriate metrics for M&E;
- Clarification of terminology and language to be used.
The working group provided advice on the study design, feedback on documents and versions of the framework, co-analysed case study data, were involved in writing publications and in other dissemination activities. All working group members had an equal say in the framework development process.
The framework development process included three distinct phases: (1) design phase, (2) testing phase, (3) consensus and alignment phase. Each of these involved a different methodological approach described below.
Phase 1: Framework design
The aim of the design phase was to develop early versions of the M&E framework by identifying (1) impacts previously reported from patient engagement initiatives, (2) the conditions needed to achieve these impacts, (3) suggested metrics for M&E of patient engagement. A scoping literature review, three key informant interviews and six test case studies with PARADIGM partners were conducted. Informed by these primary analyses, researchers developed an early version of the framework. This draft framework was then reviewed by PARADIGM partners, adapted and subsequently entered the test phase.
Phase 2: Framework applicability test
The aim of the test phase was to apply the draft framework to real-world patient engagement initiatives in the context of medicines development. We used the M&E framework to validate the identified (sub)components and to select and test suggested metrics in practice. In total, 24 patient engagement initiatives were included as cases. Case study contributors were asked to describe their patient engagement initiative per component of the M&E framework and to select appropriate metrics. Reflection meetings were held between researchers and the case contributors to discuss the framework, metrics and their applicability. A ‘tailored’ M&E framework for each case was developed using an iterative approach. The research team co-analysed and integrated metrics from all cases. Furthermore, the results and applicability of the framework was reviewed by the working group and any changes that derived from the case studies were discussed during the consensus and alignment phase.
Phase 3: Framework consensus and alignment
The aim this phase was to build consensus on the framework and develop agreed ‘sets of metrics’ that align with objectives of patient engagement. An online consensus-building workshop was held with all working group members to develop these sets in multi-stakeholder groups. Next, the researchers identified possible measurement methods from the literature review and those employed in the case studies, which were mapped to the sets of metrics. Remote meetings were held with the multi-stakeholder working group to reach consensus on the final framework, including (sub)components, (sets of) metrics and measurement methods.
Lidewij Vat (VU-Athena Institute)
Callum Gunn (VU-Athena Institute)
Sevgi Fruytier (VU-Athena Institute)
Léa Darvey (VU-Athena Institute)
Teresa Finlay (University of Oxford)
|Laiba Husain (University of Oxford)
Jacqueline Broerse (VU-Athena Institute)
Paul Robinson (MSD)
Tjerk Jan Schuitmaker (VU-Athena Institute)
Mathieu Boudes (EPF)
Giorgio Barbareschi (EATG)
Ana Diaz (Alzheimer Europe)
Michaela Dinboeck (Novartis)
Lukas Eichmann (Novo Nordisk)
Elisa Ferrer (EURORDIS)
Claudia Hey (Merck)
Karina Huberman (EATG)
Maria José Vicente Edo (IACS)
Begonya Nafría (FSJD)
Mitch Herndon (UCB)
Chi Pakarinen (The Synergist)
Nicole Goedhart (VU-Athena Institute)
Nick Fahy (University of Oxford)
|All workshop and case study participants/contributors
Oana Bernard-Poenaru (Servier)
Hilde Piryns (Janssen)
Rob Camp (EURORDIS)
Nicholas Fahy (University of Oxford)
Robert Kroes (Lilly)
Bojan Cigan (EATG)
Daniel Lowman (Covance)
Melissa Herman (Lundbeck)
Marie-Laure Kurzinger (Sanofi)
Anne-Sophie Chalandon (Sanofi)
Geoff Cook (Novartis)
Laurence Maes (Janssen)
Tjerk Jan Schuitmaker
Download the Guidance