Data collection strategies
Surveys are a common evaluation tool. Surveys allow organizations to collect data from many individuals in a structured way. There are many examples of surveys that evaluate features of the engagement process and some even include basic impact measures such as the Public and Patient Engagement Evaluation Tool (PPEET).
Case Example: Evaluating Patient Partnering at HQO
Health Quality Ontario (now Ontario Health, Quality Division) has evaluated their patient engagement efforts since 2015 using a modified version of the Public and Patient Engagement Evaluation Tool (PPEET). The survey is used with all patient partnering activities. Measures from the survey have been rolled up to be indicators within HQO’s corporate scorecard.
Interviews can be used by organizations to gather the experiences and insights of individuals involved in patient engagement processes (e.g., patient partners, staff, organization leaders) and of those who are being informed by the outputs of patient engagement initiatives (e.g., decision makers). Interviews allow you to gain in-depth information, from the perspective of the participants, organizers and/or end users, about the perceived impacts of patient engagement work. Consider both individual and group interviews - some individuals may prefer the group approach.
Case Example: Interviews with patient and family advisors within the Nova Scotia Health Authority
The Nova Scotia Health Authority (NSHA) wanted to understand the impact that their patient advisors were having throughout their programs in the province. To do so, they completed a series of key informant interviews with the staff running these programs asking a simple set of questions: 1. What did you want to learn? 2. What did you do? 3. What did you learn? 4. What did you do with the feedback? 5. Who did you tell?
The ‘gold standard’ when it comes to evaluation evidence is to conduct a trial looking at how a project or activity unfolds with patient engagement and comparing this to how it unfolds without patient engagement or with different ‘doses’ of engagement. Trials are logistically complex and resource intensive, but can provide high-quality evidence about the effects of engaging patient, family and caregiver partners.
Case Example: Impact of patient engagement on health care improvement priorities
A cluster randomized trial was used in Quebec to understand the impact of patient engagement on the selection of health care improvement priorities by Health and Social Services Centres. Each centre was required to select priorities from a list of 37 indicators to be included in their accountability agreements. In all regions, baseline public priorities were determined. The centres were then randomized to the control or intervention groups. In the control groups experts determined the priorities with no knowledge of the public's perspectives. In the intervention groups, both public members and professionals worked together to select the priorities, and they received the results of the public consultation.
The two groups were compared to determine if there were differences in the priorities selected, and in the match between professional and public priorities. Public engagement in the process was found to increase the agreement between public and professional priorities by 41% and the public and professional members were able to find more common ground working together.
Boivin A, Lehoux P, Burgers J, Grol R. (2014). What Are the Key Ingredients for Effective Public Involvement in Health Care Improvement and Policy Decisions? A Randomized Trial Process Evaluation. The Milbank Quarterly, 92(2):319 – 350.