The value of patient and public involvement and engagement (PPIE) within the development and refinement of outcome measures is becoming increasingly recognized. The aim of this paper is to provide an overview of how PPIE was integrated within the development of a new measure designed for use in economic evaluations across health and social care, the EQ Health and Wellbeing (EQ-HWB™).
Four PPIE sessions were held at key stages. Discussions from each session and the outcome of any tasks were shared with the wider research team and used to help inform decision-making.
Results and discussion
PPIE covered several components of outcome measure development including; review of conceptual model; discussion on sub-domain inclusion; item refinement and reduction; pre-testing of items; selection of items for the measure; and design of the measure. Key learning points for future projects were highlighted including; consideration of practicalities, resources and logistics of PPIE activities; how sessions and activities are managed effectively; and how to managing expectations and communication from both researcher and PPIE perspectives.
The PPIE group provided invaluable insight into perspectives of future patients and carers. Their input was fed into a number of developmental stages. The formal involvement from the PPIE group meant that the voice of the general public was heard. This helped ensure the appropriateness of the design of the final measure.
The value of patient and public involvement and engagement (PPIE) within the development and refinement of outcome measures and preference-based measures is becoming increasingly recognized [1,2,3,4,5]. Guidance exists to help researchers plan and conduct meaningful PPIE [6,7,8,9]. There remains a paucity of documentation for this important component, and shared learning from such activities is often overlooked. The aim of this report is to provide an overview of how PPIE was integrated within the development of a new measure, the EQ Health and Wellbeing (EQ-HWB™) using an existing framework ; to describe the activities used to facilitate discussion and decision-making; and to reflect upon our experiences, using the GRIPP2-SF checklist .
The international Extending the QALY (E-QALY) project was initiated to develop a measure which captures aspects of quality of life or wellbeing for use in economic evaluations across health and social care. The project encompassed several stages (Fig. 1). Further details of the project are described elsewhere .
The PPIE group
PPIE members were recruited via several sources. We sought members of the public with a health condition (not specified), or informal carers of adults over 18 years. Firstly, an advertisement on the National Institute for Health Research (NIHR) INVOLVE website designed to advertise opportunities for public involvement in the National Health Service (NHS), public health and social care research (https://www.peopleinresearch.org/) recruited four PPIE members. Secondly, the lead PPIE Officer of the School of Health and Related Research (ScHARR), University of Sheffield approached individuals with previous PPIE experience (n = 2). Finally, one of the stakeholder groups of the EQALY project (National Institute for Clinical Excellence, NICE) approached existing PPIE representatives from within their own network (n = 1). The PPIE group membership consisted of seven members, who had experienced a range of health conditions, both personally and/or in a caring capacity (for a family member).
Four face-to-face PPIE sessions were held at key stages (Table 1).
The aim of Session 1 was to obtain feedback and input on the results of Stage 1. As this was the first time the PPIE group had convened it was felt important to include a ‘getting to know each other’ exercise to make members feel more comfortable and to encourage discussion and interactions. This was followed by a brief introduction to the overall aim of the project, including an explanation of the structure of the teams and governance groups; the role of the PPIE group, expectations; and what their involvement would be over the course of the project.
The main activity of the session was to reflect upon the findings of Stage 1 . High-level themes and potential subthemes for the new measure identified through the review were shown (and described) to the group. Members were asked to work in pairs to consider each of the themes/subthemes including whether they should be included or merged with other subthemes. Members were encouraged to include any comments to explain their decision or thought process. Each theme was considered for approximately 15 min, followed by feedback in a wider group discussion. Results were collated and distributed to PPIE members after the meeting for member checking [13, 14]. Members were encouraged to add any additional thoughts or comments having had further time to reflect on the task. Results were used to inform the final selection of themes and subthemes .
Members were updated on the project including which of the potential themes/subthemes raised in the Session 1 had been taken forward and why. The process for identifying potential items and criteria for judging the best item were introduced. The group then discussed the draft items from Stage 2 arranged into themes/sub-themes and considered these from the perspective of a respondent in a future study . They discussed issues ranging from ambiguous interpretation, translation, intrusiveness, social desirability and ease of completion. This discussion supported refinement and in some cases dropping of items. The research team’s choice to present items without the response choices (which had yet to be agreed) nor in a questionnaire format resulted in considerable confusion, which hindered the efficiency of this meeting. Results informed the item selection process for Stage 2.
Session 3 began with a presentation of the key findings of the face validity studies (Stage 3) conducted in the UK and internationally . Some examples were provided to show which items had been excluded for consideration, with reasons behind their exclusion. The aim of Session 3 was to obtain members’ views on potential item selection. This was facilitated by a practical exercise where each domain was considered in turn. Potential items were presented on coloured cards, with different colours used for each domain. The group were asked to discuss and allocate items to one of three categories; include, reject or undecided. Participants moved around the room, and placing the items on boards for each of the aforementioned categories. After each domain, members had the opportunity to reflect upon their decisions, and were given an opportunity to change the allocation of items. The final allocation of items were noted alongside discussions that were observed. The use of colours to reflect the different domains helped members to think about whether the proposed items adequately covered the overarching themes and the conceptual model for the measure. Results (Table 2) were used alongside other evidence in consultation regarding which items to include for the EQ-HWB .
The aim of the session was to inform the final layout and presentation of the measure. Much consideration was given to the instructions on how respondents complete it. Members were asked for views on the ordering of items within the questionnaire. Laminated cards were used to help members visualize what the questionnaire could look like. This was then produced in a mock format, after which further feedback was given on the layout. Members provided views on whether items should be alternately shaded (for reading ease), whether items should be numbered, and spacing between the items. They expressed that the layout should incorporate colour, and that the final format of the questionnaire should be of high-quality production, such that it would suggest it was an important thing for respondents to complete. Members had mixed views about the exact order of items although agreed that the opening item should be easy to complete, and final items should minimise the risk of leaving respondents feeling negative emotions. Members were keen to ensure that the title of the measure, and any invite to complete the measure, should clearly communicate what is being collected and why. The draft version was modified with members’ input then printed and shared at the meeting. In addition, they advocated for clear instructions on which perspective the respondent completing the measure should approach it from, i.e. carers complete it in relation to themselves, and not on behalf of the person they care for. Members also contributed the perspective of future respondents completing in English, where this is not their first language, and highlighted potential ambiguities and difficulties they may face with interpretation. This is a good example of the PPIE group reminding researchers to communicate in plain language. The feedback contributed to the layout of the experimental version of the EQ-HWB.
A list of activities and key inputs from the PPIE group are shown in Table 1. The write-up from each session was shared with the broader project team and formed part of the body of evidence that was used in decision making at each stage of the project. A summary of engagement across each of the development stages is shown in Table 3.
The role of PPIE in the Extending the QALY project was very valuable, both in terms of helping to develop the EQ-HWB, but also in providing experience for the research team in how to best integrate and conduct PPIE more broadly [16,17,18]. There are several aspects worth highlighting that we would advocate others to consider when planning PPIE for developing measures.
Practicalities, resources and logistics
Existing literature and guidance documents clearly state the importance of good planning for PPIE activities [4, 18, 19]. We found this to be challenging, particularly when considering the timing of PPIE meetings within the project stages when important outputs of the meeting were required. Practical considerations such as ensuring meeting rooms were fully accessible to people of all physical abilities, ease of access to other facilities (such as prayer rooms), location of the buildings and links to public transport were all important activities requiring time [19, 20]. It was also necessary to allocate enough resource to the design, preparation and circulation of both pre-reads and session materials [4, 21].
Managing the sessions and activities
We found it beneficial to have several members of the research team attend PPIE meetings. This supported smooth running of the sessions as well as administrative logistics. Delays in reimbursing out of pocket expenses can compromise working relationships . Having several members of the research team present allowed for us to engage with the PPIE members during the breaks. This helped foster good relationships within the group, and facilitated an environment where members felt valued and comfortable to express their views openly [20, 23]. Having a number of researchers present allowed for a change in personnel to lead aspects both within and between sessions. This proved beneficial when explaining terminology and/or the tasks.
We noted the sessions became more productive and fruitful as the project progressed. During the early sessions the concepts and visualization of the “end product” was challenging for the PPIE members. There were lots of discussion of issues that, whilst very important (often relating to implementation of measures within the real-world setting), were outside of the remit of the E-QALY project. Frustrations were felt from both PPIE members and researchers, and there was a sense of ‘lack of purpose’, particularly during Sessions 1 and 2. We noted the PPIE members became very absorbed when presented with a “task” versus a more general discussion. This was particularly apparent during Sessions 3 and 4. Here members could start to see the questionnaire beginning to take shape and felt that they could see the merit of their involvement within the development process. We would strongly advocate researchers to think of ways to integrate and incorporate task-based exercises in order to facilitate discussions . We found this to be very useful, particularly when demonstrating the impact of decision-making on the length (i.e. number of items) of the proposed measure.
Managing expectations and communication
The sessions themselves are very demanding. From a researcher perspective, it would have been preferable for the meetings to last longer. The meetings felt very concentrated and intense with many issues to cover. Each meeting started with a recap and debrief as to what stage the overall project was at. Whilst it was tempting to rush this component, trying to do so proved counterproductive. It resulted in the group seeking additional clarification later in the session. It was very challenging to cover everything within the scheduled time that was available. It was important that sessions finished on time due to work, travel and/or caring commitments. Comfort breaks were included within each session . Each one of the sessions ended with a feeling that more could have been achieved.
When aspects of the overall project became delayed, this resulted in periods where limited communication with the PPIE members occurred. We were fortunate that our inactions did not lead to member disengagement, however this is certainly a possibility when poor communication can result in members feeling undervalued or their input viewed as ‘tokenistic’. We endeavoured to inform PPIE members about the stages (and results) of the overall project, and when we anticipated that we would wish to reconvene with them. However, PPIE members noted a concern with lack of regular contact.
Another aspect of communication is that of feedback. Whilst we did ask for feedback following each session, this was relatively informal. It was useful to consider areas that worked well as well as to identify issues that could be improved for subsequent sessions. Location of the meetings (i.e. closer to public transport links) was one example that led to a positive change for subsequent sessions. Researchers need to learn about the most efficient way of incorporating PPIE when developing measures. The opportunity to seek more formal evaluation of the PPIE process was not maximized during the project, but should be encouraged.
One of the roles of PPIE is to formally incorporate lived experiences to research. In this project our PPIE group membership was small, and it possible that a larger group with wider representation across different health conditions may have resulted in different decisions, particularly in Session 3. It is plausible a person who has not experienced difficulties with specific items (or concepts) may not rank its importance as highly as more personally relevant items. Furthermore, despite the introduction and discussion of scope of the project being outlined in Session 1, it is possible that members brought their own interpretations of health and wellbeing that did not align with domains/themes provided in the EQ-HWB. Whilst this is not a limitation per se, it is feasible that individual’s definitions of health and wellbeing may have affected the relative importance of any items (Session 3). There is a balance to be struck between ensuring a suitable number of PPIE members required to ensure breadth of experiences, and practicalities of group sessions.
Throughout the development of the EQ-HWB measure, the PPIE group provided invaluable insight into perspectives of future patients and carers. Their input fed into a number of developmental stages, helping to ensure the appropriateness of the design and content of the final measure.
Wiering B, de Boer D, Delnoij D (2017) Patient involvement in the development of patient-reported outcome measures: the developers’ perspective. BMC Health Serv Res 17(1):635. https://doi.org/10.1186/s12913-017-2582-8
Grundy A, Keetharuth AD, Barber R, Carlton J, Connell J, Taylor Buck E, Barkham M, Ricketts T, Robotham D, Rose D, Kay J, Hanlon R, Brazier J (2019) Patient and public involvement in health outcomes research: Lessons learnt from the development of the recovering quality of life (ReQoL) measures. Health Qual Life Outcomes 17(1):60. https://doi.org/10.1186/s12955-019-1123-z
Feldman D, Kruger P, Delbecque L, Duenas A, Bernard-Poenaru O, Wollenschneider S et al (2021) Co-creation of practical “how-to guides” for patient engagement in key phases of medicines development-from theory to implementation. Res Involv Engagem 7(1):57. https://doi.org/10.1186/s40900-021-00294-x
Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S et al (2017) GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ 358:j3453. https://doi.org/10.1136/bmj.j3453
Brazier JE, Peasgood T, Mukuria C, Marten O, Kreimeier S, Luo N, et al (2022) The EQ Health and Wellbeing: overview of the development of a measure of health and wellbeing and key results. Value Health 25(4):482–491. https://doi.org/10.1016/j.jval.2022.01.009
Mukuria C, Connell J, Carlton J, Peasgood T, Scope A, Clowes M et al (2022) Qualitative review on domains of quality of life important for patients, social care users and informal carers to inform the development of the EQ-HWB. Value Health 25(4):492–511. https://doi.org/10.1016/j.jval.2021.11.1371
Lincoln YS, Guba EG (1985) Naturalistic inquiry. Sage, Newbury Park
Tong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 19:349–357. https://doi.org/10.1093/intqhc/mzm042
Carlton J, Peasgood T, Mukuria C, Connell J, Brazier J, Ludwig K, Marten O, et al (2022) Generation, selection and face validation of items for a new generic measure of quality of life: The EQ Health and Wellbeing. Value Health 25(4):512-524. https://doi.org/10.1016/j.jval.2021.12.007
Al-Janabi H, Coles J, Copping J, Dhanji N, McLoughlin C, Murphy J et al (2021) Patient and Public Involvement (PPI) in Health Economics Methodology Research: reflections and recommendations. Patient 14(4):421–427. https://doi.org/10.1007/s40271-020-00445-4
Boylan AM, Locock L, Thomson R, Staniszewska S (2019) “About sixty per cent I want to do it”: Health researchers’ attitudes to, and experiences of, patient and public involvement (PPI): a qualitative interview study. Health Expect 22(4):721–730. https://doi.org/10.1111/hex.12883
Shippee ND, Domecq Garces JP, Prutsky Lopez GJ, Wang Z, Elraiyah TA, Nabhan M et al (2015) Patient and service user engagement in research: a systematic review and synthesized framework. Health Expect 18(5):1151–1166. https://doi.org/10.1111/hex.12090
We would like to thank NICE for highlighting the methodological research need to the MRC that resulted in the funding call entitled "Beyond the QALY" which led to this research being funded. We would like to acknowledge the support of the National Institute for Health Research Yorkshire and Humber Applied Research Collaboration (formerly CLAHRC) and the National Institute for Health Research Clinical Research Network (NIHR CRN). We acknowledge the invaluable contributions of members of the project steering group, advisory group and public and patient involvement and engagement groups. We would also like to thank members of the EuroQol Group Association for their input at plenary and academy meetings. Finally, we would like to acknowledge the contribution of all the patients, social care users and informal carers who took part in all the studies across the different countries. The views expressed in this manuscript are those of the authors and not necessarily of our funders, the National Institute of Health and Care Excellence, the Department of Health and Social Care or those acknowledged. We would like to thank all the members of the EQ-HWB Patient and Public Involvement and Engagement (PPIE) group for their support and enthusiasm throughout the project.
This study is independent research funded by the UK Medical Research Council (Grant number 170620), and the EuroQol Research Foundation.
Authors and Affiliations
School of Health and Related Research (ScHARR), University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK
Jill Carlton, Clara Mukuria & Julie Johnson
Melbourne School of Population and Global Health, University of Melbourne, Melbourne, Australia
JC was a member of the research project team, was one of the researchers facilitating the sessions, and wrote the initial draft of the manuscript. TP, CM and JJ were members of the research team, were one of the researchers facilitating the sessions, and were major contributors in writing the manuscript. MO and WT were members of the PPIE group, and have contributed to the content of the manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Carlton, J., Peasgood, T., Mukuria, C. et al. The role of patient and public involvement and engagement (PPIE) within the development of the EQ Health and Wellbeing (EQ-HWB).
J Patient Rep Outcomes6, 35 (2022). https://doi.org/10.1186/s41687-022-00437-y