- Open Access
Using co-production to implement patient reported outcome measures in third sector organisations: a mixed methods study
Journal of Patient-Reported Outcomes volume 6, Article number: 78 (2022)
Third sector organisations such as charities and community groups are using Patient Reported Outcome Measures (PROMs) at an aggregated service level to demonstrate their impact to commissioners to generate or retain funding. Despite this motivation, organisations can struggle with implementing PROMs. Previous studies have identified facilitators including organisations using an appropriate measure, co-producing the PROMs process with staff, and investing resources to support the use of measures. However, to date no studies have applied this learning to third sector organisations to evaluate whether taking an evidence-informed implementation approach improves the use of PROMs.
A Community-Based Participatory Research approach was used which involved university-based researchers supporting two third sector organisations to implement PROMs. The researchers provided evidence-informed advice and training. The organisations were responsible for implementing PROMs. The researchers evaluated implementation through a mixed methods approach including five key informant interviews, four evaluation groups and analysis of collected PROMs data (n = 313).
Both third sector organisations faced considerable constraints in incorporating known facilitators and addressing barriers. The organisations involved staff in choosing an acceptable measure. However, competing priorities including external pressures to use specific PROMs, busy workloads and staff opinions created challenges to using measures. Investment of time and energy into developing an outcomes-based organisational culture was key to enable the prioritisation of PROMs. For example, discussing PROMs in supervision so that they were viewed as part of people’s job roles. Organisations found that implementation took several years and was disrupted by other pressures.
Whilst organisations were motivated to implement PROMs to obtain or retain funding, they faced considerable practical and ideological challenges. Consequently, some stakeholders felt that alternative methods to measuring impact could potentially be more feasible than PROMs.
Plain English summary
Organisations may use Patient Reported Outcome Measures (PROMs). Service-users complete a set of questions about their health before and after having help to see if their health has improved. However, staff find it hard to use PROMs. To solve this, researchers worked with two charities to help them use PROMs. Together, we chose a PROM and trained staff. We found that managers had to spend a lot of time trying to get PROMs used. This took several years and cost money. Staff needed a lot of support to use PROMs. Given the challenges, organisations may find it easier to use other types of methods to test if their service works.
Organisations delivering health and wellbeing services often use Patient Reported Outcome Measures (PROMs) to measure their impact on people’s health and wellbeing . PROMs are standardised questionnaires that enable a person to rate their health, wellbeing or symptoms . Measures include the Warwick-Edinburgh Mental Wellbeing Scale (WEMWBS)  EQ-5D-5L  and the Adult Social Care Outcomes Toolkit (ASCOT) . Changes in someone’s health can be measured by completing the PROM at more than one time point, for example before and after attending a service. Organisations may aggregate the results from a number of service-users to evaluate the impact of a service . For example, NHS England has recommended the use of PROMs in social prescribing programmes . In the United States of America, the PROMIS programme is developing item-banks of measures for organisations to use .
Despite being motivated to use PROMs to demonstrate impact and secure funding , organisations have often struggled with implementation, characterised by low completion rates of measures . Implementation is the process between an organisation deciding to adopt PROMs and their use within the organisation . A number of contextual and process related factors impacting on the implementation of PROMs have been identified . They may be a facilitator or barrier depending on their execution . Identified contextual factors include organisations having to implement PROMs because external commissioners demand their use in return for funding . Internally, organisations need a strategic commitment to using PROMs and resource the collection and analysis of measures e.g. investing in data management systems [13,14,15]. Having an Implementation Lead responsible for getting measures used  and consulting and training front-line workers appears beneficial [17, 18].
One type of organisation facing challenges implementing PROMs are Third Sector Organisations (TSOs). TSOs, also known as charities and community groups, are increasingly commissioned to deliver health and wellbeing services . For example, within the United Kingdom, TSOs are viewed as a key provider within health care . TSOs deliver a range of services including social groups for people experiencing bereavement, healthy eating courses, social prescribing and mental wellbeing programmes. TSOs are non-profit organisations which exist to bring change to the lives of their targeted population and are separate to the state [20, 21]. TSOs are often commissioned or funded on short-term contracts by government or grant giving agencies and are required to demonstrate their impact through PROMs to justify their funding . Despite this motivation, TSOs, like clinical services with a long history of using PROMs, can struggle with implementation . TSOs report that having to use inappropriate measures mandated by commissioners is a key barrier . Consequently, the use of PROMs may be improved by organisations choosing their measure and taking an evidence-informed approach to implementation. This approach has been successful in clinical services including oncology  and rehabilitation . The aim of the study was to evaluate the experiences of two TSOs taking an evidence-informed PROMs implementation approach.
Community-based participatory research
We used Community-Based Participatory Research (CBPR), which involved university-based researchers supporting two TSOs (Organisation A and B) with both implementing PROMs and evaluating their experiences. We used CBPR because taking a collaborative approach can facilitate the implementation of PROMs . CBPR is a widely used type of participatory action research , where researchers and stakeholders are partners in the research process, co-producing knowledge together that is beneficial for all [26, 27]. The design of CBPR projects, role of different partners and levels of participation between researchers and other stakeholders varies between studies but there is the underlying principle of partnership and having an impact .
We used an action research spiral model , which involved the organisations making decisions on, and undertaking the implementation of PROMs. The lead author (AF) supported this process by sharing their knowledge on implementing PROMs and providing training to front-line workers on collecting measures. Alongside, researchers evaluated the organisations’ experiences of implementation, feeding back learning so that TSOs could improve the process (Fig. 1). The TSO leaders felt that the researchers were best placed to undertake the evaluation because they had the time and technical ability.
The collaboration between the researchers and the TSOs meant that together we were co-constructing the implementation process and learning . This contrasts with other study designs, where stakeholders are solely participants . These dual roles can create tensions including decisions on the PROMs implementation process and prioritising research activities. To manage these tensions, the lead author recorded reflections in her diary and had regular meetings with TSO stakeholders and an experienced participatory researcher (JH).
Description of the TSOs involved
We involved two organisations that were at different stages of their implementation journey, enabling us to learn more about the process. Both organisations delivered a range of support including employment and advocacy services in neighbourhoods experiencing socioeconomic deprivation . The TSOs were relatively large, each having an income of over £1,000,000 annually [32, 33] generated from a mixture of commissioned contracts with statutory agencies, funding from grant-giving organisations and income generating activities. Approximately 3000 service-users were supported in each organisation per year. Each TSO had a health and wellbeing team consisting of approximately 10 front-line workers who delivered a range of support including individual lifestyle coaching, social prescribing and men’s groups . The intention was for PROMs to be utilised by all workers in these teams.
The inclusion of the organisations was opportunistic as the researchers had prior relationships with the TSOs and knew they were interested in improving the use of PROMs in their organisations. Both TSOs had struggled to use PROMs mandated by commissioners, partly because the organisations considered the previous measures unsuitable. This was because they felt the questions were difficult to understand or did not consider relevant constructs. The organisations needed to implement PROMs to satisfy their commissioners and hoped that the study would help them to use a measure within their organisation that was acceptable to staff.
The two TSOs were involved differently within the study because they were at different stages of implementation. This study ran formally between June 2018 and January 2020 (19 months). Organisation A became involved because they were at the start of their PROMs’ implementation process. During the study, the focus was on choosing a measure, deciding how to implement the PROM and attempting to utilise it. In contrast, we had been working with Organisation B since 2016 to choose a measure. During this study, the organisation sought to use the PROM routinely within their services.
University-based researchers supported the two TSOs with implementation . The work was led by AF, who was undertaking the study as part of their PhD. Two supervisors and a post-graduate student supported the study. The researchers shared learning from previous studies with the organisations, who then made their own decisions on designing how to collect, process and use PROMs data. In both organisations, a manager acted as the Implementation Lead, they were responsible for making decisions and progressing implementation.
The researchers met regularly with each Implementation Lead during the study to discuss implementation issues including which measures to use, designing a PROMs process and to identify solutions to arising issues. Alongside giving evidence-informed advice, the researchers provided additional capacity including writing guidelines for collecting measures and delivering training on using PROMs data to front-line workers. After designing the PROMs process, the two TSOs planned to trial, then embed, the PROMs by using the chosen measures throughout their wellbeing services.
We used a mixed methods research design to evaluate the implementation of PROMs. We analysed collected PROMs data to explore completion rates and variation between staff members. We undertook several qualitative approaches including key informant interviews, evaluation meetings and analysed a researcher’s reflective diary to explore facilitators and barriers to implementation.
Qualitative methods and analysis
We held evaluation meetings in each organisation. A researcher (AF) facilitated the meetings. They were attended by front-line workers, managers and Implementation Leads. The groups discussed implementation issues and potential solutions, which were then acted on by the organisations. At the beginning of the meetings, the researcher explained the study and informed attendees that notes would be taken of arising issues. People were able to leave the meeting if they did not want to be part of the study, this opt out consent process is commonly used within participatory research  and received ethical approval.
Three events were held in Organisation A, two were in Months 4 and 6 to discuss the choice of PROM. The third event was in Month 18 to reflect on progress. In Organisation B, one event was held in Month 16 to review the implementation process. Fewer events were held in Organisation B because they were at a later stage of their implementation journey. We used visual methods including brainstorming to capture the discussions, these were recorded within the researchers’ (AF’s) reflective diary .
Key informant interviews
Key informant interviews were undertaken in Months 15–18 to gain the insights of people pivotal to implementation . The sample size was small because the interviews were supplementing other data sources. AF undertook the interviews, using a topic guide developed for each interviewee. Written consent was taken and each interview was audio-recorded . These were transcribed verbatim, anonymised and uploaded to NVivo Version 12.
As the lead researcher, AF kept a diary throughout the study to reflect on the implementation process (similar to the use of field notes in ethnography) . After any contact with stakeholders, the researcher recorded notes about the encounters along with reflections. The diary was anonymised and uploaded to NVivo Version 12 for analysis, using the same process as for interview transcripts.
Qualitative data analysis
AF, supported by the other researchers, undertook thematic analysis to analyse the qualitative data collected through the methods described above . Analysis entailed:
Familiarisation We read the transcripts and the reflective diary several times.
Coding We grouped findings by commonality.
Searching for themes We identified connections between the codes to develop themes.
Reviewing the themes We checked whether the themes reflected the data.
Defining and revising the themes We developed the overarching message of each theme.
Analysis of routinely collected data
We used the collected PROMs data to analyse how extensively the measures were used in each organisation. Although our intention was to include both organisations, we could only analyse data for Organisation B because Organisation A had not reached a stage of using the measures extensively nor uploaded them into their data management system. Organisation B provided an anonymised dataset of PROMs scores. This consisted of a service-user reference number and information on each measure completed including the date of administration, scores for each question and name of the staff member who administered the PROM. The data was collected between January 2019 and February 2020. Organisation B did not link demographic or service use data to the PROMs data, so this was not available for analysis. AF converted the file into SPSS, cleaned the dataset and conducted descriptive analysis to explore how completion rates differed between specific front-line workers and over time. We hypothesised that completion rates would increase as PROMs became embedded within the organisation.
Integrating the findings
A ‘following the thread’ technique was used to synthesise the findings from the evaluation groups, key informant interviews, reflective diary and analysis of PROMs data . This entailed identifying a finding in one data source and exploring how it transpired in the other data sources. For example, the quantitative data indicated that PROMs collection decreased over time and we used the qualitative data to identify reasons for the reduction. The findings below are reported in themes that draw on both the qualitative and quantitative data.
Approval was granted by the University of Sheffield (Application: 020700).
In Organisation A, a mixture of managers and front-line workers attended the meetings. Twelve people attended the first two meetings and 7 attended the final meeting. In Organisation B, 8 front-line workers attended the single meeting. The different make-up of groups reflected the preferences of the individual TSOs. Events lasted between 90 and 120 min and were held on the organisations' premises.
Key Informant Interviews
Five participants were interviewed, three worked at Organisation B (Implementation Lead, Senior Manager and a front-line worker) and two were from Organisation A (Implementation Lead and a front-line worker). The interviews generally lasted about 60 min, with three taking place at the organisations’ premises and two at the university.
Analysis of routinely collected PROMs data
The dataset from Organisation B consisted of 313 service-users who had completed a PROM at the start of attending a wellbeing activity, 62 of these completed a second PROMs after receiving support (19.8%).
Overview of themes
Both TSOs designed a bespoke PROM because stakeholders felt that this would help workers to use the measure. However, organisations faced significant challenges including competing priorities and having to develop an outcomes-based organisational culture. Four key themes were: (1) Rationale for utilising PROMs. (2) Co-producing an appropriate bespoke measure. (3) Competing priorities and (4) Embedding PROMs into an organisation.
Rationale for utilising PROMs
Both TSOs were implementing PROMs because there was an expectation from commissioners that the organisations needed to demonstrate outcomes to receive funding. The TSOs viewed PROMs as something they had to use on an aggregate level to demonstrate the impact of the service. A secondary purpose was front-line workers using the tool with service-users to reflect on their needs and progress. Previously, commissioners had mandated specific measures that the TSOs considered inappropriate for use within their organisation. In response, the organisations hoped that choosing a PROM themselves would result in staff being more engaged and commissioners no longer imposing their choice of measure.
So the experience of [a previous PROM] was so negative that actually, you’re not actually getting very good quality stuff because you’re just thinking, well you’re filling it in for the sake of it. [Implementation Lead A]
There were differences in how the two TSOs embraced PROMs. Organisation B proactively committed to becoming an outcomes-focused organisation that used PROMs. In contrast, the Implementation Lead in Organisation A did not feel that PROMs were a valid way to measure impact but felt obliged to use measures in response to the commissioning environment.
It’s not relevant and it’s not appropriate to ask customers but I do feel quite confident that the staff are embedded in the community and have got the links and the relationships with the people participating. [Implementation Lead A]
Co-producing an appropriate bespoke measure
Both TSOs involved front-line workers in designing an organisation-specific bespoke measure. Neither set out to design a measure but felt existing measures did not meet their needs. This was because the front-line workers and Implementation Leads wanted a PROM which was short, used clear language and included graphics to help service-users with understanding the text. The organisations felt this was more of a priority than using a measure which had undergone psychometric testing. The organisations involved front-line workers in choosing a PROM because they hoped this would motivate them to use it. We hypothesised that the bespoke PROMs would be used by front-line workers because they considered it suitable for their service-users.
Spoke to front-line workers at Organisation B and they feel the design of the PROM was much better than what they had to use previously and reported that service-users find the graphics useful. [Extract from reflective diary]
The PROM tool that we developed, I’m really proud of that because it was developed using staff, yourselves and all that kind of stuff, so it was something that was co-produced, so it had meaning to all those stakeholder groups. [Senior Manager B]
Organisation B’s measure combined questions from several validated measures including the Short-version of WEMWBS and the ASCOT. However, Organisation B developed one question themselves, asking service-users ‘How are you feeling’ (Additional file 2). They added this as an introductory question to orientate service-users. In contrast, Organisation A predominantly wrote their own questions because they did not feel that questions from existing PROMs were suitable for their service-users. The questions focused on service-users’ ability to cope with problems and development of support networks. For example, How well do you cope with managing problems? (Additional file 3). The measures did not undergo psychometric testing as the aim of the study was to understand whether the PROMs would be used. Despite having acceptable measures, the organisations faced challenges implementing them, these barriers are discussed below.
The organisations had to take a pragmatic approach to implementing PROMs because their staff were juggling multiple priorities including completing PROMs mandated by commissioners and busy workloads. However, prioritisation also appeared to be influenced by an individual’s personal opinions, with people who valued measures seemingly more likely to find time to progress implementation than people who felt that PROMs were inappropriate for use in TSOs.
Having met with [Name of front-line worker], I am really concerned about how we will get them using the PROM. They appear resistant to any changes being enforced. [Extract from reflective diary]
Prioritising commissioner mandated PROMs
The organisations faced an ongoing process of implementing different PROMs at the request of commissioners. These PROMs had to be prioritised over the organisation’ measures as funding was conditional on their use. TSOs had hoped that commissioners would accept their chosen measure so that they could take one organisational approach to collecting PROMs, but this did not happen. Implementation Lead B reflected that they may need to amend their bespoke measure to avoid front-line workers having to collect multiple PROMs.
We’ve been told we are to report these outcome measures […].so it could be rather than them [front-line workers] doing two questionnaires we might then make the decision to amend it. [Implementation Lead B]
Implementation Leads relying on support from researchers
The Implementation Leads faced multiple, time critical workload responsibilities including grant applications, managing staff and overseeing service delivery. Consequently, the Implementation Leads valued having support from researchers who had the time and skills to dedicate to implementation. Researchers provided a range of support including delivering training to front-line workers and designing methods of displaying the collected data. The organisations felt that this input was essential.
Not a chance we would have done that. We could have started it off, but we would never have finished it […] we can’t do it on our own. Really so undoubtedly and crucially we wouldn’t be able to do it without the university. [Implementation Lead B]
Front-line workers prioritising support for service-users
In both organisations, some front-line workers reported finding it difficult to complete PROM related tasks because they prioritised spending time supporting service-users. For example, people felt that they did not have time to input collected measures into data management systems and wanted administrative support for this task. This concern may have partly arisen because of the PROMs primarily being used to demonstrate service impact rather than as a progress tool with individual service-users. To address the time pressure, Implementation Leads made changes to the front-line workers’ roles so that they would have time to spend on PROMs related tasks.
I think that message is getting through that we need, what we need to do is quality and we need to be able to show and prove that its good and so and giving them [front-line workers] permission to take on less volume [of service-users] and giving them permission to have time in the office to do the paperwork and that, that is valid. [Implementation Lead A]
Having a flexible PROMs process to enable the prioritisation of service-user needs
The organisations designed a flexible PROMs data collection process because they wanted to prioritise the needs of service-users. Examples include being able to administer the PROMs in person or over the telephone and administering measures when it felt right for the service-user rather than at specific timepoints. However, front-line workers felt there was almost too much flexibility because they were not always sure when to utilise measures. This was detrimental for data quality.
So, it was retrospective because we weren’t using it when the group set up in January and I tried to get them to think where they were when they first came, where they are now and hopefully where they’re gonna be, well let’s just say just before Christmas, if they want to do it again. [Front-line worker A]
Embedding PROMs into an organisation
The TSOs invested time into developing their culture including supporting front-line workers so that PROMs were embedded into the norms and practices of the organisation. However, the organisations struggled to sustain the use of their bespoke measure indicating that TSOs faced considerable implementation challenges especially when a PROM was not mandated by commissioners.
Developing organisational culture to be receptive to PROMs
Stakeholders found that they needed to invest time in developing their organisational culture to be compatible with outcome measurement. In Organisation B, the Implementation Lead and senior managers invested time in evolving the norms, values, and expectations of the organisation so that PROMs were viewed as part of their core practice. They felt that it would take several years. In Organisation A, the Implementation Lead felt that PROMs were not necessarily compatible with their community-led organisational culture and thus was less inclined to make changes.
At this stage it’s about creating a culture and creating infrastructure within the organisation. [Implementation Lead B]
Having a trial period
Both TSOs had a trial period, where they tested their bespoke measures on service-users to ensure they were acceptable. In Organisation B, this was a formalised process, they piloted the measure over a few months with service-users from different wellbeing services. In Organisation A, they tried the measure in one group activity, on one occasion. Interviewees from both organisations felt having a trial was useful as reassured front-line workers that service-users felt the measure was appropriate.
Email exchange with [Name of front-line worker] about how the trialling period went. They had used it in the [name of wellbeing activity]. People had found it easy and straightforward. [Name of front-line workers] feels it would be suitable to use with service-users. [Extract from reflective diary]
Challenges sustaining the use of PROMs with front-line workers needing ongoing support
Despite the organisations designing their own measure that was acceptable to them and investing time in implementation, the quantitative data indicated issues with sustaining the use of PROMs. In Organisation B, rates of collecting follow-up measures as a proportion of baseline measures per front-line worker ranged between 8.8 and 41.9% (Mean: 19.8%). Furthermore, the collection of PROMs decreased over time (Fig. 2). This was surprising as the expectation was that the use of measures would increase as staff became more experienced at collecting PROMs. Some front-line workers discussed that whilst the bespoke measure was an improvement on previous measures, they still believed that using PROMs was not compatible with the relationships they had built with service-users. Other front-line workers reported forgetting to administer the measures. Reflecting on this, Implementation Lead B felt that front-line workers needed greater ongoing support to use PROMs. For example, reminders within supervision to iterate that collecting PROMs was part of people’s job.
Just to see, it is still here, this is the usage, we need to keep on using it, and just to encourage the senior workers to embed that as part of their supervision, cos I actually put it on the supervision form. [Implementation Lead B]
The TSOs invested significant time and resources into implementation which included designing their own measures. Despite aspirations to utilise their choice of measure, the TSOs had to prioritise PROMs mandated by commissioners. Furthermore, beliefs, culture and conflicting priorities interacted to create significant barriers to collecting measures, raising questions about the feasibility of using PROMs in some organisations. Despite the increasing evidence-base on best practice when implementing PROMs, the organisations struggled to address barriers in the short time frame of this study.
Strengths and limitations
The study had three strengths. (1) Through being involved with the organisations for over a year, we were able to observe the interplay of different facilitators and barriers. (2) Taking a mixed methods approach enabled us to observe how PROMs collection rates reduced and explore reasons for this. (3) Utilising CBPR ensured a reciprocal approach, with the TSOs receiving support from the researchers. There were four limitations. (1) The study would have benefited from the TSOs being more involved in the evaluation, for example the Implementation Leads keeping reflective diaries. (2) We were not able to undertake quantitative analysis of collected PROMs in Organisation A because they had not collected a sufficient number of measures. However, this was a finding in itself. (3) Sometimes organisations made decisions which did not align with the PROMs’ implementation evidence-base and the researchers could have been more persuasive whilst still adhering to participatory principles. (4) Implementation took longer than planned and thus the study finished whilst the organisations were still on their implementation journey. Consequently, we were unable to explore the organisations’ experiences later on in their implementation journey. For example, it would have been valuable to have re-interviewed senior managers about the decreasing PROMs completion rates.
Comparison with existing literature
We compared our findings with a range of existing studies on implementing PROMs. We build upon previous research by identifying how using an appropriate measure may be less of an influencing factor than external pressures, input from Implementation Leads and the intersection of organisational priorities and culture. Whilst our research focused specifically on TSOs, our findings reflect factors identified in other settings.
Our finding that external commissioners had considerable influence over the use of PROMs because of TSO’s receiving short-term funding reflects experiences in both primary care  and mental health services . Both TSOs chose to design a bespoke measure rather than use a validated one, reflecting concerns raised by others that there are issues with using existing validated PROMs within the third sector. Foster et al. (2020) found that the use of bespoke measures was common practice within the sector , which raises concerns about the psychometric validity of the measures being used. Researchers have proposed developing a specific third sector PROM that is psychometrically tested . However an item bank format, which can be adapted to the specific context may be more acceptable .
It appeared that using an acceptable measure was not the only barrier to overcome and that other challenges were more influential. Previous studies have focused on whether an organisation’s culture is amenable to PROMs [14, 48]. Our research identified that organisational culture is not static, if leaders invest time and resources, the culture can evolve to become more outcomes-focused. However, Implementation Leads and managers need to buy into the concept of outcome measurement to enable this. We identified that implementation takes a considerable period of time, which reflects a previous study where it took three years to implement PROMs . In our study, the use of PROMs decreased over time, indicating that ongoing support is needed to sustain their use, which reflects other studies . Our experience indicates that TSOs and commissioners may want to explore alternative methods to PROMs [22, 23].
We found many similar facilitators and barriers to other studies, indicating that organisations can learn from each other irrespective of their specific focus. Similarities included the importance of having buy in from stakeholders, the influence of organisational culture, the need for ongoing training and the amount of time and resources implementation takes. Organisations may not have full control over their PROMs processes because they must respond to requirements from commissioners. Consequently, organisations may have to implement different PROMs and change their processes to meet external needs, making implementation an ongoing rather than finite process. Developing a third sector specific measure which takes an ‘item bank’ format may overcome the challenges of both acceptability and psychometric validity. Organisations need to be prepared to invest considerable time (often years) and resources into progressing implementation of PROMs including making outcome measurement part of the culture and providing ongoing support to sustain their use. However, even with these changes it is possible that organisations may not have the resources to address some implementation barriers such as Implementation Leads having competing priorities. A solution may be to look for alternatives to PROMs.
Despite investing considerable time and resources into implementation, and using a bespoke PROM that staff valued, two TSOs struggled to implement measures within their practice, partly because not all stakeholders valued PROMs. This raises questions about whether the issue with PROMs is less about specific measures and more about the feasibility of using them in this context.
Availability of data and materials
Due to the nature of the study, the data and material relating to the study is not publicly available.
Initials of Lead Researcher (Alexis Foster-Lead Author)
Adult Social Care Outcomes Toolkit
Community-based Participatory Research
Consolidated Framework for Implementation Research
Initial of Janet Harris, participatory research expert and co-author
Office for National Statistics 4
Patient Reported Outcome Measures
Patient-reported Outcomes Measurement Information System
Statistical Package for the Social Sciences
Short Warwick-Edinburgh Mental Wellbeing Scale
Third Sector Organisations
Warwick-Edinburgh Mental Wellbeing Scale
Devlin N, Appleby J (2010) Getting the most out of PROMs—putting health outcomes at the heart of NHS decision-making: the King's Fund. https://www.kingsfund.org.uk/sites/default/files/Getting-the-most-out-of-PROMs-Nancy-Devlin-John-Appleby-Kings-Fund-March-2010.pdf. Accessed 15 Mar 2018
Greenhalgh J, Pawson R, Wright J, Black N, Valderas J, Meads D et al (2014) Functionality and feedback: a protocol for a realist synthesis of the collation, interpretation and utilisation of PROMs data to improve patient care. BMJ Open 4(7):e005601. https://doi.org/10.1136/bmjopen-2014-005601
Tennant R, Hiller L, Fishwick R, Platt S, Joseph S, Weich S et al (2007) The Warwick-Edinburgh Mental Well-being Scale (WEMWBS): development and UK validation. Health Qual Life Outcomes 5(63):52. https://doi.org/10.1186/1477-7525-5-63
Herdman M, Gudex C, Lloyd A, Janssen M, Kind P, Parkin D et al (2011) Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res 20(10):1727–1736. https://doi.org/10.1007/s11136-011-9903-x
Rand S, Malley J, Towers A, Netten A, Forder J (2017) Validity and test-retest reliability of the self-completion adult social care outcomes toolkit (ASCOT-SCT4) with adults with long-term physical, sensory and mental health conditions in England. Health Qual Life Outcomes 15(1):163. https://doi.org/10.1186/s12955-017-0739-0
Aaronson N, Elliott T, Greenhalgh J, Halyard M, Hess R, Miller D, et al. (2015) User’s guide to implementing patient-reported outcomes assessment in clinical practice—version 2. International Society for Quality of Life Research, https://www.isoqol.org/wp-content/uploads/2019/09/2015UsersGuide-Version2.pdf. Accessed 11 Jan 2022
NHS England and NHS Improvement (2020) Social prescribing link workers: Reference guide for primary care networks—technical annex. https://www.england.nhs.uk/wp-content/uploads/2020/06/pcn-reference-guide-for-social-prescribing-technical-annex-june-20.pdf. Accessed 11 Jan 2022
Alonso J, Bartlett S, Rose M, Aaronson N, Chaplin J, Efficace F et al (2013) The case for an international patient-reported outcomes measurement information system (PROMIS®) initiative. Health Qual Life Outcomes 11:210. https://doi.org/10.1186/1477-7525-11-210
Foster A, O’Cathain A, Harris J (2020) How do third sector organisations or charities providing health and well-being services in England implement patient-reported outcome measures (PROMs)? A qualitative interview study. BMJ Open 10(10):e039116. https://doi.org/10.1136/bmjopen-2020-039116
Kyte D, Cockwell P, Lencioni M, Skrybant M, von Hildebrand M, Price G et al (2016) Reflections on the national patient-reported outcome measures (PROMs) programme: where do we go from here? J R Soc Med 109(12):441–445. https://doi.org/10.1177/0141076816677856
Nilsen P (2015) Making sense of implementation theories, models and frameworks. Implement Sci 10:53. https://doi.org/10.1186/s13012-015-0242-0
Bach-Mortensen A, Montgomery P (2019) Does sector matter for the quality of care services? A secondary analysis of social care services regulated by the Care Inspectorate in Scotland. BMJ Open 9(2):e022975. https://doi.org/10.1136/bmjopen-2018-022975
Churchill K, Warner L, Keogh E, Al SF (2021) Implementation of EQ-5D-5L as a routine outcome measure in Community Outpatient and Specialized Rehabilitation Services. J Patient Rep Outcomes 5(103):52. https://doi.org/10.1186/s41687-021-00369-z
Øvretveit J, Zubkoff L, Nelson E, Frampton S, Lehmann Knudsen J, Zimlichman E (2017) Using patient-reported outcome measurement to improve patient care. Int J Qual Health Care 29(6):874–879. https://doi.org/10.1093/intqhc/mzx108
Stover A, Haverman L, van Oers H, Greenhalgh J, Potter C, ISOQOL PROMs/PREMs in Clinical Practice Implementation Science Work Group (2021) Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res 30(11):3015–3033. https://doi.org/10.1007/s11136-020-02564-9
Biber J, Ose D, Reese J, Gardiner A, Facelli J, Spuhl J et al (2018) Patient reported outcomes—experiences with implementation in a University Health Care setting. J Patient Rep Outcomes 2:34. https://doi.org/10.1186/s41687-018-0059-0
Gibbons E, Fitzpatrick R (2018) An alternative approach to implementing patient-reported outcome measures. Pilot Feasibility Stud 4:96. https://doi.org/10.1186/s40814-018-0289-1
Howell D, Rosberger Z, Mayer C, Faria R, Hamel M, Snider A et al (2020) Personalized symptom management: a quality improvement collaborative for implementation of patient reported outcomes (PROs) in “real-world” oncology multisite practices. J Patient Rep Outcomes 4(1):47. https://doi.org/10.1186/s41687-020-00212-x
NHS (2019) The NHS long term plan. NHS. https://www.longtermplan.nhs.uk/. https://www.longtermplan.nhs.uk/wp-content/uploads/2019/08/nhs-long-term-plan-version-1.2.pdf
Crampton P, Woodward A, Dowell A (2001) The role of the third sector in providing primary care services—theoretical and policy issues. Soc Policy J N Zeal 17:1–21
Macmillan R (2010) The third sector delivering public services: An evidence review. University of Birmingham, Birmingham
Gamsu M, Southby K, South J, Newton R, Di Martino S, Bagnall A (2019) Debating what works for Community Wellbeing: Findings from two national public hearings on community wellbeing, health inequalities and local systems. What Works Centre for Wellbeing. What Works Centre for Wellbeing. https://whatworkswellbeing.org/product/public-debate-wellbeing/. Accessed 11 Jan 2022
Harris J, Springett J, Mathews D, Weston G, Foster A (2021) Using knowledge brokering to produce community-generated evidence. Evid Policy. https://doi.org/10.1332/174426421X16190024737973
Foster A, Croot L, Brazier J, Harris J, O’Cathain A (2018) The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews. J Patient Rep Outcomes 2:46. https://doi.org/10.1186/s41687-018-0072-3
Wallerstein N, Duran B (2006) Using community-based participatory research to address health disparities. Health Promot Pract 7(3):312–323. https://doi.org/10.1177/1524839906289376
Israel B, Eng E, Schulz A, Parker E (2005) Methods in community-based participatory research for health. Jossey Bass, San Francisco
Israel B, Schulz A, Parker E, Becker A, Allen A, Guzman J (2003) Critical issues in developing and following community-based participatory research principles. In: Minkler M, Wallerstein N (eds) Community-based participatory research for health. Jossey-Bass, San Francisco, pp 56–73
Kemmis S, McTaggert R (2000) Participatory action research. In: Denzin N, Lincoln Y (eds) Handbook of qualitative research, 2nd edn. Sage, Thousand Oaks, pp 567–607
Trondsen M, Sandaunet A (2009) The dual role of the action researcher. Eval Program Plann 32(1):13–20. https://doi.org/10.1016/j.evalprogplan.2008.09.005
Alvesson M, Skoldberg K (2000) Reflexive methodology: new vistas for qualitative research. Sage, London
Henderson J (2015) Community anchors. What works Scotland. http://whatworksscotland.ac.uk/wp-content/uploads/2015/11/WWSthinkpiece-Community-Anchors-Nov151.pdf. Accessed 15 Jan 2020
Charity Commission (2020) Register of individual charities. Charity Commission for England and Wales. https://apps.charitycommission.gov.uk/showcharity/registerofcharities/RegisterHomePage.aspx. Accessed 15 Jan 2020
NCVO (2017) Civil Society Almanac-2017. NCVO. https://ncvo-app-wagtail-mediaa721a567-uwkfinin077j.s3.amazonaws.com/documents/ncvo-uk-civil-society-almanac-2017.pdf
Sheffield City Council (Undated) Poverty in Sheffield: an assessment of need. Sheffield City Council. http://democracy.sheffield.gov.uk/documents/s17338/Tackling%20Poverty%20Strategy%202.pdf. Accessed 19 Nov 2019
Gonzalez J, Trickett E (2014) Collaborative measurement development as a tool in CBPR: measurement development and adaptation within the cultures of communities. Am J Community Psychol 54(1–2):112–124. https://doi.org/10.1007/s10464-014-9655-1
Pain R, Whitman G, Milledge D, Lune Rivers Trust (2012) Participatory action research toolkit: an introduction to using PAR as an approach to learning, research and action. Community Learning Partnership, http://communitylearningpartnership.org/wp-content/uploads/2017/01/PARtoolkit.pdf. Accessed 15 Mar 2017
Lee C, Mellor T, Dilworth-Anderson P, Young T, Brayne C, Lafortune L (2018) Opportunities and challenges in public and community engagement: the connected for cognitive health in later life (CHILL) project. Res Involv Engagem 4:42. https://doi.org/10.1186/s40900-018-0127-x
McKenna S, Iwasaki P, Stewart T, Main D (2011) Key informants and community members in community-based participatory research: one is not like the other. Prog Community Health Partnersh 5(4):387–397
Tessier S (2012) From field notes, to transcripts, to tape recordings: evolution or combination? Int J Qual Methods 11(4):446–460. https://doi.org/10.1177/160940691201100410
Phillippi J, Lauderdale J (2018) A Guide to field notes for qualitative research: context and conversation. Qual Health Res 28(3):381–388. https://doi.org/10.1177/1049732317697102
Braun V, Clarke V, Hayfield N, Terry G (2019) Thematic analysis. In: Liamputtong P (ed) Handbook of research methods in health social sciences. Springer, Singapore, pp 843–860
Moran-Ellis J, Alexander V, Cronin A, Dickinson M, Fielding J, Sleney J et al (2006) Triangulation and integration: processes, claims and implications. Qual Res 6:45–59. https://doi.org/10.1177/1468794106058870
Hong Q, Gonzalez-Reyes A, Pluye P (2018) Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract 24(3):459–467. https://doi.org/10.1111/jep.12884
Litchfield I, Greenfield S, Turner G, Finnikin S, Calvert M (2021) Implementing PROMs in routine clinical care: a qualitative exploration of GP perspectives. BJGP Open 5(1):135. https://doi.org/10.3399/bjgpopen20X101135
Wolpert M (2014) Uses and abuses of patient reported outcome measures (PROMs): potential iatrogenic impact of PROMs implementation and how it can be mitigated. Adm Policy Ment Health 41(2):141–145. https://doi.org/10.1007/s10488-013-0509-1
Polley M, Richards R (2019) A guide to selecting Patient Reported Outcome Measures (PROMs) for social prescribing. University of Westminster, London. https://www.london.gov.uk/sites/default/files/a_guide_to_selecting_outcomes_measures_in_social_prescribing_final.pdf. Accessed 15 Jan 2020
Terwee C, Crins M, Boers M, de Vet H, Roorda L (2019) Validation of two PROMIS item banks for measuring social participation in the Dutch general population. Qual Life Res 28(1):211–220. https://doi.org/10.1007/s11136-018-1995-0
Thestrup Hansen S, Kjerholt M, Christensen S, Hølge-Hazelton B, Brodersen J (2019) Haematologists’ experiences implementing patient reported outcome measures (PROMs) in an outpatient clinic: a qualitative study for applied practice. J Patient Rep Outcomes 3(1):74. https://doi.org/10.1186/s41687-019-0166-6
Chan E, Edwards T, Haywood K, Mikles S, Louise NL (2019) Implementing patient-reported outcome measures in clinical practice: a companion guide to the ISOQOL user’s guide. Qual Life Res 28(3):621–627. https://doi.org/10.1007/s11136-018-2048-4
We would like to thank the National Institute for Health Research for funding the Doctoral Research Fellowship-Alexis Foster, DRF-2016-09-007. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the National Institute for Health Research, Health Education England or the Department of Health and Social Care. We would like to thank the staff and service-users at the participating third sector organisations for taking part in and supporting the research.
The study was funded by a fellowship (Doctoral Research Fellowship-<REDACTED. DRF-2016-09-007) supported by the National Institute for Health Research. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the National Institute for Health Research, Health Education England or the Department of Health and Social Care.
Ethical approval and consent to participate
Approval was granted by the University of Sheffield (Application: 020700).
Consent for publication
Key informant interviewees signed a consent form agreeing to publication including the use of quotes. The two participating TSOs were partners in the research and their Implementation Leads are co-authors.
None of the authors have any completing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Foster, A., O’Cathain, A., Harris, J. et al. Using co-production to implement patient reported outcome measures in third sector organisations: a mixed methods study. J Patient Rep Outcomes 6, 78 (2022). https://doi.org/10.1186/s41687-022-00485-4
- Patient-reported outcomes
- Outcome assessment
- Third sector organisations