- Open Access
Stepping into the real world: a mixed-methods evaluation of the implementation of electronic patient reported outcomes in routine lung cancer care
Journal of Patient-Reported Outcomes volume 6, Article number: 70 (2022)
To realize the broader benefits of electronic patient-reported outcome measures (ePROMs) in routine care, we used the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework to inform the translation of a clinically effective ePROM system (hereafter referred to as the PRM system) into practice. The study aimed to evaluate the processes and success of implementing the PRM system in the routine care of patients diagnosed with lung cancer.
A controlled before-and-after mixed-methods study was undertaken. Data sources included a self-report questionnaire and interviews with healthcare providers, electronic health record data for PRMs patients and historical controls, and field notes. Descriptive statistics, logistic regression modelling, negative binomial models, generalized estimating equations and repeated measures ANOVA were used to analyze quantitative data. Qualitative data was thematically analyzed.
A total of 48/79 eligible people diagnosed with lung cancer completed 90 assessments during the 5-month implementation period (RE-AIM reach). Every assessment breached the pre-defined threshold and care coordinators reviewed and actioned 95.6% of breaches, resulting in 146 referrals to allied health services, most frequently for social work (25.3%), dietetics (18.5%), physiotherapy (18.5%) and occupational therapy (17.1%). PRMs patients had significantly fewer visits to the cancer assessment unit for problematic symptoms (M = 0.23 vs. M = 0.43; p = 0.035), and were significantly more likely to be offered referrals (71% vs. 29%, p < 0.0001) than historical controls (RE-AIM effect). The levels of ‘organizational readiness for implementing change’ (ORIC) did not show much differences between baseline and follow-up, though this was already high at baseline; but significantly more staff reported improved confidence when asking patients to complete assessments (64.7% at baseline vs. 88.2% at follow-up, p = 0.0046), and when describing the assessment tool to patients (64.7% at baseline vs. 76.47% at follow-up, p = 0.0018) (RE-AIM adoption). A total of 78 staff received PRM system training, and 95.6% of the PRM system alerts were actioned (RE-AIM implementation); and all lung cancer care coordinators were engaged with the PRM system beyond the end of the study period (RE-AIM maintenance).
This study demonstrates the potential of the PRM system in enhancing the routine care of lung cancer patients, through leveraging the capabilities of automated web-based care options.
Plain English summary
Research has shown the clear benefits of using electronically collected patient-reported outcome measures (ePROMs) for cancer patients and health services. However, we need to better understand how to implement ePROMs as part of routine care. This study evaluated the processes and outcomes of implementing an ePROMs system in the routine care of patients diagnosed with lung cancer. Key findings included: (a) a majority of eligible patients completed the scheduled assessments; (b) patient concerns were identified in every assessment, and care coordinators reviewed and actioned almost all of these, including making significantly more referrals to allied health services; (c) patients completing assessments regularly were less likely to present to the cancer assessment unit with problematic symptoms, suggesting that ePROMs identified patient concerns early and this led to a timely response to concerns; (d) staff training and engagement was high, and staff reporting increased confidence when asking patients to complete assessments and when describing the assessment tool to patients at the end of the implementation period. This study shows that implementing ePROMs in routine care is feasible and can lead to improvements in patient care.
Lung cancer is the leading cause of cancer death globally . Despite improved survival and prognosis, the physical and psychosocial issues which appear early in the cancer journey often persist for patients who move into long-term survivorship, and often precipitate unnecessary hospitalization and significant individual and health system burden [2, 3].
The anticipated increased demand for cancer services, plus the COVID-19 pandemic, highlight the need for flexible patient care. Telehealth, web-based care and long-term follow-up are potentially viable alternative/complementary models of care for this growing demand [4,5,6]. Remote web-based patient-reported outcome measures (ePROMs) facilitate patients reporting issues of concern to their care team, thereby may prompt timely care according to level of need [7, 8].
Health care services are increasingly incorporating ePROMs to inform person-centred care and evaluate services . In the research context, well-integrated ePROM systems are demonstrated to be acceptable and feasible to implement  with improved patient and health system outcomes, including patient-provider communication, patient satisfaction , health-related quality of life [12, 13], compliance with chemotherapy ; earlier detection of relapse in lung cancer patients ; reduced emergency department (ED) presentations [15,16,17]; and improved cancer survival [16, 18].
In 2013, our team developed an ePROM system, PROMPT-Care (Patient Reported Outcome Measures for Personalized Treatment and Care) and demonstrated its acceptability, feasibility , and impact on ED presentations . This evidence provided the impetus for its implementation in routine care. However, despite good evidence of ePROMs’ effectiveness in improving patient and health service outcomes and some guidelines available [19,20,21], implementation in routine clinical care remains challenging; and internationally, there are few examples to guide large-scale ePROM implementation in oncology, with Cancer Care Ontario being a noteworthy exception .
Implementation science can inform the translation of clinically effective interventions, such as ePROMs, into practice . The identified potential barriers and facilitators to implementing ePROMs in health services [9, 23,24,25] suggest a strong need for organizational preparedness through stakeholder engagement and organizational resourcing. Recommended strategies include communicating the robustness and value of collecting PROMs, integrating PROMs into the electronic medical record (eMR), training clinicians in using them, integrating ePROM collection into clinical workflows, and technical support and feedback mechanisms in clinics [9, 24, 25].
This study aimed to evaluate the processes and success of implementing our ePROM system in the routine care of patients diagnosed with lung cancer.
Ethics and governance approval was received from the South Western Sydney Local Health District (SWSLHD) ethics committee, Ref. 2020/ETH01052. Standards for reporting implementation studies of complex interventions (StaRI) guidelines  were followed.
A controlled before-and-after mixed-methods study was undertaken to evaluate the implementation of the new model of care.
Implementation context and setting
The implementation context for ePROMs was influenced by the socio-economic environment of participating hospitals, which is known as a socially, economically, culturally, and linguistically diverse population, with a lot of its population born overseas, and almost half of its population speaking a language other than English at home (as previously described ). This study was conducted in (SWSLHD), a metropolitan area in Sydney, Australia. SWSLHD comprises six hospitals, serving more than 966,000 people (12% of residents in New South Wales (NSW), Australia’s most populous state). The area contributes 10% of total new cancer cases in NSW . Participating hospitals had care coordinator resources, access to specialist lung cancer clinicians and treatments, and previous PROMPT-Care system experience.
The ePROM (PRM) system
The PROMPT-Care Version 2 ePROM system, previously described , is fully integrated into the patients’ eMR and supports patient management through (a) monthly physical and psychosocial wellbeing ePROM using the Distress Thermometer (DT) and associated checklist , and the Edmonton Symptom Assessment Scale (ESAS) , (b) automated email clinical alerts notifying the care team of unresolved clinical issues, and (c) tailored patient self-management resources to help them meet their identified needs and concerns. The Cancer Institute NSW (the Government-funded state-wide cancer control agency) adapted Version 2, including automating much of its functionality. This adapted Version 3, the system implemented in routine care in SWSLHD, is hereafter referred to as the PRMs system. The key changes on the latest version of the PRMs system were: (a) the system being hosted and integrated by the state-wide government-funded cancer authority (the CINSW) and not the individual local health districts, (b) deleting the 9-item Supportive Care Needs Survey-Screening Tool 9, (c) patients being able to elect receipt of surveys via email or text message (SMS), (d) incorporation of greater privacy measures (e.g. a 2-step device authentication, including acceptance to receive or decline surveys).
We adopted the RE-AIM framework to guide implementation. Its five domains are: (a) reach of the target population; (b) effect on key outcomes; (c) adoption by people responsible for its delivery; (d) success of its implementation; and (e) potential for it to be maintained .
Health care professionals (HCPs)
Staff at participating hospitals involved in lung cancer care delivery, including but not limited to administrative, specialist, nursing and allied health staff, had potential involvement in implementation.
All eligible patients were introduced to PRMs screening as part of their routine care, if they: (a) had a confirmed diagnosis of lung cancer, (b) were receiving care in a participating hospital, (c) were able to complete (themselves or with assistance) the PRMs assessment in English, and (d) were not currently participating in a clinical trial.
Only de-identified patient data was required for evaluation purposes. Data were extracted from the eMR, providing general data on ED presentations, and the oncology information system (OIS), providing data related to diagnosis, cancer assessment unit (CAU), and medical and radiation oncology. The CAU is where unwell cancer patients can present for assessment and management by an experienced nurse and oncology trainees to prevent ED presentation and hospital admissions.
During their first post-diagnosis clinic appointment, patients were informed that they would complete PRMs assessments regularly as part of their routine care. In clinic, care coordinators provided patients with an iPad to complete their first assessment, with subsequent assessments completed either in-clinic or remotely from home at a frequency determined by each participating hospital. Care coordinators reviewed patients’ PRMs results generated in the eMR and responded to automated email clinical alerts identifying patients with high levels of psychosocial distress or physical symptoms, offering interventions or referrals as necessary.
The implementation plan is described elsewhere . A multidisciplinary implementation advisory group was formed, with representation from participating hospitals, to identify local champions to adapt and incorporate the PRMs model of care into the local contexts. Stakeholders decided on patient eligibility (lung cancer selected for initial roll-out), onboarding PRMs processes, and follow-up and referral pathways at planning meetings. Processes had to align with existing workflows, use existing staff resources, minimize staff and patient burden, and maximize onboarding. Implementation planning occurred from October 2019 to November 2020, GO-Live for the enhanced PRMs system was achieved on 16th November 2020, and evaluation occurred five months later. The period of implementation spanned the COVID-19 outbreak, and included multiple periods of local community lockdowns. This had significant effects on the models of care of the entire cancer service and the PRMs implementation programme, including using much more teleconferencing and much less face to face discussion. Some cancer care coordinators preferred conducting the initial onboarding of patients via phone consultations beyond the COVID-19 outbreak. The implementation group met regularly to problem shoot during this period.
The historical control group included new lung cancer patients seen at participating centres between mid-November 2018 and mid-April 2019, coinciding with the 5-month implementation evaluation period, in the year immediately preceding the COVID-19 pandemic. For comparability between the PRMs and control cohorts, clinicians screened control patients to ensure eligibility matched the PRMs patients.
Data collection sources
Clinic audit logs
We conducted a pre-implementation clinic audit to determine the proportion of patients with access to out-of-clinic internet; preferring to complete PRMs assessments on paper versus electronically; capable of completing assessments in English; and anticipating requiring assistance to complete assessments. This audit included patients with all cancer types and from hospitals in SWSLHD and elsewhere, as reported previously . This paper only reports on the results of the lung cancer sub-group of the published audit.
Invitations to complete a baseline online survey were emailed to HCPs who participated in any workshops, meetings, orientation and training regarding PRMs system implementation between June and November 2020. Five months after the GO-Live date, a subsequent invitation to complete an online follow-up survey was emailed to those who completed the baseline survey. Both the baseline and follow-up surveys included the Organizational Readiness for Implementing Change (ORIC) scale and HCP survey (see below). The follow-up survey included additional questions regarding HCPs’ engagement with the PRMs reports.
The ORIC survey  comprises 12-items and two subscales—change commitment (a shared resolve to implement a change within an organization), and change efficacy (collective capability to implement a change). Higher scores (range 12–60) indicate greater organizational readiness for change. The scale has strong psychometric properties and has been validated in real-world hospital settings .
The HCP survey was adapted from a previously published measure , examining staff and environmental factors related to successful implementation of interventions in hospital settings . The 27-item questionnaire (Additional file 1) assessed barriers (8-items), knowledge and attitudes (4-items), HCP confidence and role (6-items); and demographic and workplace characteristics (9-items); and included a free-text question on barriers.
At study completion, 11 HCPs and OIS staff were purposefully selected to provide feedback on the PRMs system in semi-structured virtual interviews. Staff were selected to represent the continuum of activities across the PRMs system, including system infrastructure and IT support (n = 2), onboarding patients, reviewing PRMs results and coordinating patient care (n = 3 care coordinators), oncology care (n = 2 medical, and n = 3 radiation oncologists), and response to allied health referrals (n = 1 social worker). Depending on their specialty, participants were asked about aspects of implementation including what worked well and did not, strategies for improving clinician engagement, and recommendations for system enhancements before wider-scale implementation with other tumour groups.
HCP training log
A log was maintained of the number of HCPs participating in training, including completing orientation and receiving the implementation resources/toolkit.
Patient health records
OIS staff extracted patient demographic, clinical, and health services data; and eMR staff extracted data on the use of the CAU and ED presentations.
Stakeholder meetings and HCP workshops were audio-recorded and monitored via field notes, as detailed elsewhere .
Table 1 details how specific components of the PRMs system implementation were evaluated based on the RE-AIM domains. To evaluate the implementation process: (1) baseline data were extracted for pre-GO-Live and follow-up data five months later (Reach, Adoption, Maintenance); (2) data were collected throughout implementation (Reach, Adoption, Implementation, Maintenance); and (3) historical control data were compared with implementation data throughout implementation (Effect).
Quantitative data collected via audits, logs and surveys were analyzed using descriptive statistics including means (and Standard Deviations (SD)), frequencies, and proportions. Multivariable logistic regression modelling was used to analyze the number of above-threshold breaches (< 13 vs. ≥ 13; 13 being the average number of items breached across all assessments), ED presentations (≥ 1 visit vs. no visits), CAU visits (≥ 1 visit vs. no visits), and referral acceptance (yes/no). Negative binomial models were used to analyze the numbers of ED presentations, hours hospitalized due to ED presentations, CAU presentations, and referred patients. ED presentations were also compared against the group of patients who declined PRM assessments. Generalized estimating equations (GEE) were used to analyze the likelihood of referral acceptance for those breaching a high number of items. Repeated measures ANOVA was used to analyze ORIC scores over time. Statistical analysis was performed in SAS Enterprise Guide version 8.2.
Qualitative data from open-ended survey responses, field notes and interviews were thematically analyzed . Staff interviews were audio-recorded, transcribed verbatim, and analyzed, with two researchers (AG, OR) independently reading the transcripts and generating initial codes. Identified codes were collated into emerging themes and refined, resolving discrepancies through discussion and consensus. This occurred throughout implementation and post-implementation.
The number, proportion and representativeness of patients who attended lung cancer clinics was assessed . Of the 64 audited patients, 54.7% (35/64) reported having internet/mobile phone access; 34.4% (22/64) had email access; and 68.7% (44/64) felt capable of completing assessments in English, with the top three other languages nominated being Vietnamese (27.3%, 6/22), Serbian (13.6%, 3/22) and Arabic (9.1%, 2/22). Almost half (45.3%; 29/64) of the audited patients reported requiring assistance to complete assessments; and 57.8% (37/64) expressed a preference for paper assessments, 28.1% (18/64) using a device and 6.2% (4/64) either.
As shown in the CONSORT diagram (Fig. 1), of the 79 eligible new lung cancer patients, 95% (n = 75) were screened during the 5-month evaluation period, and 61% (n = 48) of the target population was onboarded, i.e. reached by implementation efforts. Out of 75 screened patients, 27 (36%) declined (see Fig. 1 for reasons). Reasons why some patients (5%, n = 4) were systematically missed at Hospital #1 are unknown. Characteristics of onboarded patients are shown in Table 2 (n = 48).
Although participants were expected to complete an assessment on a monthly basis, the patient-centred approach of the cancer service meant that some patients were scheduled to complete assessments more or less frequently, depending on the stage of their cancer journey, treatment plan and personal preferences (e.g. some participants preferred to complete assessments less frequently). For these reasons, approximately half of the participants (54.1%, n = 26) completed only one assessment during the evaluation period. Of n = 12 who were due for a second assessment, only n = 2 declined to complete it (4% of the whole sample), with other reasons for non-completion including patient deceased (n = 4, 8%) or technical reasons for assessment not being sent (n = 7, 14%).
The 48 PRMs patients all completed at least one assessment, with a total of 90 assessments completed overall. Every assessment resulted in at least one breach (100% breach rate), with 13.4 out of 49 items (27%) breached on average. The three most breached items were worry (65.6%), tiredness (64.4%) and fatigue (61.1%). Multivariable logistic regression analysis revealed that patients breaching ≥ 13 items were younger (p = 0.002) and more likely to be female (p = 0.004) than patients breaching < 13 items. Stage of disease did not affect the number of items breached.
There were no significant differences in gender, age or stage of disease between the PRMs group (n = 48), patients who declined onboarding (n = 27) and historical controls (n = 63).
No between-group differences were detected in the number of patients with ≥ 1 ED presentation (multi-variable logistic regression; PRMs 58.3%, 28/48; Controls 60.3%, 38/63; Declined 63%, 17/27); in the number of hours hospitalised due to ED presentations, controlling for age, gender and stage (multivariable negative binomial regression; PRMs M = 114.31 h, SD = 240.69; Controls M = 91.94 h, SD = 145.93; Declined M = 75.70 h, SD = 129.22); nor in the average number of ED presentations per patient (PRMs M = 1.31; Control M = 1.13; Declined M = 1.04).
PRMs patients were significantly less likely to have ≥ 1 CAU visit compared to historical Controls (PRMs 20.8%, 10/48; Controls 31.8%, 20/63; Declined 14.8%, 4/27, p = 0.021), after adjusting for gender, age and disease stage. Similarly, the PRMs group had a significantly lower average number of CAU presentations per patient than Controls (multivariable negative binomial regression; M = 0.229 vs. M = 0.429; p = 0.035).
There was no significant relationship between the total number of breached items and the likelihood of visiting the CAU (mean of 16 breached items for CAU attendees versus 13 for non-attendees).
Allied health referrals
Across all assessments completed and breached (n = 90), 146 referrals were made to allied health services, most frequently for social work (25.3%), dietetics (18.5%), physiotherapy (18.5%) and occupational therapy (17.1%) (see Table 3).
A significantly higher proportion of PRMs patients (70.8%, n = 34/48) were offered ≥ 1 referral during the evaluation period compared to Controls (28.7%, n = 18/63, p < 0.0001). The PRMs group also had a significantly higher average number of referrals per patient than Controls (M = 1.27 vs. M = 0.38, p < 0.0001). Patients with more breached items were more likely to accept a referral (p = 0.0005). The likelihood of accepting a referral was not influenced by gender or stage of disease (gender p = 1; stage p = 0.779), or by patient’s age (p = 0.615).
Feasibility of the implementation amongst HCPs
ORIC assessments were completed by HCPs, administrative and management staff, among others (baseline, n = 37; follow-up, n = 28; see demographics in Table 4). At baseline, organizational readiness for change (mean total ORIC score = 47.24, SD = 8.03), change commitment (M = 20.57 (SD = 3.51)) and change efficacy (M = 26.68, SD = 5.33) were all high, and repeated measures ANOVA (paired responses only), revealed no significant changes at follow-up.
Twenty-one staff completed the baseline HCP surveys (see demographics, Table 4), and 17 completed follow-up HCP surveys. Only paired responses (n = 17) were included in the nominal symmetry test. Significant changes were detected in 2/18 items at follow-up, with more staff reporting a higher level of confidence when asking patients to complete assessments (64.7% at baseline vs. 88.2% at follow-up, p = 0.0046), and when describing the assessment tool to patients (64.7% at baseline vs. 76.47% at follow-up, p = 0.0018). See Additional file 2 for all other results.
Change agents implementing ePRO
The change agents implementing PRMs included lung cancer care coordinators (n = 4/4), PRMs champions including administrative, management and nursing staff who facilitated patient onboarding (n = 6/6), medical oncologists (n = 6/11), and radiation oncologists (6/9), resulting in 73% (n = 22/30) total change agents.
Number and proportion of patients screened
At Hospital #1, there were 32 eligible new lung cancer patients in the 5-month evaluation period, of whom 87.5% (n = 28) were screened, 43.8% (n = 14) were onboarded, 43.8% (n = 14) declined and 12.5% (n = 4) were systematically missed. At Hospital #2, all 34 (100%) eligible patients were screened and 70.6% (n = 24) were onboarded, 29.4% (n = 10) declined. At Hospital #3, all 13 (100%) eligible patients were screened, 76.9% (n = 10) were onboarded and 23.1% (n = 3) declined.
Clinical alerts actioned
Every completed assessment (n = 90) had breached items; 95.6% (n = 86) were actioned by HCPs.
Number of cancer staff receiving orientation
Across the three hospitals, 78 staff received orientation and/or training and implementation resources (training materials, user guides). Trained staff included allied health professionals (n = 27), administrative and management staff (n = 22), care coordinators (n = 7), radiation oncologists (n = 6), medical oncologists (n = 6), nurses (n = 5), IT Specialists (n = 2), pathologist (n = 1), radiation therapist (n = 1). The PRMs webpage housing PRMs resources, accessible by any hospital staff, had 52 views.
Staff maintaining engagement
The model of care adopted for implementation was driven by the care coordinators , all of whom (n = 4, 100%) maintained engagement over the implementation period, through onboarding eligible patients, assisting patients to complete assessments as required, reviewing the PRMs reports and actioning clinical alerts for breached assessments.
More broadly, in the HCP follow-up survey, more than a third (35%, 6/17) reported discussing the PRMs assessment results with a few, most, or all of their lung cancer patients. Furthermore, more than a third (35%, 6/17) said yes or sometimes to reviewing the care recommendations provided for above-threshold scores on the assessments. However, it is worth noting that the HCP survey participants included allied health professionals (n = 10), care coordinators (n = 5), and nurses (n = 2), and the questions were most relevant to care coordinators and nurses.
Staff feedback on PRMs implementation
The main themes generated from staff interviews and surveys, and field notes, centred on: (a) benefits of PRM implementation, including perceived improvements in patient care; (b) how to engage non-English speaking patients; (c) the role of care coordinators and the extent of clinicians’ engagement; and (d) value of extending this model to all cancers. Illustrative HCP quotes are included in Table 5.
Benefits of PRM implementation
Some participants perceived that the PRMs assessments significantly improved their clinical service and increased patient contact (Table 5, section A). Others thought that using the tool in routine care formalizes their current practice while improving the patient experience (Table 5, section B).
Engaging non-English speaking patients
Staff highlighted challenges engaging non-English speaking patients and made suggestions regarding implementing PRMs with these patients and those with low digital literacy (Table 5, section C). Some improvements in procedures were recommended, including patient onboarding and tailoring assessment frequency (Table 5, section D).
Care coordinator role and clinician engagement
Care coordinators were perceived as integral to successful delivery of PRMs-informed care by oncologists (Table 5, section E) and allied health staff (Table 5, section F). However, while doctors largely relied on care coordinator notes to prioritize patients’ concerns, care coordinators perceived a greater need for clinicians to engage and acknowledge PRMs assessment reports during the clinical consultation, to facilitate patient engagement (Table 5, section G).
Extending this model of care
Most staff strongly endorsed the system being broadly implemented across all tumour sites, with doctors in particular suggesting that it would make it easier for clinicians, who would get used to expecting to see assessments from ALL patients, rather than a sub-group. However, the importance of the model being adequately resourced was unanimously reiterated, particularly with tumour-specific care coordinators and allied health services to deal with increased referrals (Table 5, section H). Implementation across the whole cancer service was also perceived to support a more holistic, team-based model of care (Table 5, section I).
To leverage the established benefits of utilizing ePROMs in routine clinical care, we drew on an implementation science approach using the RE-AIM framework to plan and evaluate the PRMs system implementation in the routine care of lung cancer patients. These patients are often older, with co-morbidities, commonly diagnosed at an advanced stage, and with a complex array of supportive care needs, hence the imperative for readily identifying and addressing their needs .
Our results suggest that the PRMs system was overall successfully implemented into routine care, with a high level of reach in the target population, and with measurable effect in increased allied health referrals and fewer PRMs patients attending the CAU than control patients. The results suggest timely response to the early identification of patients’ concerns. Staff engagement in PRMs training activities was high, and likely associated with improved confidence when asking patients to complete assessments and when describing the assessment tool to patients (adoption, implementation). Staff supported extending the PRMs system to all tumour groups, provided adequate care coordinator and allied health resources were available.
This study contributes important new knowledge to understanding how to translate effective interventions into routine care and address some of the specific local challenges arising from implementation processes. In preparation for implementation, we had conducted a clinic audit to understand our specific population’s capabilities for completing PRMs assessments. Despite two-thirds of patients in the audit reporting being capable of completing assessments in English , PRM implementation revealed that almost half (46%) of the target population were deemed ineligible due to lower levels of literacy, predominantly due to language. Implementation sites had large non-English speaking populations, and the availability of English-only assessments at the time of implementation meant the majority of this target group were not onboarded unless they were assisted by family or interpreters. A multi-language PRMs system is in development to extend the reach of PRM implementation. However, the results highlight the importance of other teams undertaking implementation projects to fully understand their local contexts and populations, to “leave no one behind” and avoid widening existing disparities .
Measuring the effect of implementation requires careful thought about the relevant outcomes and timeframe for detecting change. Our published trial of the ePROM system  reported 33% significantly fewer ED visits in the intervention group. While this implementation study failed to detect a similar finding, potentially due to a limited 5-month evaluation period (tied to funding timeline) and limited sample size, we observed other effects indicative of an intervention impact. Compared to historical controls, PRMs patients received significantly more allied health service referrals and had fewer CAU attendances. These two results are likely inter-related, with the higher rate of allied health referrals potentially reducing the need for CAU visits, suggesting that early management of symptoms may mitigate the escalation of toxicities  and supporting the effect of implementation.
As part of evaluating adoption, understanding organizational readiness for change before PRMs implementation in routine care is critical, as staff in organizations with higher readiness for change are more likely to initiate change, be collaborative and cooperative, and willing to implement new evidence-based practices [42, 43]. With staff at two of the three hospitals having been familiarised with ePROMs during our previous trial , it is unsurprising that the baseline level of organizational readiness for change was high, with little shift throughout implementation. Furthermore, rigorous stakeholder engagement activities resulted in a high rate of relevant staff (73%) being implementation change agents, aligning with the literature highlighting stakeholder engagement  and use of change agents or clinical champions [45, 46] as important implementation strategies. Our results support the importance of carefully selecting initial implementation sites with greater readiness for change and clinical champions, as these become influential when engaging new sites. In the local context, the remaining hospitals in SWSLHD requested the PRMs system as soon as feasible after seeing the level of support from initial sites (personal communication).
Our trial  supported PRMs implementation driven by care coordinators, which is consistent with the majority of published ePROM trials reporting nursing staff reviewing and actioning ePROM results [47,48,49,50], compared to only a few studies referring to doctors . Implementation success was facilitated by including the lung cancer care coordinators in the implementation planning, ensuring a functional and efficient workflow was developed for onboarding patients, reviewing assessment results, making referrals, triaging and providing direct care, as required. Coordinators were critical “filtering agents” for oncologists, reviewing PRM reports and highlighting specific issues to address during patient consultations. This process was highly valued by doctors, for whom it was a significant time-saver. Implementation success could have been further enhanced if the doctors made more explicit reference to PRMs reports during consultations. Coordinators perceived this lack of explicit acknowledgement as resulting in some patients losing interest in completing assessments. Our findings are consistent with a UK trial showing that most doctors (63%) did not explicitly refer to PROMs data and patients perceive their doctors to have greater knowledge than nurses . Doctors’ acknowledgement of ePROMs results could potentially improve patient engagement levels and should be included in training/orientation processes in future implementation projects.
Staff engagement strategies, including PRMs orientation, training and re-training in accessing PRM reports, appeared to have a significant impact on staff confidence at follow-up, when describing and asking patients to complete the ePROM tool. This is consistent with evidence about the importance of supportive strategies such as staff education . Furthermore, staff perceived the PRMs system as beneficial to patient care, strongly endorsing its implementation across all tumour groups. It aligns with systematic review evidence and meta-synthesis that oncology clinicians’ perception of the value of the intervention may promote adoption . The evaluation data suggests that the PRMs system was successfully adopted by staff at participating hospitals.
Our PRMs system was successfully implemented at all three hospitals, with high levels of patients onboarding, review and actioning of clinical alerts. Two of the three hospitals successfully onboarded all eligible patients (100%); and nurses demonstrated an even higher rate of reviewing and actioning clinical feedback reports (95.6%) in routine care than previously achieved in research settings [17, 50]. The response to clinical alerts included significantly more referrals to allied health services in PRMs versus historical control groups. Importantly, the referral recipients, allied health staff, highlighted that they received more appropriate referrals and that the PRMs reports helped inform care delivery.
The COVID-19 pandemic period, especially the community lockdowns, highlighted the importance of collecting routine PRMs from patients. The lack of face to face consultations during this period are likely to have reduced clinical staff’s ability to detect distress or other psychological issues in patients via their usual consultations methods. During this period patients expressed a strong desire to retain more remote consultations rather than the inconvenience, infection risk and cost of face to face consultations. It was thought that in the future PRMs could actually support a move to more remote consultations with patients even in periods without community lockdown.
Whilst five months is insufficient to assess whether the PRMs system was successfully maintained over time, the RE-AIM framework guided planning to increase likely maintenance. The staff interviews provided important contextual insights to support ongoing maintenance, particularly the need for adequate human resource infrastructure of care coordinators and allied health staff. This is consistent with systematic review evidence that resource availability enables successful PRMs implementation and sustainability . Staff recommended other strategies to support sustained implementation, including refinements to onboarding procedures for first and subsequent ePROMs, and tailoring ePROM survey frequency to patient need. As oncology care settings learn from their experience, the use of the PRMs system will continually optimize the timing of administering ePROMs .
Participating hospitals were based in a community with a high proportion of the population born overseas (43%), and almost half (45%) speaking a language other than English at home . Since the PRMs system was only available in English, the sample excludes a sizable proportion of the local community. This study may have benefited from patient feedback regarding the system-level changes they experienced, to compare their views with those from healthcare professionals. The COVID-19 pandemic impacted some outcomes, including ED presentations. Other impacts on implementation effectiveness are unclear, such as visits to the CAU.
Conclusion and practical implications
Our study demonstrates the potential of the PRMs system to enhance the routine care of lung cancer patients, and the value of using the RE-AIM framework for implementation planning and defining outcomes to measure implementation success in a real-world setting. Critical success factors were developing multi-faceted evidence-based implementation strategies that were tailored to local contexts, as was large-scale engagement and training of stakeholders. These factors are recommended for adoption for future PRMs implementation initiatives. Successful PRMs system implementation in routine care could pave the way for redefining models of care that leverage the capabilities of automated web-based strategies and engaging staff across multiple disciplines in implementation processes .
Availability of data and materials
De-identified data that supports the findings of this study are available upon reasonable request to the corresponding author and approval from the South Western Sydney Local Health District ethics committee.
Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F (2021) Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: Cancer J Clin 71(3):209–249. https://doi.org/10.3322/caac.21660
Enstone A, Greaney M, Povsic M, Wyn R, Penrod JR, Yuan Y (2018) The economic burden of small cell lung cancer: a systematic review of the literature. Pharmacoecon Open 2(2):125–139
Goodridge D, Lawson J, Duggleby W, Marciniuk D, Rennie D, Stang M (2008) Health care utilization of patients with chronic obstructive pulmonary disease and lung cancer in the last 12 months of life. Respir Med 102(6):885–891
Mann DM, Chen J, Chunara R, Testa PA, Nov O (2020) COVID-19 transforms health care through telemedicine: evidence from the field. J Am Med Inform Assoc 27(7):1132–1135
Ali SA, Arif TB, Maab H, Baloch M, Manazir S, Jawed F, Ochani RK (2020) Global interest in Telehealth during COVID-19 pandemic: an analysis of Google Trends™. Cureus 12(9):e10487
Soran A, Gimbel M, Diego E (2020) Breast cancer diagnosis, treatment and follow-up during COVID-19 pandemic. Eur J Breast Health 16(2):86
Aiyegbusi OL, Nair D, Peipert JD, Schick-Makaroff K, Mucsi I (2021) A narrative review of current evidence supporting the implementation of electronic patient-reported outcome measures in the management of chronic diseases. Ther Adv Thronic Dis 12:20406223211015960. https://doi.org/10.1177/20406223211015958
Coons SJ, Eremenco S, Lundy JJ, O’Donohoe P, O’Gorman H, Malizia W (2015) Capturing patient-reported outcome (PRO) data electronically: the past, present, and promise of ePRO measurement in clinical trials. Patient Patient Cent Outcomes Res 8(4):301–309. https://doi.org/10.1007/s40271-014-0090-z
Rutherford C, Campbell R, Tinsley M, Speerin R, Soars L, Butcher A, King M (2021) Implementing patient-reported outcome measures into clinical practice across NSW: mixed methods evaluation of the first year. App Res Qual Life 16(3):1265–1284. https://doi.org/10.1007/s11482-020-09817-2
Girgis A, Durcinoska I, Levesque JV, Gerges M, Sandell T, Arnold A, Delaney GP (2017) eHealth system for collecting and utilizing patient reported outcome measures for personalized treatment and care (PROMPT-Care) among cancer patients: mixed methods approach to evaluate feasibility and acceptability. J Med Internet Res 19(10):e330. https://doi.org/10.2196/jmir.8360
Chen J, Ou L, Hollis SJ (2013) A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res 13(1):211. https://doi.org/10.1186/1472-6963-13-211
Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, Selby PJ (2004) Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol 22(4):714–724
Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, Rogak L, Bennett AV, Dueck AC, Atkinson TM, Chou JF, Dulko D, Sit L, Barz A, Novotny P, Fruscione M, Sloan JA, Schrag D (2016) Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 34(6):557–565. https://doi.org/10.1200/jco.2015.63.0830
Denis F, Viger L, Charron A, Voog E, Letellier CJSCIC (2014) Detecting lung cancer relapse using self-evaluation forms weekly filled at home: the sentinel follow-up. Support Care Cancer 22(1):79–85. https://doi.org/10.1007/s00520-013-1954-9
Barbera L, Sutradhar R, Howell D, Sussman J, Seow H, Dudgeon D, Atzema C, Earle C, Husain A, Liu Y, Krzyzanowska MK (2015) Does routine symptom screening with ESAS decrease ED visits in breast cancer patients undergoing adjuvant chemotherapy? Support Care Cancer 23(10):3025–3032. https://doi.org/10.1007/s00520-015-2671-3
Basch E, Deal AM, Dueck AC, Scher HI, Kris MG, Hudis C, Schrag D (2017) Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 318(2):197–198. https://doi.org/10.1001/jama.2017.7156
Girgis A, Durcinoska I, Arnold A, Descallar J, Kaadan N, Koh E-S, Miller A, Ng W, Carolan M, Della-Fiorentina SA (2020) Web-based patient-reported outcome measures for personalized treatment and care (PROMPT-Care): multicenter pragmatic nonrandomized trial. J Med Internet Res 22(10):e19685
Denis F, Lethrosne C, Pourel N, Molinier O, Pointreau Y, Domont J, Bourgeois H, Senellart H, Trémolières P, Lizée T, Bennouna J, Urban T, El Khouri C, Charron A, Septans A-L, Balavoine M, Landry S, Solal-Céligny P, Letellier C (2017) Randomized trial comparing a web-mediated follow-up with routine surveillance in lung cancer patients. JNCI J Nat Cancer Inst 109(9). https://doi.org/10.1093/jnci/djx029
Osoba D (2007) Translating the science of patient-reported outcomes assessment into clinical practice. JNCI Monographs 2007(37):5–11. https://doi.org/10.1093/jncimonographs/lgm002
Aaronson N, Choucair A, Elliott T, Greenhalgh J, Halyard M, Hess R, Miller D, Reeve B, Santana M, Snyder C (2011) User’s guide to implementing patient-reported outcomes assessment in clinical practice. J Int Soc Qual Life Res
Porter I, Gonçalves-Bradley D, Ricci-Cabello I, Gibbons C, Gangannagaripalli J, Fitzpatrick R, Black N, Greenhalgh J, Valderas JM (2016) Framework and guidance for implementing patient-reported outcomes in clinical practice: evidence, challenges and opportunities. J Compr Effect Res 5(5):507–519. https://doi.org/10.2217/cer-2015-0014
Montgomery N, Howell D, Ismail Z, Bartlett SJ, Brundage M, Bryant-Lukosius D, Krzyzanowska M, Moody L, Snyder C, Barbera L, Committee TCCOPROA (2020) Selecting, implementing and evaluating patient-reported outcome measures for routine clinical use in cancer: the Cancer Care Ontario approach. J Patient-Rep Outcomes 4(1):101. https://doi.org/10.1186/s41687-020-00270-1
Stover AM, Haverman L, van Oers HA, Greenhalgh J, Potter CM, Isoqol PROMs PREMs in Clinical Practice Implementation Science Work Group (2020) Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res. https://doi.org/10.1007/s11136-020-02564-9
Foster A, Croot L, Brazier J, Harris J, O’Cathain A (2018) The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews. J Patient-Rep Outcomes 2(1):46. https://doi.org/10.1186/s41687-018-0072-3
Amini M, Oemrawsingh A, Verweij LM, Lingsma HF, Hazelzet JA, Eijkenaar F, van Leeuwen N (2021) Facilitators and barriers for implementing patient-reported outcome measures in clinical care: an academic center’s initial experience. Health Policy. https://doi.org/10.1016/j.healthpol.2021.07.001
Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, Rycroft-Malone J, Meissner P, Murray E, Patel A, Sheikh A, Taylor SJC (2017) Standards for reporting implementation studies (StaRI): explanation and elaboration document. BMJ Open 7(4):e013318. https://doi.org/10.1136/bmjopen-2016-013318
Bamgboje-Ayodele A, Arnold B, Durcinoska I, Avery S, Vinod S, Rincones O, Thomas T, Descallar J, Smith B, Delaney G, The PROMPT-Care Implementation Authorship Group, Girgis A, (2022) Implementing patient reported outcomes into routine care: an audit of our cancer patient population to inform their capabilities and preferences Aust Health Rev
South Western Sydney Local Health District (2018) 2018–2021 Midpoint Review. Strategic Plan: leading care, healthier communities. South Western Sydney Local Health District, New South Wales
Girgis A, Durcinoska I, Gerges M, Kaadan N, Arnold A, Descallar J, Delaney GP, Avery S, Carolan M, Della-Fiorentina S, Masters K, Miller A, Ng W, Sandell T, Tran TT, Levesque JV, PROMPT-Care Program Group (2018) Study protocol for a controlled trial of an eHealth system utilising patient reported outcome measures for personalised treatment and care: PROMPT-Care 2.0. BMC Cancer 18(1):845. https://doi.org/10.1186/s12885-018-4729-3
Riba MB, Donovan KA, Andersen B, Braun I, Breitbart WS, Brewer BW, Buchmann LO, Clark MM, Collins M, Corbett C (2019) Distress management, version 3.2019, NCCN clinical practice guidelines in oncology. J Natl Compr Canc Netw 17(10):1229–1249
Richardson L, Jones G (2009) A review of the reliability and validity of the Edmonton symptom assessment system. Curr Oncol 16(1):53–64
Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, Ory MG, Estabrooks PA (2019) RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health 7:64. https://doi.org/10.3389/fpubh.2019.00064
Bamgboje-Ayodele A, Avery S, Pearson J, Mak M, Smith K, Rincones O, Vinod S, Bray V, Ducinoska I, McCarthy K, Williamson K, Delaney GP, the PROMPT-Care Implementation Authorship Group, Girgis A (2022) Adapting an integrated care pathway for implementing electronic patient reported outcomes assessment in routine oncology care: Lessons learned from a case study. J Eval Clin Pract: 1–12. doi:https://doi.org/10.1111/jep.13688
Shea C, Jacobs S, Esserman D, Bruce K, Weiner B (2014) Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci. https://doi.org/10.1186/1748-5908-9-7
Storkholm MH, Mazzocato P, Tessma MK, Savage C (2018) Assessing the reliability and validity of the Danish version of organizational readiness for implementing change (ORIC). Implement Sci 13(1):1–7
Arnold BL, Halkett G, Dhillon H, Girgis A (2021) Do radiation therapists feel able to routinely screen for symptoms and distress in people with cancer: barriers impacting practice. J Med Rad Sci. https://doi.org/10.1002/jmrs.465
Geerligs L, Rankin NM, Shepherd HL, Butow P (2018) Hospital-based interventions: a systematic review of staff-reported barriers and facilitators to implementation processes. Implement Sci 13(1):1–17
Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psycho 3(2):77–101
Bouazza YB, Chiairi I, El Kharbouchi O, De Backer L, Vanhoutte G, Janssens A, Van Meerbeeck JP (2017) Patient-reported outcome measures (PROMs) in the management of lung cancer: a systematic review. Lung Cancer 113(1):140–151. https://doi.org/10.1016/j.lungcan.2017.09.011
Brownson RC, Kumanyika SK, Kreuter MW, Haire-Joshu D (2021) Implementation science should give higher priority to health equity. Implement Sci 16(1):28. https://doi.org/10.1186/s13012-021-01097-0
Mitchell AJ, Vahabzadeh A, Magruder K (2011) Screening for distress and depression in cancer settings: 10 lessons from 40 years of primary-care research. Psychooncology 20(6):572–584
Weiner BJ, Lewis MA, Linnan LA (2009) Using organization theory to understand the determinants of effective implementation of worksite health promotion programs. J Health Educ Res 24(2):292–305
Geerligs L, Shepherd HL, Butow P, Shaw J, Masya L, Cuddy J, The Adapt Program Group, Rankin NM (2021) What factors influence organisational readiness for change? Implementation of the Australian clinical pathway for the screening, assessment and management of anxiety and depression in adult cancer patients (ADAPT CP). Support Care Cancer 29(6):3235-3244. doi: https://doi.org/10.1007/s00520-020-05836-9
Stover AM, Tompkins Stricker C, Hammelef K, Henson S, Carr P, Jansen J, Deal AM, Bennett AV, Basch EM (2019) Using stakeholder engagement to overcome barriers to implementing patient-reported outcomes (PROs) in cancer care delivery: approaches from 3 prospective studies. Med Care 57(1):S92–S99. https://doi.org/10.1097/mlr.0000000000001103
Palos GR, Suarez-Almazor ME (2021) Launching an electronic patient-reported outcomes initiative in real-time clinical practice. JNCI Monogr 2021(57):23–30. https://doi.org/10.1093/jncimonographs/lgab005
Sisodia RC, Dankers C, Orav J, Joseph B, Meyers P, Wright P, Amand DS, Del Carmen M, Ferris T, Heng M (2020) Factors associated with increased collection of patient-reported outcomes within a large health care system. JAMA Netw Open 3(4):e202764–e202764
Sundberg K, Eklöf AL, Blomberg K, Isaksson A-K, Wengström Y (2015) Feasibility of an interactive ICT-platform for early assessment and management of patient-reported symptoms during radiotherapy for prostate cancer. Eur J Oncol Nur 19(5):523–528. https://doi.org/10.1016/j.ejon.2015.02.013
Monterosso L, Platt V, Bulsara M, Berg M (2019) Systematic review and meta-analysis of patient reported outcomes for nurse-led models of survivorship care for adult cancer patients. Cancer Treat Rev 73(1):62–72
Conway A, O’Donnell C, Yates P (2017) The effectiveness of the nurse care coordinator role on patient-reported and health service outcomes: a systematic review. Eval Health Prof 42(3):263–296. https://doi.org/10.1177/0163278717734610
Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, Rogak L, Bennett AV, Dueck AC, Atkinson TM (2016) Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 34(6):557
Krusche M, Klemm P, Grahammer M, Mucke J, Vossen D, Kleyer A, Sewerin P, Knitza J (2020) Acceptance, usage, and barriers of electronic patient-reported outcomes among German rheumatologists: survey study. JMIR Mhealth Uhealth 8(7):e18117
Courtenay M, Carey N, Stenner K, Lawton S, Peters J (2011) Patients’ views of nurse prescribing: effects on care, concordance and medicine taking. Br J Dermatol 164(2):396–401. https://doi.org/10.1111/j.1365-2133.2010.10119.x
Nic Giolla Easpaig B, Tran Y, Bierbaum M, Arnolda G, Delaney GP, Liauw W, Ward RL, Olver I, Currow D, Girgis A, Durcinoska I, Braithwaite J (2020) What are the attitudes of health professionals regarding patient reported outcome measures (PROMs) in oncology practice? A mixed-method synthesis of the qualitative evidence. BMC Health Serv Res 20(1):102. https://doi.org/10.1186/s12913-020-4939-7
Ahmed S, Zidarov D, Eilayyan O, Visca R (2020) Prospective application of implementation science theories and frameworks to inform use of PROMs in routine clinical care within an integrated pain network. Qual Life Res. https://doi.org/10.1007/s11136-020-02600-8
Singh AP, Berman AT, Marmarelis ME, Haas AR, Feigenberg SJ, Braun J, Ciunci CA, Bauml JM, Cohen RB, Kucharczuk JC, Shulman LN, Langer CJ, Aggarwal C (2020) Management of lung cancer during the COVID-19 pandemic. JCO Oncol Pract 16(9):579–586. https://doi.org/10.1200/OP.20.00286
In addition to the named authors, the PROMPT-Care Implementation Authorship Group also includes: Chee Fon Chang, Bianka Eifler, Sarah Elliott, Christine Hardy, Beth Ivimey, William Jansens, Nasreen Kaadan, Eng-Siew Koh, Nic Livio, Susan Lozenkovski, Gemma McErlean, Elias Nasser, Nicola Ryan, Therese Smeal, Tien Thomas, Thomas Tran and Jennifer Wiltshire. We thank the PRM team at the Cancer Institute NSW (CINSW), including Karina McCarthy, Kimberley Williamson, Shelley Rushton, Cynthia Lean, Sandra Leon and Nina Klug, for their extensive support with adapting our ICP to incorporate their state-wide CINSW PRM system.
This study was funded by the Centre for Oncology Education and Research Translation (CONCERT) through the CONCERT Seed Translational Grant Scheme. CONCERT was a cancer translational research group funded by the Cancer Institute New South Wales. Neither CONCERT nor Cancer Institute New South Wales influenced the design, data collection, analysis, or dissemination of results. AG, ABS and SA were funded through Cancer Institute New South Wales grants at the time the research was undertaken. The lung cancer care coordinator resources for this trial were funded in part by Astra Zeneca. Astra Zeneca did not influence the design of the study, the analysis or this manuscript.
Ethics approval and consent to participate
Ethics and governance approval was received from the ethics committee of South Western Sydney Local Health District (SWSLHD), Ref. 2020/ETH01052. The study adhered to the principles of the Declaration of Helsinki and the Australian National Health and Medical Research Council National Statement on Ethical Conduct in Human Research.
Consent for publication
All authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Girgis, A., Bamgboje-Ayodele, A., Rincones, O. et al. Stepping into the real world: a mixed-methods evaluation of the implementation of electronic patient reported outcomes in routine lung cancer care. J Patient Rep Outcomes 6, 70 (2022). https://doi.org/10.1186/s41687-022-00475-6