Skip to main content
Journal of Patient-Reported Outcomes
Download PDF
Download ePub

APPROACH e-PROM system: a user-centered development and evaluation of an electronic patient-reported outcomes measurement system for management of coronary artery disease

Journal of Patient-Reported Outcomes volume 8, Article number: 102 (2024) Cite this article

Abstract

Background

Coronary artery disease (CAD) confers increased risks of premature mortality, non-fatal morbidity, and significant impairment in functional status and health-related quality of life. Routine administration of electronic patient-reported outcome measures (PROMs) and its real time delivery to care providers is known to have the potential to inform routine cardiac care and to improve quality of care and patient outcomes. This study describes a user-centered development and evaluation of the Alberta Provincial Project for Outcomes Assessment (APPROACH) electronic Patient Reported Outcomes Measurement (e-PROM) system. This e-PROM system is an electronic system for the administration of PROMs to patients with CAD and the delivery of the summarized information to their care providers to facilitate patient-physician communication and shared decision-making. This electronic platform was designed to be accessible via web-based and hand-held devices. Heuristic and user acceptance evaluation were conducted with patients and attending care providers.

Results

The APPROACH e-PROM system was co-developed with patients and care providers, research investigators, informaticians and information technology experts. Five PROMs were selected for inclusion in the online platform after consultations with patient partners, care providers, and PROMs experts: the Seattle Angina Questionnaire, Patient Health Questionnaire, EuroQOL, and Medical Outcomes Study Social Support Survey, and Self-Care of Coronary Heart Disease Inventory. The heuristic evaluation was completed by four design experts who examined the usability of the prototype interfaces. User acceptance testing was completed with 13 patients and 10 cardiologists who evaluated prototype user interfaces of the e-PROM system.

Conclusion

Both patients and physicians found the APPROACH e-PROM system to be easy to use, understandable, and acceptable. The APPROACH e-PROM system provides a user-informed electronic platform designed to incorporate PROMs into the delivery of individualized cardiac care for persons with CAD.

Introduction

Heart disease, the second-leading cause of death in Canada, affects up to 8.5% of adults and accounts for an annual estimated cost of $21.2 billion [1, 2]. In addition to increased risks of premature mortality and non-fatal morbidity, coronary heart disease leads to significant symptoms and associated impairment in functional status and health-related quality of life [3, 4]. Effective patient-centered cardiovascular disease management should therefore include not only interventions that reduce mortality, but also aim to improve patients’ symptoms, functioning, and quality of life (QOL) [5].

There is growing evidence from the literature that the integration of patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) into routine care can improve health outcomes and experiences with care, better align health services delivery with individual needs, promote patient-provider communication and support decision-making [6,7,8,9,10,11,12]. Although there are well developed and validated PROMs and PREMs for cardiovascular disease, there is a lack of electronic tools for collecting and reporting these measures back to care providers and patients. Recent advances have focused on electronic administration of PROMs in several other populations, including cancer [13, 14], orthopedics [15,16,17], mental health [18–19], stroke [20], and nephrology [21, 22]. Electronic platforms may ease the deployment of PROMs and enhance the ability to communicate information to patients and health care providers. In some settings, electronic PROMs/PREMs collection and reporting systems have yielded improvements in patients’ QOL, satisfaction, and clinical outcomes [12,13,14,15].

The Alberta Provincial Project on Outcome Assessment in Coronary Heart Disease (APPROACH) maintains one of the most comprehensive data repositories of CAD management globally [23]. It contains rich clinical information and outcomes for more than 300,000 patients who received cardiac care in hospitals in the Province of Alberta, Canada, since its inception in 1995 [24]. This registry has also collected patient reported information, including PROMs, using the 19-item Seattle Angina Questionnaire (SAQ) [25], the 5-item EuroQOL (EQ-5D) [26], the 18-item Medical Outcomes Study Social Support Survey (MOS-SSS) [27], and the 14-item Hospital Anxiety and Depression Scale (HADS) [28]. These PROMs have historically been mailed to patients with return postage within 1 week of their first coronary angiogram and at 1-, 3- and 5-years post angiogram. The completed paper-based PROMs are then scanned and uploaded into APPROACH. These PROMs data have been used in aggregate for research and reporting purposes. However, PROMs have not been deployed in routine cardiac care to facilitate communication with patients and promote shared decision making. Additionally, the financial burden of administering the paper-based PROMs is not sustainable. Consequently, we explored the transition of the PROMs data collection into an electronic format in which patients remotely self-report their health sta

tus and QOL via an electronic platform, and physicians receive the summarized information to support CAD management at the point of care.

The APPROACH electronic PROM system was developed in collaboration with patient research partners, cardiac care providers, heuristic and usability experts from the University of Calgary’s Ward of the 21st Century (W21C), Cohesic Inc. (an Alberta-based Information Technology vendor), and the Clinical Research Unit at the University of Calgary, which currently maintains the APPROACH clinical information system and patient-level registry data. In this study, we describe the development and usability evaluations of the APPROACH e-PROM system.

Methods

Stakeholders engagement

In September 2018, the APPROACH research committee, in collaboration with the Libin Cardiovascular Institute of Alberta’s patient advisory group, held an interdisciplinary workshop that focused on the use of PROMs and PREMs in cardiovascular care. This was organized to understand stakeholders’ perspectives about the relevance of PROMs, the appropriateness of the existing paper-based PROMs, and feasibility of migration of the paper-based administration to electronic administration format. Participants included 12 adults with lived experience with heart disease recruited from the APPROACH registry, cardiologists, decision-makers from Alberta’s Departments of Cardiac Sciences, and researchers with expertise in patient engagement and use of PROMs/PREMs in clinical care. Three small group sessions were held with representatives from each stakeholder group to identify gaps, research priorities, and potential solutions addressing the use of PROMs/PREMs information encountered in cardiovascular care. Conversations from these sessions were documented by designated research assistants who took notes from these sessions.

Selection of PROMs for the APPROACH e-PROM system

A central aspect of the development of the e-PROM system is the selection of appropriate PROMs. We convened a working group composed of patient partners (NV, CW, WP, and PE), project investigators/ clinicians/cardiovascular researchers (CMN, MMG, SBW) and methodologists (TTS, RS), to review the appropriateness of paper-based PROMs for electronic deployment and use in routine clinical care. The working group developed recommendations for the set of PROMs to be implemented in electronic format. Factors considered included the type of outcome to be measured, the relevance of PROMs for assessing each outcome, the psychometric properties (validity, reliability, responsiveness to change, etc.) of each PROM, the burden on patient respondents to complete the instrument, and feasibility of integration of the results into clinician work-flow processes at the point of care.

Technical development of the APPROACH e-PROM prototype

The APPROACH e-PROM system was designed in partnership with Cohesic Inc, to be a web-based electronic system accessible to patients via desktop and hand-held devices (i.e., tablets and smartphones). Other unique functionalities include inbuilt reminder notifications sent to patients to report outcomes at designated times, the capacity to administer multiple PROMs, and real-time delivery of reports to attending physicians in the form of an automatically configured 2-page report that summarizes the collected PROMs data on each patient. The initial prototype of this e-PROM system was developed and reviewed by the team of investigators and patient advisors.

Heuristic evaluation of the APPROACH e-PROM system

Heuristic evaluation is a methodology for identifying design-related problems in interactive and dynamic human-computer interfaces by judging the interface’s design against a set of established standard guidelines [29]. It aims to identify design problems with user interfaces with the ultimate goal of making electronic health interventions easy to navigate for users. The heuristic evaluation of the APPROACH e-PROM system was conducted in collaboration with the University of Calgary W21C. Four heuristic evaluators, with expertise in user experience, user-centered design, data/information visualization, and human factors analysis were recruited through the network of experts available to W21C to evaluate the initial e-PROM system design features. Heuristic evaluation of the interactive patient-facing portal was structured around Nielsen’s principles for user-centered design [30]. The physician-facing static summary report prototypes were evaluated using Kelleher and Wagener’s principles of data visualization [31]. Each evaluator completed the heuristic evaluations independently and reported findings and recommendations in standardized spreadsheets. Once all evaluations were complete, a debriefing meeting was conducted for each evaluator to share their findings and answer any outstanding questions as needed. All findings and recommendations were collated into a single spreadsheet to inform the next design iteration of the e-PROM system.

User acceptance evaluation of the APPROACH e-PROM system

Usability refers to “the extent to which a product can be used by specified user to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [32]. The acceptability of the e-PROM system to patients and physicians across web-based (personal computer) and hand-held devices (mobile phones, and tablets) was evaluated using user acceptance evaluations [33,34,35]. Patients with lived experience of CAD and those who may have previously completed the paper-based survey of the PROMs but had no prior experience with the e-PROM system were recruited to evaluate this tool. Eligible patient participants were recruited through our network of patient advisors and through the database APPROACH registry participants. Patient participants first completed a brief demographic questionnaire then completed the PROMs questions on the prototype of the e-PROM system either on a personal computer, tablet, or mobile phone. The participants’ confidence in completing assigned tasks on an electronic device was quantitatively scored using a single item 7-point Likert scale with response categories that range from 1(Not at all confident) to 7 (Very confident). Prompted by a standardized script, patients completed the e-tool while thinking aloud about their thoughts and actions. All sessions were recorded with permission. A researcher reviewed the sessions to make notes on patient verbal feedback, observed errors, or observed areas of difficulty. Notes were thematically analyzed to reveal insights on tool usability and barriers to use.

The physician-facing summarized PROMs report prototype was evaluated based on user acceptance and comprehension testing. The acceptability of the PROM summary report was conducted with 10 cardiologists involved in CAD management, recruited through the network of APPROACH clinician investigators. Using a standardized script, physicians were instructed to evaluate the e-PROM summarized report prototype while thinking aloud about their thoughts and actions.

The comprehensibility of the summary report was evaluated by asking the participants a set of questions designed to determine if they could reliably extract information from the report, and if the information extracted was correct. Three prototype summary PROMs reports were created. The comprehension test consisted of 30-minute one-on-one sessions with each participant, designed to solicit qualitative feedback, and to provide quantitative assessment of the ability of physicians to comprehend the data output of the tool. See page Appendix A1 for more details about the methodology to assess comprehensibility of the report.

All usability sessions were audio/video recorded for analysis. All issues encountered and feedback provided were noted by a research associate, qualitatively analyzed for themes, and summarized to provide recommendations for improvements to the report design.

Results

Stakeholders engagement

The interdisciplinary workshop with the stakeholders yielded important feedback with the following major themes. First, almost all participants agreed that the collection of PROMs is valuable and its integration into routine clinical care is central to promoting the delivery of patient-centered cardiac care to patients with CAD. Second, patient advisors identified individualized PROMs data used to inform decisions about disease management as a key priority. The delivery of a summarized PROMs report to patients and their care providers was deemed essential for a truly patient-centered care. Patient participants noted that existing PROMs did not capture some important aspects of patients’ quality of life. In particular, questions related to living alone, sexual function, sociodemographic and lifestyle factors were identified as important factors that may contextualize PROM results for attending cardiologists and facilitate patient-physician shared decision-making. Finally, the electronic formats for assessment and reporting of PROMs and PREMs were broadly acceptable and desirable to patients.

Selection of PROMs for APPROACH e-PROM system

The PROMs administered via paper-based survey were considered too burdensome for some patients since it involved completing a total of 56 items from the four PROMs, in addition to other sociodemographic questions. Consequently, shorter versions of the PROMs were recommended to be adopted in the electronic system. For example, the 7-item SAQ [39, 40] and 8-item MOS-SS [41] scales were recommended as shorter versions of the 19-item SAQ and 18-item MOS-SS to achieve the right balance between minimizing respondent burden and capturing relevant PROMs information. Second, the HADS scale is not a sensitive measure for screening for depression in CAD populations [36]. It was recommended that this be replaced with the 9-item Patient-Health Questionnaire (PHQ-9) [37], as it has higher sensitivity and specificity values and positive likelihood ratio for screening for depression in CAD [37, 38]. Each item of the PHQ-9 is scored on a 4-point Likert scale that range from 0 (Not at all) to 3 (nearly every day), yielding a total score ranging from 0 to 27 with higher scores indicating more severe depression. In administering this instrument adopted a 2-step process for screening for depression using the PHQ instrument. In the first step, the 2-item PHQ (PHQ-2) was first administered to screen for depression while the full PHQ-9 is administered to those screen positive on the shorter version in the second step.Third, lifestyle self-management behaviors such as smoking, diet, exercise, and cardiac rehabilitation were recommended to be included in the form of 9 items relating to self-management on the Self-Care of Coronary Heart Disease Inventory (SC-CHDI) [42]. Finally, to facilitate the uptake of the summarized results, it was recommended that a two-page summarized PROM results should be presented with a high-level summary presented on the first and more granular details on each measure (e.g. subscale results on the second page). In summary, the 7-item SAQ, the 8-item MOS-SSS, PHQ-9, EQ-5D, and the SC-CHDI were adopted as the primary standardized PROMs for inclusion in the system.

Heuristic evaluation

The four heuristic evaluators considered the e-PROM patient-facing portal to be well designed with relatively minor issues identified. Notable issues included; (1) the design of the progress bar, which did not help users gauge how far along they had progressed toward the end of the tool and how much was left to be completed; (2) the content of the survey questions (i.e., wording comprehensibility and consistency, spelling and grammar); (3) user control and freedom related to stopping and starting the survey; and, (4) issues related to navigation (i.e., forward and back) and response selection. The suggested adjustments were reviewed by our team of investigators and implemented by the technical team prior to beginning usability testing with patient participants.

For the summarized PROMs report, the evaluators identified issues with; (1) numerical placeholders, icons, legend design; (2) use of stacked bar charts for data visualization; (3) color coding of the trend icons, and other color choices; and (4) redundancy of information, and layout. One proposed solution to the identified issues included the use of a line graph with strip lines. This proposed solution supports the same tasks as the current designs, including learning the patient’s current and previous scores, understanding how the patient’s score has changed over time, understanding if the patient’s score is within a target reference range, and understanding if the patient’s score change is clinically significant. The APPROACH e-PROM system was revised based on these proposed changes.

User acceptance evaluation

A total of 12 patient participants (5 female, 7 male) were recruited to evaluate the patient-facing e-PROM system across all devices. Of these, 5 completed the evaluation on their mobile phones, 5 on desktop computers, and 2 used tablets (Table 1). Participant’s ages ranged between 37 and 79. The average confidence rating (in easily completing tasks on an electronic device) is 6.0. Patient evaluators generally found the e-PROM system acceptable for its intended purpose Several participants also expressed appreciation for the thoroughness of the survey, and in particular the inclusion of the emotion-related questions:

Table 1 Distribution of patient participants’ characteristics for user acceptance testing of the APPROACH e-PROM system
Full size table

I really liked that on here you’re asking about the emotional side, that’s really good. I don’t know what you are doing about it, but somebody needs to ask.

I’ve filled out lots of surveys and none have asked questions like that.

It’s never any questions that a doctor would have asked me.

Feedback for refinement of the e-PROM system included technology-related concerns (in particular, text visibility issues on the mobile platform), physical implications (speculation that a patient lacking motor control or cognitive ability may have difficulty using the tool), as well as psychological considerations (including emotional state, particularly after depression screening questions, the ability to recall past experiences in order to accurately answer given questions, the desire for more system transparency when ending one survey and beginning the next, and overall willingness to complete the activity).

Some participants remained skeptical that their responses to the PROMs would be seen or used by the care providers in any sort of meaningful way. One participated stated:

You need to be in very rough shape to get the attention of your cardiologist… Waiting 14–15 months between appointments can be stressful, especially if you’re worried you have a problem.

Furthermore, some patient participants were unsure if their responses to questions about physical functioning should be based solely on their CAD symptoms or if their responses should include restrictions due to other physical comorbid conditions (e.g., arthritis). Participants suggested adding more specific instructions for how to respond to the PROM questions when other comorbid conditions affect their response(s). After reviewing this feedback, the APPROACH e-PROM system prototype was refined to include preamble information that prompts patients to interpret and respond the PROM questions in the context of their CAD disease. Figure 1 presents an example screenshot of the patient-facing interface of the APPROACH e-PROM for completing the PROMs.

Fig. 1
figure 1

APPROACH e-PROM System patient user interface

Full size image

The physician-facing PROMs summary report template was evaluated by 10 practicing cardiologists (3 females, 7 males). Physician ages ranged from 38 to 63 (average age 51), with an average of 17.4 years in cardiology practice and an average of 78.5% of time spent on clinical practice.

Participants reliably extracted correct information from the reports in nearly all cases (Table 2). The only notable errors were a mistake interpreting “below the 25th percentile” to mean a positive condition instead of a negative one; and some participants failed to notice supplemental information contained on the third and final page of some report prototypes (some reports only had two pages). Once familiarity is established with the report format, this is not anticipated to be an issue.

Table 2 Proportion (%) of physician participants who successfully interpret APPROACH e-PROM summary report
Full size table

Overall, the physicians had positive perceptions about the report with almost all respondents perceiving it as easy to understand. Several physicians lauded the visual appeal, its potential to provide an objective and quantitative assessment, especially for subjective symptoms, and its ability to provide an effective snapshot of a patient’s current condition and illness experience. The report was noted as a good tool to start a discussion with the patient, a “good communication tool”, and a helpful primer to prepare the physician to effectively direct the line of questioning during a consultation. The following are some illustrative quotes from the participants:

…the more information we have upfront that the patient is able to provide, the better job we can do.

I think this is a great tool and I would use it with every patient.

Visually it is attractive and easy to interpret.

I think it is pretty close to perfect.

This is wonderful. I would use it and I would encourage all my colleagues to use it. I think it’s a great time saver, and it also helps you establish that relationship with the patient.

The fact that this electronic system includes assessment of depression symptoms is very helpful. It does an assessment and gives you a recommendation in an area that we’re not as comfortable. That’s very useful.

Potential concerns raised by the physician participants included:

  1. 1.

    The potential to negatively affect the patient/physician relationship by partially replacing part of a traditional patient/physician encounter with an automated survey.

  2. 2.

    Increasing patient frustration by having them answer the same questions multiple times (i.e., in the survey, again for a resident or fellow, and again for the cardiologist).

  3. 3.

    Patients may be unwilling or unable to complete the lengthy assessment due to limitations with technical literacy, reading ability, language barriers, or physical impairments (blindness, lack of motor control, etc.).

  4. 4.

    Patients may be willing to complete the assessment once but may run out of patience if asked to complete it multiple times in a relatively short time interval (i.e., every six months).

Physician participants also expressed low confidence in interpreting the PHQ-9 measure, which was attributed to the ambiguity of the direction on how to manage or intervene on patients with mild depression (i.e., 5 ≤ PHQ-9 score ≤ 9). For example, the PHQ-9 scoring system recommends “watchful waiting” and/or repeated administration of the PHQ-9 at a follow-up visit for patients with mild depression. One participant stated:

I’m not sure if cardiologists are going to like that… it’s basically asking us to do more screening into their mental health status and I’m not sure that people feel comfortable doing that. It’s good to know they’re at risk, but they may not be seeing me again for a long time, nor am I the best person to assess or treat… That [sentence] leaves me feeling like I don’t know what I’m supposed to do.

To address this issue, we embedded additional information on relevant mental health and social support services, to which patients may be referred, in the summarized PROMs report to support physician decision-making. In Fig. 2, 3 and 4, we present screenshots from the final version of the summarized PROMs report after heuristic and user-acceptance testing.

Fig. 2
figure 2

Physician-facing APPROACH e-PROMs Summary Report illustrating presentation of summarized information from the four PROMs instruments (First page). NB SAQ-7 7-item Seattle Angina Questionnaire; PHQ-2 2-item Patient Health Questionnaire; EQ-5D 5-item EuroQOL; SSS 8-item Medical Outcomes Study Social Support Scale

Full size image
Fig. 3
figure 3

Physician-facing APPROACH e-PROMs Summary Report Illustrating Presentation of Information from the Subscales of the SAQ-7 (Second page)

Full size image
Fig. 4
figure 4

Physician-facing APPROACH e-PROMs Summary Report Illustrating Presentation of Information from the Subscales of the SAQ-7 (Third page)

Full size image

Discussion

The collection and integration of PROMs into routine clinical care to support clinical decisions has been identified as a key priority for effective cardiovascular disease management [43–44]. In this study, we describe the development and usability evaluation of the APPROACH e-PROM system for the collection of patient-reported outcomes in patients with CAD and the delivery of the summarized PROMs information to attending cardiologists to inform patient-care provider discussions during routine visits. Our study findings reveal that the APPROACH e-PROM system was perceived as easy to use, understandable, and acceptable to both patients and physicians. Although several electronic PROMs data capture systems have been implemented in routine cardiac care [45,46,47,48], the APPROACH e-PROM system is the first, to our knowledge, to explore the co-development of an e-PROM system with patients and to evaluate its usability for individuals with CAD.

An initial concern was the acceptability of an electronic system for completing PROMs in individuals with CAD, who are predominantly older adults. This is particularly important considering the electronic divide between older and younger populations. Our consultations and meeting with patients and members of the patient advisory group, revealed an overwhelming enthusiasm for electronic completion. This was further confirmed with the results of the usability evaluation, where patients rated the e-PROM system as acceptable and were confident in accessing and navigating its features from their electronic devices. However, we acknowledge that the number of patients included in the study was small, and the patients who participated in this study may not be representative of other patients encountered in clinical practice. Further research will be required to evaluate implementation and uptake of this e-PROM system among more diverse groups of people with CAD.

A recurring theme that emerged was the recommendation to deliver the PROM summary information to both the patient and the physician. There was hesitation about the feasibility of getting physicians to use PROMs results to enhance communication and shared decision making at the point of care, considering physicians’ busy workload. Delivering summarized PROMs s results directly to patients might further empower them to share concerns about their results with their cardiologists. This is consistent with evidence from the literature on the effectiveness of facilitated relay of information from patients to their health care providers [49]. Studies examining the effects of PROMs feedback have also shown improvements to decision-making, patient satisfaction, processes of care and some benefits to patients’ health outcomes [50,51,52].

Another notable finding is physicians’ limited knowledge on how to act on PROMs results relating to non-cardiac concerns such as mental health. To address this important barrier to meaningful use of PROMs in clinical care, we developed supplementary material in the form of guidance and links to resources for mental health problems and social support services embedded in the report. We have also developed an education program (providing continuing medical education credits) to encourage and support physicians to act on the PROMs results. This will be essential for the successful integration of PROMs into routine cardiac care.

The strengths of this study include its collaborative approach with patients, care providers, and researchers with interdisciplinary expertise who contributed to the user-centered design and development process of this e-PROM system. The demographic characteristics of our patient participants are representative of the general population of individuals living with CAD, confirming the generalizability of our study findings. The input from stakeholders, including patients, patient partners, care providers, cardiovascular researchers, methodologists, and user design and informatics expert were iteratively incorporated across the stages of design and development leading to iterative improvements in the prototypes evaluated in each step. We believe this iterative approach to design and development of the system improved the acceptability of the final system. A unique feature of this e-PROM system is the real-time delivery of summarized PROMs information to care providers at the point of care, in response to the recommendations from patient advisors about how PROMs should be used in the future, with a strong desire to see them used to inform decisions about their disease management and the need for the integration of the summarized information into clinical care. Evidence from previous literature reveals that the integration of PROMs in routine clinical practice has the potential to improve patient-care provider shared decision-making, disease management, and patients’ overall satisfaction with cardiovascular care [6,7,8,9,10]. The development of this e-PROM system will provide an opportunity for implementation of PROMS within clinical practice in Alberta to facilitate patient-care provider discussions and support clinical decisions about coronary heart disease management.

This study also has limitations. The development and evaluation of this APPROACH e-PROM system was impacted by the COVID-19 pandemic; most activities for this project were completed remotely due to the public health restrictions in place. This affected the recruitment of patients and physicians for in-person user acceptability evaluation of the APPROACH e-PROM system. Consequently, some of the patients completed their evaluation of the system remotely. Furthermore, our recruitment of patient participants for this study was limited to predominantly English-speaking white population. Our initial intension to ensure diversity of participants was severely inhibited as the COVID-19 pandemic disproportionately affected equity-deserving groups, such as racialized/visible minority groups and those from non-English speaking groups. Second, recruitment of 5 patients per electronic device was considered optimal for the usability evaluation of the e-PROM system user interfaces. This is at the lower end of the sample size recommended by the International Society for Pharmacoepidemiology and Outcomes Research [49]. Finally, the heuristic evaluation of the data visualization component of the APPROACH e-PROMs summarized report template using Kelleher and Wagener’s guidelines [31] proved to be difficult because this guideline was not applicable to the design of the data visualizations embedded within the PROMs summary report. Instead, we encouraged heuristic evaluators to provide their feedback and recommendations in a text file as appropriate.

Future research is required to evaluate the feasibility of implementing the APPROACH e-PROM system in selected cardiology clinics. The implementation of pilot study within clinical settings is needed to identify optimal approaches for broader prospective deployment of this e-PROM system into cardiac clinical care. Similarly, a randomized controlled trial is needed to evaluate its impact of an integrated e-PROM system in routine cardiac care on patient-physician communication, shared decision-making, processes and quality of cardiac care, and patients’ outcomes. This may be of particularly of interest to health services in jurisdictions where electronic clinical information systems exist, that can aid the collection and reporting of patient-reported data to health care providers. The APPROACH e-PROM system is designed with unique functionalities that allows for future seamless integration into electronic health record systems. Future research will evaluate the feasibility and the impact of this electronic system when deployed in routine clinical care across Canada to include diverse cultural and language groups. Additionally, while the initial goal for developing APPROACH e-PROM system is to improve individual patient and physician communication for optimal patient-centred CAD management, the ultimate goal of this initiative is to improve the processes of care and the quality of cardiac care delivery. Future research on e-PROMs systems should explore the potential effects on health system quality improvement and assurance initiatives. Finally, it is not clear to what extent are the psychometric properties to of the PROMs administered through the APPROACH e-PROM system is equivalent to the paper-based versions of these PROMs in the CAD population. Future psychometric investigations will be conducted to evaluate measurement invariance or equivalence of the PROMs administered digitally through APPROACH e-PROM system and the paper-based versions.

In conclusion, this study describes the development and evaluation of this APPROACH e-PROM system that is understandable and acceptable to patients and physicians and useful for routine collection PROMs and the integration of summarized PROMs information into cardiac care to inform communications between patients with coronary heart disease and their physicians. This feasibility of implementing this e-PROM system and its effectiveness within routine clinical care will be assessed in ongoing research.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

APPROACH:

Alberta Provincial Project for Outcome Assessment in Coronary Heart Disease Registry

CAD:

Coronary artery disease

e-PROM:

Electronic patient-reported outcomes measures

EQ-5D:

5-item EuroQOL questionnaire

PREMs:

Patient-reported experience measures

HADS:

Hospital Anxiety and Depression Scale

MOS-SSS:

Medical Outcome Study Social Support Survey

PROMs:

Patient-reported outcome measures

PHQ-9:

9-item Patient Health Questionnaire

PHQ-2:

2-item Patient Health Questionnaire

QOL:

Quality of Life

SAQ:

Seattle Angina Questionnaire

SC-CHDI:

Self-Care of Coronary Heart Disease Inventory

References

  1. Government of Canada (2019) Heart disease in Canada: highlights from the Canadian Chronic Disease Surveillance System. https://www.canada.ca/en/public-health/services/publications/diseases-conditions/heart-disease-canada-fact-sheet.html

  2. Virani SS, Alonso A, Aparicio HJ, Benjamin EJ, Bittencourt MS, Callaway CW, Carson AP, Chamberlain AM et al (2021) Heart Disease and Stroke Statistics-2021 update: a Report from the American Heart Association. Circulation 143(8):e254–e743

    Article  PubMed  Google Scholar 

  3. GBD 2015 Mortality and Causes of Death Collaborators (2016) Global, regional, and national life expectancy, all-cause mortality, and cause-specific mortality for 249 causes of death, 1980–2015: a systematic analysis for the global burden of Disease Study 2015. Lancet 388(10053):1459–1544

    Article  Google Scholar 

  4. Statistics Canada. Tables 102–0561– The ten leading causes of death (2012) CANSIM (death database) [Internet]. Ottawa (ON): Statistics Canada; 2015 Dec 10 [cited 2016 Sep 14]. www.statcan.gc.ca/pub/82-625-x/2015001/article/14296-eng.htm

  5. Spertus JA (2014) Barriers to the use of patient-reported outcomes in clinical care. Circulation: Cardiovasc Qual Outcomes 7:2–4

    Google Scholar 

  6. Rumsfeld JS, Alexander KP, Goff DC, Graham MM, Ho PM, Masoudi FA, Moser DK et al (2013) Cardiovascular health: the importance of measuring patient-reported health status: a scientific statement from the American Heart Association. Circulation 127:2233–2249

    Article  PubMed  Google Scholar 

  7. Boyce MB, Browne JP (2013) Does providing feedback on patient-reported outcomes to healthcare professionals result in better outcomes for patients? A systematic review. Qual Life Res 22:2265–2278

    Article  PubMed  Google Scholar 

  8. Kotronoulas G, Kearney N, Maguire R, Harrow A, Di Domenico D, Croy S, MacGillivray S (2014) What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol 32(14):1480–1501

    Article  PubMed  Google Scholar 

  9. Squitieri L, Bozic KJ, Pusic AL (2014) The role of patient-reported outcome measures in value-based payment reform. Value Health 20:834–836

    Article  Google Scholar 

  10. Nelson EC, Eftimovska E, Lind C, Hager A, Wasson JH, Lindblad S (2015) Patient reported outcome measures in practice. BMJ 350:g7818

    Article  PubMed  Google Scholar 

  11. Biber A, Ose J, Reese D, Gardiner J, Facelli A, Spuhl J, Brodke J (2017) Patient reported outcomes - experiences with implementation in a University Health Care setting. J Patient Rep Outcomes 2:34

    Article  PubMed  Google Scholar 

  12. Gibbons C, Porter I, Gonçalves-Bradley DC, Stoilov S, Ricci-Cabello I, Tsangaris E, Gangannagaripalli J et al (2021) Routine provision of feedback from patient-reported outcome measurements to healthcare providers and patients in clinical practice. Cochrane Database Syst Reviews 10https://doi.org/10.1002/14651858.CD011589.pub2

  13. Pérez-Alfonso KE, Sánchez-Martínez V (2021) Electronic patient-reported outcome measures evaluating cancer symptoms: a systematic review. Semin Oncol Nurs 37(2):151145

    Article  PubMed  Google Scholar 

  14. Howell D, Molloy S, Wilkinson K, Green E, Orchard K, Wang K, Liberty J (2015) Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Annals Oncol 26(9):1846–1858

    Article  CAS  Google Scholar 

  15. Rothrock NE, Bass E, Blumenthal A, Gershon RC, Hanson B, Joeris A, Kaat A et al (2009) AO patient outcomes center: design, implementation, and evaluation of a software application for the collection of patient-reported outcome measures in orthopedic outpatient clinics. J Med Internet Research: Formative Res 3(2):e10880

    Google Scholar 

  16. Marshall DA, Jin X, Pittman LB, Smith CJ (2021) The use of patient-reported outcome measures in hip and knee arthroplasty in Alberta. J Patient Rep Outcomes 5(Suppl 2):87

    Article  PubMed  PubMed Central  Google Scholar 

  17. Bernstein DN, Fear K, Mesfin A, Hammert WC, Mitten DJ, Rubery PT, Baumbauer JF (2019) Patient- reported outcomes use during orthopaedic surgery clinic visits improves the patient experience. Musculoskelet Care 17:120–125

    Article  Google Scholar 

  18. Kendrick T, El-Gohary M, Stuart B, Gilbody S, Churchill R, Aiken L, Bhattacharya A, Gimson A, Brütt AL, de Jong K, Moore M (2016) Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. Cochrane Database Syst Reviews Issue 7. Art. No.: CD011119.

  19. Krageloh CU, Phty CM, Billington DR, Kersten P, Siegert RJ (2015) Using feedback from patient- reported outcome measures in mental health services: a scoping study and topology. Psychiatric Serv 66:224–241

    Article  Google Scholar 

  20. Duncan PW, Abott RM, Rushing S, Johnson AM, Condon CN, Lycan SL, LutzBJ et al (2018) COMPASS- CP: an electronic application to capture patient-reported outcomes to develop actionable stroke and transient ischemic attack care plan. Circulation: Cardiovasc Qual Outcomes 11:e004444

  21. Davison SN, Klarenbach S, Manns B, Schnick-Makaroff K, Buzinski R, Corradetti B, Short H et al (2021) Patient-reported outcome measures in the care of in-centre hemodialysis patients. J Patient Rep Outcomes 5(Suppl 2):93

    Article  PubMed  PubMed Central  Google Scholar 

  22. Kyte D, Anderson N, Auti R, Aiyegbusi OL, Bishop J, Bissell A, Brettell E et al (2020) Development of an electronic patient-reported outcome measure (e-PROM) system to aid the management of patients with advanced chronic kidney disease. J Patient Rep Outcomes 4(1):55

    Article  PubMed  PubMed Central  Google Scholar 

  23. Ghali WA, Knudtson ML (2000) Overview of the Alberta provincial project for outcome assessment in coronary heart disease. Can J Cardiol 16:1225–1230

    CAS  PubMed  Google Scholar 

  24. Spertus JA, Winder JA, Dewhurst TA, Deyo RA, Prodzinski J, McDonell M, Fihn SD (1995) Development and evaluation of the Seattle Angina Questionnaire: a new functional status measure for coronary artery disease. J Am Coll Cardiol 25:333–341

    Article  CAS  PubMed  Google Scholar 

  25. Feng YS, Kohlman T, Janssen MF, Bucholz I (2021) Psychometric properties of the EQ-5D-5L: a systematic review of the literature. Qual Life Res 30:647–673

    Article  PubMed  Google Scholar 

  26. Sherbourne CD, Stewart AL (1991) The MOS social support survey. Soc Sci Med 32:705–714

    Article  CAS  PubMed  Google Scholar 

  27. Bjelland I, Dahl AA, Haug TT, Neckelmann D (2002) The validity of the hospital anxiety and Depression Scale: an updated literature review. J Psychosom Res 52:69–77

    Article  PubMed  Google Scholar 

  28. Norris CM, Saunders LD, Ghali WA, Brant R, Galbraith PD, Graham M, Faris P, Dzavik V, Knudtson ML (2000) Health-related quality of life outcomes of patients with coronary artery disease. Overview of the Alberta provincial project for outcome assessment in coronary heart disease. Can J Cardiol 16:1225–1230

    Google Scholar 

  29. Nielsen J, Molich R (1990) Heuristic evaluation of user interfaces, Proc. ACM CHI’90 Conf (Seattle, WA, 1–5 April), 249–256

  30. Nielsen 10 Usability Heuristics for User Interface Design, 19994

  31. Kelleher C, Wagener T (2011) Ten guidelines for effective data visualization in scientific publications. Environ Model Softw 26(6):822–827

    Article  Google Scholar 

  32. International Organization for Standardization (2013) Usability of consumer products and products for public use—part 2: summative test method. 20282–2:2013(en). I.T.

  33. Aiyegbusi OL (2020) Key methodological considerations for usability testing of electronic patient-reported outcome (ePRO) systems. Qual Life Res 29(2):325–333

    Article  PubMed  Google Scholar 

  34. Gordon S, Crager J, Howry C, Barsdorf AI, Cohen J, Crescioni M, Dahya B et al (2022) Best practice recommendations: user acceptance testing for systems designed to collect clinical outcome assessment data electronically, vol 56. Therapeutic Innovation & Regulatory Science, pp 442–453. 3

  35. Lewis JR (2018) The system usability scale: past, present, and future. Int J Human–Computer Interact 34(7):577–590

    Article  Google Scholar 

  36. Stafford L, Berk M, Jackson HJ (2007) Validity of the hospital anxiety and Depression scale and Patient Health Questionnaire-9 to screen for depression in patients with coronary artery disease. Gen Hosp Psychiatry 29(5):417–424

    Article  PubMed  Google Scholar 

  37. Kroenke K, Spitzer RL, Williams JB (2001) The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 16(9):606–613

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Haddad M, Walters P, Phillips R, Tsakok J, Williams P, Mann A, Tylee A (2013) Detecting depression in patients with coronary heart disease: a diagnostic evaluation of the PHQ-9 and HADS-D in primary care, findings from the UPBEAT-UK study. PLoS ONE 8(10):e78493

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Jones P, Chan P, Gosch K, Li Y, Reid K, Tang F, Spertus J (2013) The SAQ-7: A short version of the Seattle Angina Questionnaire. Circulation: Cardiovascular Quality and Outcomes.;6:A54

  40. Chan PS, Jones PG, Arnold SA, Spertus JA (2014) Development and validation of a short version of the Seattle angina questionnaire. Circulation: Cardiovasc Qual Outcomes 7(5):640–647

    Google Scholar 

  41. Moser A, Stuck AE, Silliman RA, Ganz PA, Clough-Gorr KM (2012) The eight-item modified Medical outcomes Study Social Support Survey: psychometric evaluation showed excellent performance. J Clin Epidemiol 65(10):1107–1116

    Article  PubMed  PubMed Central  Google Scholar 

  42. Vaughan Dickson V, Lee CS, Yehle KS, Mola A, Faulkner KM, Riegel B (2017) Psychometric testing of the self-care of Coronary Heart Disease Inventory (SC-CHDI). Res Nurs Health 40(1):15–22

    Article  PubMed  Google Scholar 

  43. Saeed S, Skaar E, Romarheim A, Chambers JB, Bleie Ø (2022) Shared decision-making and patient-reported outcome measures in Valvular Heart Disease. Front Cardiovasc Med 9:863040

    Article  PubMed  PubMed Central  Google Scholar 

  44. Damman OC, Jani A, de Jong BA, Becker A, Metz MJ, de Bruijne MC, Timmermans DR, Cornel MC, Ubbink DT, van der Steen M, Gray M, van El C (2020) The use of PROMs and shared decision-making in medical encounters with patients: an opportunity to deliver value-based health care to patients. J Eval Clin Pract 26(2):524–540

    Article  PubMed  Google Scholar 

  45. Stehlik J, Rodriguez-Correa C, Spertus JA, Biber J, Nativi-Nicolau J, Zickmund S, Steinberg BA, Peritz DC, Walker A, Hess J, Drakos SG, Kfoury AG, Fang JC, Selzman CH, Hess R (2017) Implementation of real-time assessment of patient-reported outcomes in a heart failure clinic: a feasibility study. J Card Fail 23(11):813–816

    Article  PubMed  Google Scholar 

  46. Pennucci F, De Rosis S, Passino C (2020) Piloting a web-based systematic collection and reporting of patient-reported outcome measures and patient-reported experience measures in chronic heart failure. BMJ Open 10(10):e037754

    Article  PubMed  PubMed Central  Google Scholar 

  47. Blumenthal DM, Strom JB, Valsdottir LR, Howard SE, Wagle NW, Ho KKL, Horn DM, O’Keefe SM, Wasfy JH, Metlay JP, Yeh RW (2018) Patient-reported outcomes in cardiology: comparison of two programs to assess angina burden in coronary artery disease. Circulation: Cardiovasc Qual Outcomes 11(11):e004794

    Google Scholar 

  48. Bliven BD, Kaufman S, Spertus J (2001) Electronic collection of health-related quality of life data: validity, time benefits, and patient preference. Qual Life Res 10(1):15–22

    Article  CAS  PubMed  Google Scholar 

  49. Mowlem FD, Elash CA, Dumais KM, Haenel E, O’Donohoe P, Olt J, Kalpadakis-Smith AV, James B, Balestrieri G, Becker K, Newara MC, Kern S, Electronic Clinical Outcome Assessment Consortium (2024) Best practices for the electronic implementation and Migration of patient-reported outcome measures. Value Health 27(1):79–94

    Article  PubMed  Google Scholar 

  50. Silveira Bianchim M, Crane E, Jones A, Neukirchinger B, Roberts G, Mclaughlin L, Noyes J (2023) The implementation, use and impact of patient reported outcome measures in value-based healthcare programmes: a scoping review. PLoS ONE 18(12):e0290976

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Meirte J, Hellemans N, Anthonissen M, Denteneer L, Maertens K, Moortgat P, Van Daele U (2020) Benefits and disadvantages of electronic patient-reported outcome measures: systematic review. JMIR Perioper Med 3(1):e15588

    Article  PubMed  PubMed Central  Google Scholar 

  52. Watson L, Delure A, Qi S, Link C, Chmielewski L, Photitai É et al (2021) Utilizing patient reported outcome measures (PROMs) in ambulatory oncology in Alberta: electronic reporting at the micro, meso and macro level. J Patient Rep Outcomes 5(Suppl 2):97

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We are grateful for the contributions of patients and physician participants in this study.

Funding

This study was supported by funding from the Canadian Institute of Health Research Strategy for Patient Oriented Research Catalyst and Multiyear Innovative Trials Grants and the Libin Cardiovascular Institute of Alberta.

Author information

Authors and Affiliations

  1. Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Canada

    Andrew Roberts, Peter Edgar & Michelle M. Graham

  2. Department of Medicine, University of Calgary, Calgary, Canada

    Eleanor Benterud & Matthew T. James

  3. Libin Cardiovascular Institute of Alberta, University of Calgary, Calgary, Canada

    Eleanor Benterud, Pantea Javaheri, William A. Ghali, Matthew T. James, Stephen B. Wilton & Tolulope T. Sajobi

  4. Department of Community Health Sciences, University of Calgary, 3280 Hospital Drive NW Calgary, Calgary, T4B 4B2, Canada

    Maria J. Santana, Nancy Verdin, Winnie Pearson, Joel Adekanye, Sandra Zelinsky, Jeff Caird, William A. Ghali, Matthew T. James & Tolulope T. Sajobi

  5. Cohesic Inc, Calgary, Canada

    Jordan Engbers & Christine Lorenz

  6. Ward of the 21st Century, University of Calgary, Calgary, Canada

    Courtney E. MacDonald & Sarah Simmons

  7. School of Nursing, Trinity Western University, Langley, BC, Canada

    Rick Sawatzky

  8. Department of Cardiac Sciences, University of Calgary, Calgary, Canada

    Bryan Har & Stephen B. Wilton

  9. Faculty of Nursing, University of Alberta, Edmonton, Canada

    Colleen M. Norris

Authors
  1. Andrew Roberts
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eleanor Benterud
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maria J. Santana
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jordan Engbers
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christine Lorenz
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nancy Verdin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Winnie Pearson
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Peter Edgar
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Joel Adekanye
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Pantea Javaheri
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Courtney E. MacDonald
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Sarah Simmons
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Sandra Zelinsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Jeff Caird
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Rick Sawatzky
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Bryan Har
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. William A. Ghali
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Colleen M. Norris
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Michelle M. Graham
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Matthew T. James
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Stephen B. Wilton
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Tolulope T. Sajobi
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

for the APPROACH investigators

Contributions

TTS, MMG, MTJ, CMN, MTJ, MJS and SBW conceived the study. CL and JDT oversaw the technical development of the APPROACH e-PROM system. AR coordinated the entire project and drafted the first version of the manuscript. SZ and MJS were involved in recruitment of patient participants and patient partners. JA, MMG, SBW, and BH were involved in recruitment of physician participants. WP, PE, NV, and CR were patient advisors who provided input into the design and development of the APPROACH e-PROM system. CEM and SS led the heuristic and user evaluations of the electronic system with patient and physician participants through the University of Calgary Ward of 21st Century. All authors critically reviewed and approved the final manuscript.

Corresponding author

Correspondence to Tolulope T. Sajobi.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the University of Calgary Conjoint Health Research Ethics Board (REB20-1907).

Competing interests

The authors have no competing interest to declare.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Roberts, A., Benterud, E., Santana, M.J. et al. APPROACH e-PROM system: a user-centered development and evaluation of an electronic patient-reported outcomes measurement system for management of coronary artery disease. J Patient Rep Outcomes 8, 102 (2024). https://doi.org/10.1186/s41687-024-00779-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s41687-024-00779-9

Keywords