- Short report
- Open Access
Patient reported outcomes – experiences with implementation in a University Health Care setting
Journal of Patient-Reported Outcomes volume 2, Article number: 34 (2018)
Patient-reported outcomes (PROs) have traditionally been implemented through a manual process of paper and pencil with little standardization throughout a Healthcare System. Each practice has asked patients specific questions to understand the patient’s health as it pertains to their specialty. These data were rarely shared and there has not been a comparison of patient’s health across different specialty domains. We sought to leverage interoperable electronic systems to provide a standardization of PRO assessments across sites of care.
University of Utah Health is comprised of four hospitals, 12 community clinics, over 400,000 unique annual patients, and more than 5000 providers. The enterprise wide implementation of PROs started in November of 2015. Patients can complete an assessment at home via email, or within the clinic on a tablet. Each specialty has the opportunity to add additional specialty-specific instruments. We customized the interval with which the patient answers the assessments based on specialty preference in order to minimize patient burden, while maximizing relevant data for clinicians.
Barriers and facilitators were identified in three phases: Pre-implementation, Implementation, and Post-implementation. Each phase was further broken down into technical challenges, content inclusion and exclusion, and organizational strategy. These phases are unique and require collaboration between several groups throughout the organization with support from executive leadership.
We are deploying system-wide standard and customized PRO collection with the goals of providing better patient care, improving physician-patient communication, and ultimately improving the value of the care given. Standardized assessment provides any clinician with information to quickly evaluate the overall, physical and mental health of a patient. This information is available real time to aid in patient communication for the clinician.
In recent years, there has been an increased interest in understanding the value and use of Patient Reported Outcomes (PROs) in clinical care [1,2,3,4]. PROs are directly reported by the patient, reflect the patient’s health-related quality of life and functional status, and provide insight into the patient’s perspective of their health [5,6,7]. The use of PROs can empower patients to take an active role in their care  and effectively communicate their health status [9, 10]. PRO scores can also be aggregated and used to evaluate the comparative effectiveness of treatment options [11, 12] and inform policy decisions [13, 14].
The adoption of PROs in healthcare systems remains limited . Technical and logistical challenges have limited deployment of PROs broadly within health care systems. At University of Utah Health, we are implementing the collection, reporting, and analytical use of PROs throughout the clinical enterprise. Currently, PROs collection is in place for about 75% of the ambulatory encounters within our system. Facilitators of this program include collaborating with our Information Technology (IT) team to create a home-grown application, integration of data into our Electronic Health Record (EHR), and provider champions.
Engaging patients and managing change in an organization that has four hospitals, 12 community clinics, over 400,000 unique patients on an annual basis, and more than 5000 practicing clinicians is difficult. In this paper, we describe our experiences implementing PROs including significant facilitators and barriers to PRO implementation that may benefit other academic medical centers in similar implementations.
Implementation of electronic PRO collection at University of Utah Health began in 2013 at the University Orthopaedic Center. In 2015, we began the health system-wide implementation our custom-designed platform, My Evaluation (mEVAL). We created our own platform to allow for customization and reduce vendor-imposed constraints. mEVAL is built within our Enterprise Data Warehouse with 3.0 total full-time equivalent (FTE) personnel allocated for maintenance and continued implementation. In 2016, we established the goal of reaching all ambulatory care practices by June 2018. We are currently collecting more than 15,000 assessments each month in 75 different practices ranging from primary care to sub-specialty surgical practices.
PROs are administered one of two ways:  ahead of the visit at home, via secure email link, or  in clinic on a tablet computer. In clinic, a staff member uses the mEVAL secure web-portal (Fig. 1), which includes every patient in our healthcare system, to generate an assessment, which is then completed by the patient on a tablet computer. The mEVAL portal is only used by the staff and does not require any login information from the patient. The mEVAL portal identifies patients eligible to complete PROs based on the patient schedule in the EHR and assigns the appropriate PRO instruments to the patient for completion based on the provider and visit department. If needed (e.g., the patient is a walk-in and not scheduled), the staff can also generate an ad hoc assessment for the patient.
In the EHR, clinicians are able to view patient-reported PRO information in summary scores, longitudinal graphs and individual questions and answers. These options allow each clinician to access the data they find most important at that time. Providers also have access to aggregated scoring data across their patients and can compare those data to their peers within the health system or against a national average (Fig. 2). We do not currently make the scores directly available to the patient because we feel it is important for the clinician to discuss how the scores are interpreted, but do offer the patient supplemental information (Additional file 1). We are developing and testing strategies to provide PRO scores directly to patients that include clear interpretation information.
Metrics of project success include the percentage of eligible patients who complete PROs at each practice and percentage of practices reaching the threshold of 80% of eligible patients completing PROs.
The study of the implementation of PROs was reviewed by the University of Utah Institutional Review Board, was deemed not to meet the definition of human subjects research, and was exempt from institutional review board oversight.
We assessed the barriers and facilitators for each of the three phases: pre-implementation, implementation, and post-implementation (Table 1). For each, we consider technical challenges, content, and organizational strategy.
Phase 1: Pre-implementation
The mEVAL platform provides patients with two ways to complete the PRO assessment. Seven days prior to the appointment, a patient receives a secure email with a unique link to the PRO questionnaire and can complete it at any time up to the time of the appointment, with 72% of our patients having a valid email address. In the clinic, the patient is also able to complete the assessment on a tablet computer. Overall, we are below our goal with 47% of patients complete a mEVAL assessment, 17% completing them at home and 30% completing them in clinic.
When completing the assessment remotely, the unique link in the secure email does not require the patient to input of any information, such as name or birthday. Once in clinic, the staff uses a Quick Response (QR) code to start the patient’s assessment, specific to the clinic or physician being seen. To facilitate provider use of the completed PRO assessments, scores are uploaded into the EHR within 1 minute of completion, allowing the information to be viewed by clinicians during the same clinic visit.
We created a system-wide “core assessment” to gain a better understanding of the patient’s overall health-related quality of life and provide the potential for population health assessment. The core assessment was designed by a panel of PRO subject matter experts and is completed by patients at least annually and no more often than weekly. It includes a general health question, a current health visual analog scale and PROMIS-Bank v1.2 Physical Function and PROMIS-Bank v1.0 Depression . Physical Function and Depression were chosen to evaluate overall physical and mental health of the patient [17–19]. Other domains, like Anxiety, were also considered. This core assessment can be supplemented by specialty-specific PROs selected by the clinical department and vetted by medical subject matter experts and through review of relevant literature. PROMIS instruments were chosen for their wide applicability and the brevity and responsiveness afforded by the computer adaptive testing algorithms, which selects the next question based on how the patient responded to the previous question. We identified specific thresholds for common instruments, such as PROMIS Depression, in order to allow for easier scoring interpretation and standardization throughout the healthcare system. In total, more than 200 different instruments are in use, with the core assessment implemented in each clinic.
Leadership support was crucial in advocating for the use of PROs and facilitating an enterprise adoption process. We identified key stakeholders for an Executive Steering Committee, representing Quality, Strategy, Operations, and Information Technology. This committee helped to develop a project charter, which defined the scope of the project, measures of success, and the project timeline. The Executive Steering Committee also helped identify and engage support teams including Marketing, Asset Management for tablet support, IT Training, and others.
Physician champions were identified for each clinic. These champions advocate for the use of PROs, identify specialty-specific PROs, and assist project teams in identifing and removing implementation barriers. They were the clinical expert leads demonstrating clinical use of the data to help support adoption within the clinic. The physician champions were also instrumental in organizing clinician training and education regarding scoring interpretation and standardized scoring thresholds used throughout the healthcare system.
Phase 2: Implementation
When implementing PROs within a clinic, it was essential to assign the appropriate providers and clinics to the correct questionnaires. This is particularly important if a provider is associated with more than one clinic, as is the case for many of our oncology providers, for example. This ensures that a patient receives the correct assessment associated with their diagnosis, provider, and practice.
When developing a home-grown PRO platform, each instrument must undergo careful pre-release testing to ensure that the content and scoring algorithm are accurate and instruments are displayed as intended.
Our executive dashboard included metrics around overall completion percentage, number of assessments completed by month, as well as a running total of completed assessments. This dashboard allowed for the overall progress of the project to be viewed on one screen. While the overall completion percentage is less than 80%, some clinics have completion percentages greater than 90%. This disparity has prompted the discussion of promoting provider adoption within clinics with lower completion percentages.
Phase 3: Post-implementation
We implement upgrades on a quarterly release cycle instead of releasing individual upgrades on an ad hoc basis. This allows us to plan when the different functionalities will be ready, provide training and communication to existing users, and more efficiently use personnel resources.
Feedback and PRO use
Overall assessment completion percentage helped clinics track their progress (Fig. 3) and identify clinics with lower completion percentages that may need more support, training, or education. We provide PRO data to our clinicians for use in patient care and to researchers, with appropriate regulatory approval, for secondary use. Using these data, clinicians could also evaluate how different diseases impact quality of life domains. We created a self-service portal (Fig. 2) where providers and staff can view aggregated scoring information by clinic, assessment or instrument. This allows the user to compare different patient cohorts.
In order to ensure that all providers and staff understand the organizational focus of PROs, we worked with the marketing department to develop a strategy to inform clinicians and staff about PROs and how they fit into our institution. We have partnered with physician champions to tell their story of how PROs have changed the way they provide care for their patients to help promote adoption.
Our goal is to implement PROs in all ambulatory clinics at University of Utah Health, and to date we have achieved implementation in 75% of clinics, including Primary Care, Oncology, and Specialty clinics, such as Dermatology and Cardiology. Overall, our experiences can be summarized as followed:
Pre-implementation planning facilitates higher performance. It was important for us to complete development, create a governance model, and coordinate a rollout process before we started using PROs. Creating these processes allowed for an expedited implementation schedule and prevented undue development troubleshooting.
A standardized process helps to streamline implementation. A standardized process for implementation allowed for specific action items to be defined. For new issues identified during clinic implementation, solutions were added to the implementation checklist (Table 2), avoiding future problems.
Support from organizational leadership promotes adoption. There needs to be support both from organizational and clinical leadership to allow for adoption at the provider level, culture change, and system-wide implementation, cultural changes.
Strong project collaboration is needed between several institutional groups. In our healthcare system, we have several clinics with overlapping patient cohorts. These clinics have implemented similar assessments using identical instruments where applicable. This allows for ease of patient burden while maximizing use of data in our healthcare system.
In a 15 month period, we have implemented more than 205 unique instruments and collected more than 306,000 assessments from over 200,000 unique patients. We have demonstrated the ability for a large academic medical center to implement PROs and collect data on a large-scale basis. We plan to study the data collected within our health care system for clinical improvements and improve the value of care within our system. Our experience is similar to that of others that successfully implemented electronic PRO collection and provided real-time data to clinicians [20,21,22,23] and extends this work through closer clinical integration and broader implementation across diverse clinical and specialty settings.
While we are still working towards complete implementation and increased patient uptake, the principles identified and described provide a strong basis for continued refinement and success.
It is feasible to implement PROs throughout a healthcare system, provided there is the full support of senior administration. The impact of our system-wide program for individual patient, point-of-care evaluation, as well as population health assessment will be assessed in future research. In order to determine greater feasibility it is necessary for other institutions to demonstrate the ability to implement a comprehensive program for the collection of PROs.
Electronic Health Record
Basch, E. (2016). The rise of patient-reported outcomes in oncology. Health Serv Res Qual Care, Retrieved from https://am.asco.org/daily-news/rise-patient-reported-outcomes-oncology.
Brim, R. L., & Pearson, S. D. (2013 Apr). The use and reporting of patient-reported outcomes in phase III breast cancer trials. Clin Trial, 10(2), 243–249. https://doi.org/10.1177/1740774513475529.
Deshpande, P. R., Rajan, S., Sudeepthi, B. L., & Abdul Nazir, C. P. (2011). Patient-reported outcomes: A new era in clinical research. Perspect Clin Res, 2(4), 137–144.
Hostetter, M, Klein, S. (2012). Using patient-reported outcomes to improve Heath Care quality. In Qual Matter. Retrieved from https://www.commonwealthfund.org/publications/newsletter/using-patient-reported-outcomes-improve-health-care-quality.
Parrish, RG. (2010). Measuring Population Health Outcomes. Preventing Chronic Disease, 7(4), A71.
Higgins JPT, Greene S, editors. Cochrane handbook for systematic reviews of interventions. The Cochrane collaboration; 2011.
Wiklund, I. (2004), Assessment of patient-reported outcomes in clinical trials: the example of health-related quality of life. Fundamental & Clinical Pharmacology, 18, 351-363. https://doi.org/10.1111/j.1472-8206.2004.00234.x.
McAllister, M, Dearing, A. (2015). Patient reported outcomes and patient empowerment in clinical genetics services. Clinical genetics, 88, 114-121. https://doi.org/10.1111/cge.12520.
Wintner, L. M., Giesinger, J. M., Kemmler, G., Sztankay, M., Oberguggenberger, A., Gamper, E. M., Sperner-Unterweger, B., & Holzner, B. (2012). The benefits of using patient-reported outcomes in cancer treatment: An overview. Wien Klin Wochenschr, 124(9–10), 293–303.
Schepers, S. A., et al. (2014). Patient reported outcomes in pediatric oncology practice: Suggestions for future usage by parents and pediatric oncologists. Pediatr Blood Cancer, 61(9), 1707–1710.
Huebner, J, Rose, C, Geissler, J, Gleiter, CH, Prott, FJ, Muenstedt, K, Micke, O, Muecke, R, Buentzel, J, Bottomley, A, Hofheinz, RD, European Journal of Cancer Care (2014). Integrating cancer patients' perspectives into treatment decisions and treatment evaluation using patient-reported outcomes - a concept paper. Bd. 23, H. 2, S. 173–179.
Talcott, JA, Manola, J, Chen, RC, Clark, JA, Kaplan, I, D'Amico, AV, Zietman, AL. (2014). QA/QI using patient-reported outcomes. BJU International, 114, 511-516. https://doi.org/10.1111/bju.12464.
Chang, S, Gholizadeh, L, Salamonson, Y, DiGiacomo, M, Betihavas, V, Davidson, PM, (2011) Health span or life span: The role of patient-reported outcomes in informing health policy, Health Policy, 100(1), 96-104. https://doi.org/10.1016/j.healthpol.2010.07.001.
Marquis, P, Arnould, B, Acquadro, C, Roberts, WM (2006). Patient-reported outcomes and health-related quality of life in effectiveness studies: pros and cons. Drug Dev Res, 67, 193-201. https://doi.org/10.1002/ddr.20077.
Wagle, N. (2017). Implementing patient reported outcome measures. N Engl J Med Catal, Retrieved from https://catalyst.nejm.org/implementing-proms-patient-reported-outcome-measures/.
Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. J Clin Epidemiol, 63(11), 1179–1194.
Pilkonis, P. A., Choi, S. W., Reise, S. P., Stover, A. M., Riley, W. T., & Cella, D. (2011). Item banks for measuring emotional distress from the patient-reported outcomes measurement information system (PROMIS®): Depression, anxiety, and anger. Assessment, 18(3), 263–283.
McNamara, R. L., Spatz, E. S., Kelley, T. A., Stowell, C. J., Beltrame, J., Heidenreich, P., et al. (2015). Standardized outcome measurement for patients with coronary artery disease: Consensus from the international consortium for health outcomes measurement (ICHOM). J Am Heart Assoc, 4(5), e001767.
Rodrigues, I. A., Sprinkhuizen, S. M., Barthelmes, D., Blumenkranz, M., Cheung, G., Haller, J., et al. (2016). Defining a minimum set of standardized patient-centered outcome measures for macular degeneration. Am J Ophthalmol, 168, 1–12.
Vickers, A. J., Savage, C. J., Shouery, M., Eastham, J. A., Scardino, P. T., & Basch, E. M. (2010). Validation study of a web-based assessment of functional recovery after radical prostatectomy. Health Qual Life Outcomes, 8(1), 82.
Abernethy, A., Ahmad, A., Zafar, A., Wheeler, S., Reese, J., & Lyerly, J. (2010). Electronic patient-reported data capture as a foundation of rapid learning cancer care. Med Care, 48(6 Suppl), S32–S38.
Weldring, T., & Smith, S. (2013). Patient-reported outcomes (PROs) and patient-reported outcome measures (PROMS). Health Services Insights, 6, 61–68.
Papuga, M. O., Dasilva, C., McIntyre, A., Mitten, D., Kates, S., & Baumhauer, J. F. (2017). Large scale clinical implementation of PROMIS computer adaptive testing with direct incorporation into the electronic medical record. Health Syst. https://doi.org/10.1057/s41306-016-0016-1.
University of Utah Orthopaedics Center for initially creating the mEVAL platform in conjunction with the University of Utah Enterprise Data Warehouse team.
Availability of data and materials
All data used was collected and analyzed from within the University of Utah’s Enterprise Data Warehouse.
Ethics approval and consent to participate
Consent for publication
All authors have consented for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Biber, J., Ose, D., Reese, J. et al. Patient reported outcomes – experiences with implementation in a University Health Care setting. J Patient Rep Outcomes 2, 34 (2018). https://doi.org/10.1186/s41687-018-0059-0
- Patient reported outcome measures
- Information systems
- Patient-centered care
- Hospital administration