Skip to main content
Journal of Patient-Reported Outcomes
Download PDF
Download ePub

Feasibility of PROMIS using computerized adaptive testing during inpatient rehabilitation

Journal of Patient-Reported Outcomes volume 7, Article number: 44 (2023) Cite this article

Abstract

Background

There has been an increased significance on patient-reported outcomes in clinical settings. We aimed to evaluate the feasibility of administering patient-reported outcome measures by computerized adaptive testing (CAT) using a tablet computer with rehabilitation inpatients, assess workload demands on staff, and estimate the extent to which rehabilitation inpatients have elevated T-scores on six Patient Reported Outcomes Measurement Information System® (PROMIS®) measures.

Methods

Patients (N = 108) with stroke, spinal cord injury, traumatic brain injury, and other neurological disorders participated in this study. PROMIS computerized adaptive tests (CAT) were administered via a web-based platform. Summary scores were calculated for six measures: Pain Interference, Sleep Disruption, Anxiety, Depression, Illness Impact Positive, and Illness Impact Negative. We calculated the percent of patients with T-scores equivalent to 2 standard deviations or greater above the mean.

Results

During the first phase, we collected data from 19 of 49 patients; of the remainder, 61% were not available or had cognitive or expressive language impairments. In the second phase of the study, 40 of 59 patients participated to complete the assessment. The mean PROMIS T-scores were in the low 50 s, indicating an average symptom level, but 19–31% of patients had elevated T-scores where the patients needed clinical action.

Conclusions

The study demonstrated that PROMIS assessment using a CAT administration during an inpatient rehabilitation setting is feasible with the presence of a research staff member to complete PROMIS assessment.

Introduction

Patient-reported outcomes (PROs) have been embraced increasingly as relevant endpoints of clinical trials and clinical research as well as routine clinical practice [1, 2]. In the United States, the current focus on patient-centered research and care has reinforced the value of PROs and the inclusion of the patient’s voice in research [3, 4]. There is a growing body of evidence that PROs contribute to improved quality of care and patient-provider communication [5, 6], aid in the management of chronic conditions [7], screen for specific health disorders [8, 9], highlight symptoms that providers systematically miss or underappreciated [10, 11], and increase patient satisfaction with care [12]. PROs have also been predictive of distal outcomes such as disease status, mortality, morbidity, and function in a range of conditions and diseases that are objective measures of disease [13,14,15]. There is also growing interest in the use of PROs in clinical care for measuring facility performance [16], particularly for surgical care, as they provide complementary information to typically monitored clinical outcomes such as morbidity and mortality [17].

Several other factors have contributed to the rapid increase in the adoption and implementation of PROs in clinical settings, including the application of item response theory (IRT) measurement to the development of PROs [18,19,20], and progress within the technological infrastructure, allowing for wider use of information technology for PRO administration, scoring, display of results, and interpretation via tablet computers [21,22,23], smartphone apps [24,25,26], and electronic medical records [27, 28]. In addition, there is an increased demand by payers, accreditors, professional organizations, and clinicians to measure and improve PRO-measured outcomes at the patient, clinic, and healthcare system levels [29].

Most of the literature reporting on the use of PROs in rehabilitation-focused clinical settings has focused on outpatient ambulatory clinics or the transition from inpatient to outpatient settings [30,31,32]. In other clinical settings, such as oncology, the feasibility of patients reporting during outpatient clinic visits via waiting room tablet computers has been established, with mean compliance rates ranging from 75 to 85%, high patient satisfaction, and good usability of systems even among those who are not familiar with the internet, are elderly, or frail [33, 34].

PROs have also been used in inpatient settings, although the literature is considerably sparse in this context [35]. The assessment of hospitalized patients may pose additional challenges and require additional resources, as these patients are likely to require assistance in completing measures [36]. Depending on the clinical population being assessed, there may be cognitive, communication, and physical challenges ranging from intravenous lines in arms to functional limitations that limit PRO data collection [21, 37]. Patients in acute care settings typically have short stays, requiring consideration of the timing and frequency of PRO administration [1, 38]. The hospital environment may influence their responses [39] or render the content of the PRO items and their response categories irrelevant. Conversely, patients hospitalized in rehabilitation settings have longer stays and structured schedules, allowing easier integration of PRO assessments into their daily routine.

The importance of PROs to rehabilitation research and clinical practice has been noted, as has the value of obtaining outcomes-related data from patients themselves [40,41,42]. Standardized PRO measures can be used to assist in the clinical care of patients for several purposes: to determine an individual’s strengths and weaknesses; to facilitate effective interdisciplinary communication; to determine readiness to move to the next level of rehabilitation or discharge from inpatient care; and following discharge, to track functional independence, participation, health status, and health-related quality of life [43, 44]. Beyond clinical care, PROs have a role in rehabilitation comparative effectiveness research, clinical trials [45], and in assessing provider performance.

There are important considerations in the choice and implementation of PROs in rehabilitation assessment [46, 47]. These considerations include content and response formats of the questions, such as time references that might not be relevant for individuals with highly variable symptoms. Rehabilitation populations often have unique barriers to completing PROs that need to be addressed. For example, stroke patients may have cognitive, communication, or other functional deficits, limiting them from completing PROs or answering items in a reliable manner [48]. Assessment by a proxy may substitute for patient self-assessment [49, 50], although differences between patient and proxy reports suggest proxy reports should be considered complementary and not a substitute [51]. Older rehabilitation inpatients may have comorbid conditions, which argues for measures that capture health status across the conditions, to the extent possible, as opposed to the burden of additional measures [52, 53]. In addition, PRO assessment in rehabilitation settings faces the challenge common to most clinical settings: the absence of a widely accepted measure and lack of consensus regarding which measures to use [54, 55].

The Patient Reported Outcomes Measurement Information System (PROMIS), initiated by the National Institutes of Health (NIH) in 2004, is a collection of person-centered measures that can be used to evaluate the physical, mental, and social health of adults and children, both in the general population and individuals living with chronic conditions. It was developed and validated using state-of-the-science methods to be psychometrically sound and to transform how life domains are measured. Most of the measures are universal and are designed to be relevant across a wide range of conditions for the assessment of symptoms and functions. PROMIS measures are reported using a T-score metric in which 50 is the mean of a reference population and 10 is the standard deviation. In PROMIS measures, high T-scores represent more of what is being measured. For example, high T-scores of fatigue mean a severe level of fatigue while high T-scores of physical function represent the good condition of the body. Most PROMIS measures are based on the mean score of a sample of individuals that matched the US 2000 General Census with respect to gender, age, race/ethnicity, and education [56].

PROMIS measures offer several valuable features: they are not disease-specific, which allows for comparisons across conditions or populations; there are several administration options, including fixed-length “short forms” and dynamic assessment using computerized adaptive testing (CAT), which allows for brief, precise assessments with a reduced respondent burden; and the item banks are constructed to cover the full range of a trait, reducing or eliminating floor and ceiling effects [57,58,59]. CAT is an individually-tailored test with items selected based on the patient's level on the trait being measured [60, 61]. As CAT restricts questioning to a specific number of distinct items, CAT can be more responsive than traditional, fixed-length evaluation tools and reduce response burden precisely [19].

Building on the development and methodology of PROMIS, the development of several rehabilitation-relevant measurement systems has been funded. Quality of Life in Neurological Disorders (Neuro-QoL) [62] which evaluates and monitors the physical, mental and social effects experienced by adults and children living with neurological conditions was funded by the NIH. Moreover, the Spinal Cord Injury-Quality of Life (SCI-QOL) measurement system [63]; and the Traumatic Brain Injury-Quality of Life measurement system (TBI-QOL) [64] are funded by the National Institute on Disability and Rehabilitation Research.

PROMIS measures have been used in several studies involving rehabilitation populations [65] though their use in an inpatient rehabilitation setting has been reported recently [36, 66, 67]. CAT platforms enhance treatment and decision-making for patients through the collection of PROs [68]. CAT administration is useful in collecting patients’ outcomes in primary care settings these days [69, 70]. After being discharged from inpatient rehabilitation, PROs have been usually administered by telephone interviews or mailed questionnaires. But these approaches were tedious, costly, and had low response rates from the patients [31, 71, 72]. Therefore, CAT administration was conducted to collect patient-reported outcomes for post-rehabilitation patients and the study showed that it was feasible for a subset of patients [32]. So, CAT can be another choice for increasing patient involvement and minimizing costs in order to measure PROs [73, 74]. Few studies assessed the feasibility of PRO data collection using CAT in medical rehabilitation [32]. Completion rates, acceptability, time, and type of survey administration measured the feasibility study [70]. In this study, our objective was to evaluate the feasibility of administering PROMIS measures using a tablet computer with rehabilitation inpatients by examining the burden on patients as well as the clinical staff. Feasibility was assessed by completion time, completion rate, and staff assistance required. We also sought to estimate the extent to which rehabilitation inpatients have elevated scores on PROMIS Pain Interference, Sleep Disruption, Anxiety, Depression, Illness Impact Positive, and Illness Impact Negative measures. We sought to collect and report PROMIS measures shortly before routinely scheduled team conferences to enhance clinical relevance.

Methods

Study approach

The Shirley Ryan AbilityLab, formerly the Rehabilitation Institute of Chicago (RIC), is an internationally recognized specialty hospital and healthcare network dedicated to the care and rehabilitation of physical and neurological disabilities. Of the 182 inpatient beds at the flagship hospital, 24 were taken for conducting the study. The floor served individuals with neurological disorders including stroke, spinal cord injury, traumatic brain injury, and other neurological disorders. Patients completed the CAT through a web-based platform using tablet computers and the data were stored in a SQL database. Finally, T-scores provide the symptom severity to the clinicians. Figure 1 depicts the study approach.

Fig. 1
figure 1

Study approach: PROMIS on pain interference, sleep disruption, anxiety, depression, illness impact positive, and illness impact negative during inpatient rehabilitation using a CAT

Full size image

Participants

Patients were eligible if they had a neurological disorder including stroke, spinal cord injury, traumatic brain injury, and could complete the CAT without assistance. The patients who required assistance reading items or reporting responses were coded as “not appropriate”. We excluded patients who were unavailable, refused to participate, and had other impairments including language barrier, behavioral issues, etc.

Data collection

This study obtained input from rehabilitation physicians, nurses, and allied health therapists regarding the issues they considered most important to assess during inpatient rehabilitation. PROMIS item banks that aligned with the issues they identified were Pain Interference, Sleep Disruption, Anxiety, and Depression. Clinicians were also interested in understanding how patients perceived the negative and positive consequences of the conditions that precipitated their admissions. Thus, they recommended administering Illness Impact Positive and Illness Impact Negative. Northwestern University Department of Medical Social Sciences staff consulted with the hospital’s Information Systems department to create a local installation of the NIH Assessment CenterSM, a web-based platform for PROMIS CAT administration and scoring, behind the hospital’s secure firewall. AbilityLab’s Information Systems staff set up a SQL database that received data from tablet computers over the hospital’s Wi-Fi network.

Our initial plan was to complete assessments the day before each patient’s weekly team conference. Nurses were charged with approaching patients and asking them to complete the assessments within 24 h of the weekly team conference as part of their routine clinical duties. They were instructed to invite patients to complete the CATs using a tablet computer and to demonstrate how to use the equipment but provided no additional assistance. The study had two phases and the sole procedural change between these phases was the substitution of a research assistant for a nursing role in instrument administration. It became evident quickly that the nursing staff was not able to administer instruments consistently and in a timely manner in addition to their other responsibilities. Even though patients are apt to have a relationship with nursing staff and be amenable to answering PRO questions, the loss of PRO data imperiled the success of the project. Thus, we decided to assign assessment responsibility to a research assistant to document the staff workload that was required.

Implementation planning

We considered several of the questions listed in the International Society for Quality of Life Research Guide to implement PRO assessments in clinical practice [75]. We list the key decisions in Table 1.

Table 1 Several questions and decisions as part of the implementation planning
Full size table

Results

Implementation effectiveness

The feasibility was assessed with completion time, completion rate, and staff assistance required. Nineteen of 49 eligible patients (39%) completed an initial assessment by nurses during the first phase of the study which lasted 6 weeks. Reasons for not completing the assessments included patient unavailability (27); cognitive or communicative impairments (2), and patient sleeping (1). During the second modified phase, a dedicated staff member was appointed to assist patients with the assessments. A total of 98 assessments from 40 of the 59 admitted patients (68%) were completed. Out of the 98 completed assessments, 40 patients completed CATs on 1 occasion, 13 patients completed CATs on 2 occasions, 5 patients completed CATs on 3 occasions, 3 patients on 4 occasions, and 1 patient on 5 occasions. In this context, the patients cooperated with our staff on multiple occasions. Patients’ lengths of stay vary widely, reflecting the extent of functional improvement, goal attainment, and readiness of the family for discharge among other considerations. Thus, we had the opportunity to evaluate the feasibility of routine reassessments over varying lengths of stay. Clinicians used results from repeated assessments and changes (or stability) over time to inform clinical decision-making. Our focus was on the feasibility of repeated assessments—would patients object to or cooperate? We certainly expect within-patient repeated measures to be correlated as this is well established. The median time to complete the 6 CATs was 12 min (interquartile range = 5–46 min). This completion time in a busy inpatient rehabilitation setting proves our successful implementation as planned. Patients completed 12% of the assessments independently; the staff members read questions to facilitate 3% of the assessments and read and recorded responses for 85% of assessments. We documented PROMIS item responses regardless of staff assistance. The proportion of assessments requiring staff assistance helps inform the staffing requirements for routine PROMIS administration. Patient eligibility and recruitment status are reported in Table 2.

Table 2 Patient eligibility and recruitment
Full size table

PROMIS results

During the first 6 weeks, nurses completed initial assessments. It was evident that the added burden on nurses resulted in many missed assessments. They did not have the time to make multiple attempts to complete assessments or wait for patients to wake or visitors to depart. We modified the protocol during a second phase lasting 11 weeks by designating a dedicated staff member to complete PROMIS assessments and providing assistance as needed to complete assessments. In this way, we adapted the procedure of staffing requirements for routine data collection in the rehabilitation setting and in doing so, the completion rate increased from 39 to 68%. Assistance included reading questions and recording answers as directly as possible when requested by patients. When the second phase started, we also documented the extent of assistance requested by patients. The staff member asked patients to complete the assessments and offered the minimum amount of assistance required, reading items and progressing to recording responses. Level of assistance was not a criterion since the staff time would be similar to monitoring PROMIS completion or providing assistance. We deemed the patient and staff time commitment and completion rate as satisfactory in this pilot study.

Table 3 shows that while mean T-scores were in the low 50 s, indicating an average symptom level, the score distribution was such that between 19 and 31% of patients had PROMIS T-scores with clinically actionable results (T-scores > 60), and 2% to 7% had scores more than 2 standard deviations above the mean (T-scores > 70). Most of these patients had not been identified by the clinical team as demonstrating significant clinical concerns.

Table 3 Descriptive statistics for PROMIS administrations
Full size table

Discussion

The CAT usage in rehabilitation is appealing as it can reduce the burden on the respondent. In this study, the goal was to evaluate the feasibility of collecting data from patients with neurological disorders using PROMIS CAT during an inpatient rehabilitation setting. For this research, the data was collected from patients themselves by obtaining response formats of the survey questions in the assessment. Previous studies attempted PRO data collection during inpatient hospitalizations and after inpatient rehabilitation. In one study, only 7% of the eligible patients completed the CAT-administered PRO as they were using the internet or telephone after being discharged from inpatient rehabilitation [32]. In addition, completion rates for other feasibility studies during inpatient (51%) and outpatient (41%) were almost the same as ours (39% and 67% in the initial and modified phase respectively) [21, 76]. Although our rate appears to be on the low end of estimates compared to one study [77], the response rate reflects our ability to provide sufficient staffing to collect PRO data.

PROMIS item banks were developed with the general population and clinical samples. Results are reported as T-scores and have been published in a variety of peer-reviewed journals [78]. Investigators routinely use parametric statistics to compare groups and examine change over time [79]. T-scores for CAT are more reliable and have instant outcomes in a real-world scenario [80]. In addition, this scoring metric has been used in different PROMIS feasibility and comparison studies [80,81,82]. So, we calculated the T-score metric based on the collected data to interpret the PROMIS scores. From Table 3, we can see that all six PROMIS measures have a T-score mean (average symptom level) which is less than 60. Pain Interference has 7% of patients who had T-scores greater than 70. Again, Psychological Illness Impact Negative and Pain Interference have 30.6% and 30.5% of patients, respectively, with T-scores above 60. In this context, a higher T-score means it is important or advised to take appropriate clinical action. Conversely, for the cases of Psychological Illness Impact Positive and Depression, most of the patients had lower T-scores. In this scenario, T-score provides an overall health status for the patients who participated in the PROMIS CAT in an inpatient rehabilitation setting.

To design our feasibility study, we focused on the required key areas including acceptability, implementation, practicality, and adaptation according to the feasibility study design [83]. One aspect of the feasibility relates to patient cooperation; another relates to the staffing requirements for routine data collection. Results allow us to estimate the staffing required for routine data collection and the efforts required to complete assessments outside of therapies, meals, bowel and bladder programs, and visits by family members and friends. The added burden on nurses resulted in many missed assessments during the initial phase of the study. That’s why we modified the protocol during the second phase according to the requirements and context. Moreover, the time to complete the CAT was also acceptable in a busy inpatient setting. Therefore, results from this feasibility study provide valuable lessons that will help guide PROs collection during inpatient rehabilitation. First, clinicians identified six PROMIS item banks as relevant to patients’ concerns. While most T-scores were not in a range that required clinician action, the information could help inform patient care decisions. Following the completion of this feasibility project, clinicians decided to update the assessment protocol and omit Illness Impact Positive and Illness Impact Negative as they found information from these item banks to be less actionable.

We acknowledge some limitations in the research. Study limitations include the evaluation of PROMIS feasibility at only one inpatient rehabilitation hospital. As patients were engaged from only one rehabilitation hospital, results cannot be generalized confidently to other rehabilitation hospitals and units. However, sometimes patients couldn’t participate to complete the CAT due to their family in the room or they were sleeping. Also, the staff of the rehabilitation hospital had limited time, making it difficult to make multiple attempts to complete the assessment. In our inpatient study, it was a required step to assign a research assistant to assist patients in completing the assessments [36]. We do not believe that a research assistant would introduce markedly different bias than would a nurse—both kinds of staff members arrived with a tablet computer and asked patients to complete the PROs as independently as possible. We note that the level of assistance may create a bias. However, for the purposes of this study, the need for a nonclinical staff member to collect the data is an important finding. In our study, we couldn’t provide the observed standard deviations for each T-score. Despite these lackings, the results demonstrate how CAT assessment provides overall health status during busy rehabilitation settings within a short time. Future studies should develop a multilingual CAT platform and assign more staff members for collecting inpatient data to increase the response rate. Future research should also focus on compliance monitoring that measures patient and staff time to complete assessments in the field of the CAT platform during inpatient rehabilitation.

Conclusions

The collected data supports that Patient-reported outcome measure (PROM) assessments using CAT are feasible during an inpatient rehabilitation setting with the presence of a research staff member to complete PROMIS assessment. We noted that the small sample limits the generalizability of the study findings. The primary focus of the study was to evaluate the feasibility of PRO data collection and estimate the resources that are required. For these purposes, the sample was sufficient. There is enough variation in patient-level outcomes to support their consideration as patient-reported outcome-based performance measures (PROM-PMs). Further work is needed to identify the frequency of patient and institutional barriers that affect the feasibility of routine assessment across patient populations during inpatient settings and the utility of PROMs to support the development of performance measures.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

PROs:

Patient-reported outcomes

CAT:

Computerized adaptive testing

PROMIS®:

Patient-Reported Outcomes Measurement Information System®

PROMs:

Patient-reported outcome measures

IRT:

Item response theory

RIC:

Rehabilitation Institute of Chicago

SQL:

Structured Query Language

NIH:

National Institute of Health

References

  1. Snyder CF, Aaronson NK, Choucair AK, Elliott TE, Greenhalgh J, Halyard MY et al (2012) Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res 21:1305–1314

    Article  PubMed  Google Scholar 

  2. Marshall S, Haywood K, Fitzpatrick R (2006) Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract 12:559–568

    Article  PubMed  Google Scholar 

  3. Chung AE, Basch EM (2015) Incorporating the patient’s voice into electronic health records through patient-reported outcomes as the “review of systems.” J Am Med Inform Assoc. https://doi.org/10.1093/jamia/ocu007

    Article  PubMed  PubMed Central  Google Scholar 

  4. Selby P, Velikova G (2018) Taking patient reported outcomes centre stage in cancer research—why has it taken so long? Res Involv Engagem. https://doi.org/10.1186/s40900-018-0109-z

    Article  PubMed  PubMed Central  Google Scholar 

  5. Santana MJ, Haverman L, Absolom K, Takeuchi E, Feeny D, Grootenhuis M et al (2015) Training clinicians in how to use patient-reported outcome measures in routine clinical practice. Qual Life Res 24:1707–1718

    Article  PubMed  Google Scholar 

  6. Snyder CF, Blackford AL, Aaronson NK, Detmar SB, Carducci MA, Brundage MD et al (2011) Can patient-reported outcome measures identify cancer patients’ most bothersome issues? J Clin Oncol 29:1216–1220

    Article  PubMed  Google Scholar 

  7. Aiyegbusi OL, Kyte D, Cockwell P, Marshall T, Gheorghe A, Keeley T et al (2017) Measurement properties of patient-reported outcome measures (PROMs) used in adult patients with chronic kidney disease: a systematic review. PLoS ONE 12:e0179733

    Article  PubMed  PubMed Central  Google Scholar 

  8. Wells G, Li T, Maxwell L, Maclean R, Tugwell P (2008) Responsiveness of patient reported outcomes including fatigue, sleep quality, activity limitation, and quality of life following treatment with abatacept for rheumatoid arthritis. Ann Rheum Dis 67:260–265

    Article  CAS  PubMed  Google Scholar 

  9. Greenhalgh J, Abhyankar P, McCluskey S, Takeuchi E, Velikova G (2013) How do doctors refer to patient-reported outcome measures (PROMS) in oncology consultations? Qual Life Res 22:939–950

    Article  PubMed  Google Scholar 

  10. Banerjee AK, Okun S, Edwards IR, Wicks P, Smith MY, Mayall SJ et al (2013) Patient-reported outcome measures in safety event reporting: PROSPER consortium guidance. Drug Saf 36:1129–1149

    Article  PubMed  PubMed Central  Google Scholar 

  11. Weingart SN, Gandhi TK, Seger AC, Seger DL, Borus J, Burdick E et al (2005) Patient-reported medication symptoms in primary care. Arch Intern Med 165:234–240

    Article  PubMed  Google Scholar 

  12. Reeve BB, Wyrwich KW, Wu AW, Velikova G, Terwee CB, Snyder CF et al (2013) ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res 22:1889–1905

    Article  PubMed  Google Scholar 

  13. Gotay CC, Kawamoto CT, Bottomley A, Efficace F (2008) The prognostic significance of patient-reported outcomes in cancer clinical trials. J Clin Oncol 26:1355–1363

    Article  PubMed  Google Scholar 

  14. Coyle PK, Khatri B, Edwards KR, Meca-Lallana JE, Cavalier S, Rufi P et al (2017) Patient-reported outcomes in relapsing forms of MS: real-world, global treatment experience with teriflunomide from the Teri-PRO study. Mult Scler Relat Disord 17:107–115

    Article  PubMed  Google Scholar 

  15. Pasalic D, Ludmir EB, Allen PK, Thaker NG, Chapman BV, Hanna EY et al (2020) Patient-reported outcomes, physician-reported toxicities, and treatment outcomes in a modern cohort of patients with sinonasal cancer treated using proton beam therapy. Radiother Oncol. https://doi.org/10.1016/j.radonc.2020.05.007

    Article  PubMed  Google Scholar 

  16. Basch E, Torda P, Adams K (2013) Standards for patient-reported outcome-based performance measures. JAMA 310:139–140

    Article  CAS  PubMed  Google Scholar 

  17. Bilimoria KY, Cella D, Butt Z (2014) Current challenges in using patient-reported outcomes for surgical care and performance measurement: everybody wants to hear from the patient, but are we ready to listen? JAMA Surg 149:505–506

    Article  PubMed  Google Scholar 

  18. Ware JE, Gandek B, Sinclair SJ. Item response theory and computerized adaptive testing: Implications for outcomes measurement in rehabilitation. Rehabilitation. Available: https://psycnet.apa.org/journals/rep/50/1/71/

  19. Cheville AL, Yost KJ, Larson DR, Dos Santos K, O’Byrne MM, Chang MT et al (2012) Performance of an item response theory-based computer adaptive test in identifying functional decline. Arch Phys Med Rehabil 93:1153–1160

    Article  PubMed  PubMed Central  Google Scholar 

  20. Fries JF, Bruce B, Cella D (2005) The promise of PROMIS: using item response theory to improve assessment of patient-reported outcomes. Clin Exp Rheumatol 23:S53–S57

    CAS  PubMed  Google Scholar 

  21. Heinemann AW, Deutsch A, Cella D, Cook KF, Foster L, Miskovic A et al (2018) Feasibility of collecting patient-reported outcomes for inpatient rehabilitation quality reporting. Health Serv Res 53:1834–1850

    Article  PubMed  Google Scholar 

  22. Heiden BT, Subramanian MP, Nava RG, Patterson AG, Meyers BF, Puri V et al (2022) Routine collection of patient-reported outcomes in thoracic surgery: a quality improvement study. Ann Thorac Surg 113:1845–1852

    Article  PubMed  Google Scholar 

  23. Schick-Makaroff K, Molzahn A (2015) Strategies to use tablet computers for collection of electronic patient-reported outcomes. Health Qual Life Outcomes 13:2

    Article  PubMed  PubMed Central  Google Scholar 

  24. Richter JG, Nannen C, Chehab G, Acar H, Becker A, Willers R et al (2021) Mobile app-based documentation of patient-reported outcomes—3-months results from a proof-of-concept study on modern rheumatology patient management. Arthritis Res Ther. https://doi.org/10.1186/s13075-021-02500-3

    Article  PubMed  PubMed Central  Google Scholar 

  25. Broderick JE, May M, Schwartz JE, Li M, Mejia A, Nocera L et al (2019) Patient reported outcomes can improve performance status assessment: a pilot study. J Patient Rep Outcomes 3:41

    Article  PubMed  PubMed Central  Google Scholar 

  26. Fischer KI, De Faoite D, Rose M (2020) Patient-reported outcomes feedback report for knee arthroplasty patients should present selective information in a simple design—findings of a qualitative study. J Patient Rep Outcomes 4:6

    Article  PubMed  PubMed Central  Google Scholar 

  27. Zhang R, Burgess ER, Reddy MC, Rothrock NE, Bhatt S, Rasmussen LV et al (2019) Provider perspectives on the integration of patient-reported outcomes in an electronic health record. JAMIA Open 2:73–80

    Article  PubMed  PubMed Central  Google Scholar 

  28. Horn ME, Reinke EK, Mather RC, O’Donnell JD, George SZ (2021) Electronic health record–integrated approach for collection of patient-reported outcome measures: a retrospective evaluation. BMC Health Serv Res 21:1–11

    Article  Google Scholar 

  29. Rosenlund S, Broeng L, Holsgaard-Larsen A, Jensen C, Overgaard S (2017) Patient-reported outcome after total hip arthroplasty: comparison between lateral and posterior approach: a randomized controlled trial in 80 patients with 12-month follow-up. Acta Orthop 88:239–247

    Article  PubMed  PubMed Central  Google Scholar 

  30. Coster W, Haley S, Jette A (2006) Measuring patient-reported outcomes after discharge from inpatient rehabilitation settings. J Rehabil Med 38:237–242

    Article  PubMed  Google Scholar 

  31. Haley SM, Siebens H, Coster WJ, Tao W, Black-Schaffer RM, Gandek B et al (2006) Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: I. Activity outcomes. Arch Phys Med Rehabil 5:1033–1042. https://doi.org/10.1016/j.apmr.2006.04.020

    Article  Google Scholar 

  32. Wong AWK, Heinemann AW, Miskovic A, Semik P, Snyder TM (2014) Feasibility of computerized adaptive testing for collection of patient-reported outcomes after inpatient rehabilitation. Arch Phys Med Rehabil 95:882–891

    Article  PubMed  Google Scholar 

  33. Basch E, Iasonos A, Barz A, Culkin A, Kris MG, Artz D et al (2007) Long-term toxicity monitoring via electronic patient-reported outcomes in patients receiving chemotherapy. J Clin Oncol 25:5374–5380

    Article  PubMed  Google Scholar 

  34. Recinos PF, Dunphy CJ, Thompson N, Schuschu J, Urchek JL 3rd, Katzan IL (2017) Patient satisfaction with collection of patient-reported outcome measures in routine care. Adv Ther 34:452–465

    Article  PubMed  Google Scholar 

  35. Masskulpan P, Riewthong K, Dajpratham P, Kuptniratsaikul V (2008) Anxiety and depressive symptoms after stroke in 9 rehabilitation centers. J Med Assoc Thail 91:1595–1602

    Google Scholar 

  36. Heinemann AW, Nitsch KP, Gracz K, Ehrlich-Jones L, Engel E, Wilson M et al (2022) Implementing patient-reported outcome measures in inpatient rehabilitation: challenges and solutions. Arch Phys Med Rehabil 103:S67–S77

    Article  PubMed  Google Scholar 

  37. Reeves M, Lisabeth L, Williams L, Katzan I, Kapral M, Deutsch A et al (2018) Patient-reported outcome measures (PROMs) for acute stroke: rationale, methods and future directions. Stroke 49:1549–1556

    Article  PubMed  Google Scholar 

  38. Santana M-J, Feeny D, Johnson JA, McAlister FA, Kim D, Weinkauf J et al (2010) Assessing the use of health-related quality of life measures in the routine clinical care of lung-transplant patients. Qual Life Res 19:371–379

    Article  PubMed  Google Scholar 

  39. Veenstra M, Moum T, Garratt AM (2006) Patient experiences with information in a hospital setting: associations with coping and self-rated health in chronic illness. Qual Life Res 15:967–978

    Article  PubMed  Google Scholar 

  40. Jensen RE, Snyder CF, Abernethy AP, Basch E, Potosky AL, Roberts AC et al (2014) Review of electronic patient-reported outcomes systems used in cancer clinical care. J Oncol Pract. https://doi.org/10.1200/jop.2013.001067

    Article  PubMed  Google Scholar 

  41. Katzan IL, Thompson NR, Lapin B, Uchino K (2017) Added value of patient-reported outcome measures in stroke clinical practice. J Am Heart Assoc. https://doi.org/10.1161/JAHA.116.005356

    Article  PubMed  PubMed Central  Google Scholar 

  42. Johnston B, Flemming K, Narayanasamy MJ, Coole C, Hardy B (2017) Patient reported outcome measures for measuring dignity in palliative and end of life care: a scoping review. BMC Health Serv Res 17:574

    Article  PubMed  PubMed Central  Google Scholar 

  43. Shisler R, Sinnott JA, Wang V, Hebert C, Salani R, Felix AS (2018) Life after endometrial cancer: a systematic review of patient-reported outcomes. Gynecol Oncol 148:403–413

    Article  PubMed  Google Scholar 

  44. Greenhalgh J, Meadows K (1999) The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract. https://doi.org/10.1046/j.1365-2753.1999.00209.x

    Article  PubMed  Google Scholar 

  45. Fong E, Li C, Aslakson R, Agrawal Y (2015) Systematic review of patient-reported outcome measures in clinical vestibular research. Arch Phys Med Rehabil 96:357–365

    Article  PubMed  Google Scholar 

  46. Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ (2010) The routine use of patient reported outcome measures in healthcare settings. BMJ 340:c186

    Article  PubMed  Google Scholar 

  47. Thorborg K, Roos EM, Bartels EM, Petersen J, Hölmich P (2010) Validity, reliability and responsiveness of patient-reported outcome questionnaires when assessing hip and groin disability: a systematic review. Br J Sports Med 44:1186–1196

    Article  CAS  PubMed  Google Scholar 

  48. Barrett AM (2010) Rose-colored answers: neuropsychological deficits and patient-reported outcomes after stroke. Behav Neurol 22:17–23

    Article  PubMed  PubMed Central  Google Scholar 

  49. Katzan IL, Thompson N, Uchino K (2016) Innovations in stroke: the use of PROMIS and NeuroQoL scales in clinical stroke trials. Stroke 47:e27-30

    Article  PubMed  Google Scholar 

  50. Kozlowski AJ, Singh R, Victorson D, Miskovic A, Lai J-S, Harvey RL et al (2015) Agreement between responses from community-dwelling persons with stroke and their proxies on the NIH neurological quality of life (Neuro-QoL) short forms. Arch Phys Med Rehabil 96:1986–92.e14

    Article  PubMed  PubMed Central  Google Scholar 

  51. Oczkowski C, O’Donnell M (2010) Reliability of proxy respondents for patients with stroke: a systematic review. J Stroke Cerebrovasc Dis 19:410–416

    Article  PubMed  Google Scholar 

  52. Gosselin S, Desrosiers J, Corriveau H, Hébert R, Rochette A, Provencher V et al (2008) Outcomes during and after inpatient rehabilitation: comparison between adults and older adults. J Rehabil Med 40:55–60

    Article  PubMed  Google Scholar 

  53. Patrick L, Knoefel F, Gaskowski P, Rexroth D (2001) Medical comorbidity and rehabilitation efficiency in geriatric inpatients. J Am Geriatr Soc 49:1471–1477

    Article  CAS  PubMed  Google Scholar 

  54. Lohr KN, Zebrack BJ (2009) Using patient-reported outcomes in clinical practice: challenges and opportunities. Qual Life Res 18:99–107

    Article  PubMed  Google Scholar 

  55. Fung CH, Hays RD (2008) Prospects and challenges in using patient-reported outcomes in clinical practice. Qual Life Res 17:1297–1302

    Article  PubMed  PubMed Central  Google Scholar 

  56. Liu H, Cella D, Gershon R, Shen J, Morales LS, Riley W et al (2010) Representativeness of the patient-reported outcomes measurement information system internet panel. J Clin Epidemiol 63:1169–1178

    Article  PubMed  PubMed Central  Google Scholar 

  57. Cook KF, O’Malley KJ, Roddey TS (2005) Dynamic assessment of health outcomes: time to let the CAT out of the bag? Health Serv Res 40:1694–1711

    Article  PubMed  PubMed Central  Google Scholar 

  58. Cella D, Gershon R, Lai J-S, Choi S (2007) The future of outcomes measurement: item banking, tailored short-forms, and computerized adaptive assessment. Qual Life Res 16(Suppl 1):133–141

    Article  PubMed  Google Scholar 

  59. Cella D, Choi SW, Condon DM, Schalet B, Hays RD, Rothrock NE et al (2019) PROMIS® adult health profiles: efficient short-form measures of seven health domains. Value Health 22:537–544

    Article  PubMed  PubMed Central  Google Scholar 

  60. Harniss M, Amtmann D, Cook D, Johnson K (2007) Considerations for developing interfaces for collecting patient-reported outcomes that allow the inclusion of individuals with disabilities. Med Care 45:S48-54

    Article  PubMed  PubMed Central  Google Scholar 

  61. Reeve BB (2006) Special issues for building computerized-adaptive tests for measuring patient-reported outcomes: the National Institute of Health’s investment in new technology. Med Care 44:S198-204

    Article  PubMed  Google Scholar 

  62. Salsman JM, Victorson D, Choi SW, Peterman AH, Heinemann AW, Nowinski C et al (2013) Development and validation of the positive affect and well-being scale for the neurology quality of life (Neuro-QOL) measurement system. Qual Life Res 22:2569–2580

    Article  PubMed  Google Scholar 

  63. Tulsky DS, Kisala PA, Victorson D, Tate DG, Heinemann AW, Charlifue S et al (2015) Overview of the Spinal cord injury-Quality of life (SCI-QOL) measurement system. J Spinal Cord Med. https://doi.org/10.1179/2045772315y.0000000023

    Article  PubMed  PubMed Central  Google Scholar 

  64. Tulsky DS, Kisala PA, Victorson D, Carlozzi N, Bushnik T, Sherer M et al (2016) TBI-QOL: development and calibration of item banks to measure patient reported outcomes following traumatic brain injury. J Head Trauma Rehabil 31:40–51

    Article  PubMed  Google Scholar 

  65. Amtmann D, Cook KF, Johnson KL, Cella D (2011) The PROMIS initiative: involvement of rehabilitation stakeholders in development and examples of applications in rehabilitation research. Arch Phys Med Rehabil 92:S12–S19

    Article  PubMed  PubMed Central  Google Scholar 

  66. Heinemann AW, Nitsch KP, Ehrlich-Jones L, Malamut L, Semik P, Srdanovic N et al (2019) Effects of an implementation intervention to promote use of patient-reported outcome measures on clinicians’ perceptions of evidence-based practice, implementation leadership, and team functioning. J Contin Educ Health Prof. https://doi.org/10.1097/ceh.0000000000000249

    Article  PubMed  Google Scholar 

  67. Nitsch KP, Stipp K, Gracz K, Ehrlich-Jones L, Graham ID, Heinemann AW (2020) Integrating spinal cord injury-quality of life instruments into rehabilitation: Implementation science to guide adoption of patient-reported outcome measures. J Spinal Cord Med. https://doi.org/10.1080/10790268.2020.1712893

    Article  PubMed  PubMed Central  Google Scholar 

  68. Chang C-H (2007) Patient-reported outcomes measurement and management with innovative methodologies and technologies. Qual Life Res 16(Suppl 1):157–166

    Article  PubMed  Google Scholar 

  69. Turner-Bowker DM, Saris-Baglama RN, DeRosa MA, Giovannetti ER, Jensen RE, Wu AW (2012) A computerized adaptive version of the SF-36 is feasible for clinic and Internet administration in adults with HIV. AIDS Care 24:886–896

    Article  PubMed  Google Scholar 

  70. Wu W-W, Johnson R, Schepp KG, Berry DL (2011) Electronic self-report symptom and quality of life for adolescent patients with cancer. Cancer Nurs. https://doi.org/10.1097/ncc.0b013e31820a5bdd

    Article  PubMed  Google Scholar 

  71. Chien T-W, Wang W-C, Wang H-Y, Lin H-J (2009) Online assessment of patients’ views on hospital performances using Rasch model’s KIDMAP diagram. BMC Health Serv Res 9:135

    Article  PubMed  PubMed Central  Google Scholar 

  72. Haley SM, Gandek B, Siebens H, Black-Schaffer RM, Sinclair SJ, Tao W et al (2008) Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: II. Participation outcomes. Arch Phys Med Rehabil. https://doi.org/10.1016/j.apmr.2007.08.150

    Article  PubMed  PubMed Central  Google Scholar 

  73. Rose M, Bezjak A (2009) Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples. Qual Life Res 18:125–136

    Article  PubMed  Google Scholar 

  74. Jones JB, Snyder CF, Wu AW (2007) Issues in the design of Internet-based systems for collecting patient-reported outcomes. Qual Life Res 16:1407–1417

    Article  PubMed  Google Scholar 

  75. Aaronson N, Choucair A, Elliott T, Greenhalgh J, Halyard M, Hess R, et al. (2011) User’s guide to implementing patient-reported outcomes assessment in clinical practice. Int Soc Qual Life Res. Available: https://proms.waitematadhb.govt.nz/assets/Uploads/Implementing-PROMs-in-Clinical-Practice.pdf

  76. Moura LMV, Schwamm E, Junior VM, Seitz MP, Hsu J, Cole AJ et al (2016) Feasibility of the collection of patient-reported outcomes in an ambulatory neurology clinic. Neurology 87:2435–2442

    Article  PubMed  PubMed Central  Google Scholar 

  77. Kwong E, Black N (2018) Feasibility of collecting retrospective patient reported outcome measures (PROMs) in emergency hospital admissions. J Patient Rep Outcomes 2:54

    Article  PubMed  PubMed Central  Google Scholar 

  78. O’Hara NN, Richards JT, Overmann A, Slobogean GP, Klazinga NS (2020) Is PROMIS the new standard for patient-reported outcomes measures in orthopaedic trauma research? Injury 51(Suppl 2):S43–S50

    Article  PubMed  Google Scholar 

  79. Rothrock NE, Amtmann D, Cook KF (2020) Development and validation of an interpretive guide for PROMIS scores. J Patient-Rep Outcomes. https://doi.org/10.1186/s41687-020-0181-7

    Article  PubMed  PubMed Central  Google Scholar 

  80. Senders A, Hanes D, Bourdette D, Whitham R, Shinto L (2014) Reducing survey burden: feasibility and validity of PROMIS measures in multiple sclerosis. Mult Scler 20:1102–1111

    Article  PubMed  PubMed Central  Google Scholar 

  81. Segawa E, Schalet B, Cella D (2020) A comparison of computer adaptive tests (CATs) and short forms in terms of accuracy and number of items administrated using PROMIS profile. Qual Life Res 29:213–221

    Article  PubMed  Google Scholar 

  82. Brown T, Chen S, Ou Z, McDonald N, Bennett-Murphy L, Schneider L et al (2022) Feasibility of assessing adolescent and young adult heart transplant recipient mental health and resilience using patient-reported outcome measures. J Acad Consult Liaison Psychiatry 63:153–162

    Article  PubMed  Google Scholar 

  83. Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D et al (2009) How we design feasibility studies. Am J Prev Med 36:452–457

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Thanks to the Rehabilitation Institute of Chicago (now doing business as Shirley Ryan AbilityLab) for financial and organizational support. The Henry B. Betts Innovation award is named after the hospital’s long-serving President, CEO, and Chair of Physical Medicine and Rehabilitation at Feinberg School of Medicine, Northwestern University.

Funding

Shirley Ryan AbilityLab made an award of ~ $25 k as part of an internal competition for innovative ideas.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of North Texas, Denton, TX, 76201, USA

    Riyad Bin Rafiq & Mark V. Albert

  2. Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern University, Evanston, USA

    Susan Yount & David Cella

  3. Ann & Robert H. Lurie Children’s Hospital, Chicago, USA

    Sara Jerousek

  4. Department of Physical Medicine and Rehabilitation, Feinberg School of Medicine, Northwestern University, Evanston, USA

    Elliot J. Roth & Allen W. Heinemann

  5. Department of Biomedical Engineering, University of North Texas, Denton, USA

    Mark V. Albert

  6. Center for Rehabilitation Outcomes Research, Shirley Ryan AbilityLab, Chicago, USA

    Allen W. Heinemann

Authors
  1. Riyad Bin Rafiq
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Susan Yount
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sara Jerousek
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Elliot J. Roth
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Cella
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mark V. Albert
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Allen W. Heinemann
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

AWH developed and directed the study. EJR, SY, and DC consulted on project implementation. SJ served as project manager. RBR and MVA prepared the manuscript. All authors contributed to writing and editing the manuscript and approved the submitted version.

Corresponding author

Correspondence to Riyad Bin Rafiq.

Ethics declarations

Ethics approval and consent to participate

The IRB deemed this activity as part of routine clinical activities, not research, so patient consent was not required.

Consent for publication

Not applicable.

Competing interests

The author's declared that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rafiq, R.B., Yount, S., Jerousek, S. et al. Feasibility of PROMIS using computerized adaptive testing during inpatient rehabilitation. J Patient Rep Outcomes 7, 44 (2023). https://doi.org/10.1186/s41687-023-00567-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s41687-023-00567-x

Keywords