Skip to main content

Patient-reported outcomes feedback report for knee arthroplasty patients should present selective information in a simple design - findings of a qualitative study



Technical innovation to assess patient-reported outcomes (PROs) facilitates their implementation in clinical practice. In particular, mobile applications (apps) allow PROs to be assessed outside of the clinical setting. A patient’s health status can be remotely monitored and evaluated after discharge, and their recovery process tracked. This is of particular interest for patients after knee arthroplasty, as the recovery phase after surgery usually takes place in an outpatient setting and requires a high level of patient engagement. Providing results of PRO assessments to patients in the form of a feedback report could increase patient engagement and may improve communication between health care professionals and patients. The aim of the study is to develop a PRO feedback report for mobile devices that is comprehensible and provides valuable information for patients after knee arthroplasty.


In an iterative development process, our expert group developed two preliminary feedback reports (a text-based version and a graphical display) based on previous research results and practical experience. In a second step, we discussed these reports with orthopedic patients (n = 8) in terms of comprehensibility and value using semi-structured interviews and cognitive debriefing methods. Participants assessed the reports as informative, but had some difficulties in fully comprehending all of the information provided. Based on the feedback from patients, we modified both versions and reduced complexity to increase comprehensibility.


A PRO feedback report for patients for mobile app use has to take account of the heterogeneous user group, particularly demographics such as age and experience with mobile devices. Information should be presented in a simple way to be comprehensible and of value to patients. Technological advancements allow a simple default report to be set, something which enables patients interested in additional information to make customizations.


Patient-reported outcomes (PROs) play an increasingly important role in health care [1]. In clinical practice, PROs are used to assess the health status and wellbeing of patients [2, 3]. In addition, PROs are used as screening and monitoring tools to support health care professionals in providing patient-centered care and facilitate communication between patients and health care professionals [1, 2, 4].

Electronic assessment of PROs is becoming established in research and clinical practice, not in the least because scores can be automatically calculated and provided in real-time to health care professionals [5, 6]. Moreover, mHealth technologies, referring to health care delivery via mobile devices [7], enable PRO assessments at home [8,9,10,11]. The increasing usage of smartphones in all age groups [12] and the high potential seen in mHealth solutions also for an older generation [13], support the new direction of PRO data collection using mobile devices. In this way, health care professionals are able to monitor patients, evaluate their health status and communicate with patients, outside of the clinic [10].

The provision of PRO data to patients, in the form of an automatically generated PRO feedback report, which can be directly accessed by patients after completion, is not common practice. Health care professionals frequently provide information on PRO results during the consultation [6, 14, 15]. In an outpatient setting, patients thereby have delayed access to information on their PRO results, if PROs are collected off-site. However, timely access to PRO scores enables patients to reflect on their progress and promotes active management of their health situation [3].

In orthopedics, the advancement of surgical techniques, as well as cost pressures in clinics, has led to a decrease in the length of hospital stays [16, 17]. The majority of the recovery time after knee replacement takes place in an outpatient setting, which requires a substantial degree of personal responsibility and active disease management by the patient [18]. One way of supporting orthopedic patients to actively manage their recovery phase after surgery is through early health education and training as well as access to health information such as PRO results by providing mHealth solutions [19,20,21,22].

A newly developed education app for knee replacement patients was introduced to support them during their treatment. The app includes information about the post-surgery recovery phase, such as advice on physical activity after surgery, medication and healthy diet. In addition, the PROMIS® measures of Physical Function and Pain Interference are included in the app. Patients are invited to answer these measures at predefined measurement time points. Currently, the PRO scores are only shown in the health care professionals’ dashboard. This information is used to track the recovery phase of individual patients and patient groups. Integration of a PRO feedback report into the app would allow patients to access and monitor their PRO results as well. This information could facilitate patient engagement, improve communication with health care professionals as the PRO feedback report could support patients to address health issues and concerns, based on PRO scores [23].

An automatically generated PRO feedback report for patients using the app is possible, and enables patients to access their PRO results immediately via smartphone. This raises the questions of what kind of information is of value to patients and how a report should look like in order to be easy to understand and evaluate.



The aim of the study is to develop a PRO feedback report for mobile devices, which can be integrated into the newly developed patient education app for patients after knee arthroplasty. This PRO feedback report should be comprehensible and provide valuable information for adult patients after knee arthroplasty, using the PROMIS® health domains of Physical Function and Pain Interference.

Development process of a PRO feedback report for patients

In a first step, we conducted a literature review to examine examples and results of PRO feedback reports that provides the basis for the subsequent development steps. The focus of the literature review was to identify publications describing any kind of PRO feedback report for health care professionals or patients, in particular those addressing issues of how to present PRO data. The literature search was carried out in Medline combining search components for PROs and feedback reports in PubMed. For the search component ‘PRO’, we used the Mesh terms ‘Patient Outcome Assessment’, ‘Patient Reported Outcome Measures’ and ‘Quality of Life’ and combined these by use of the Boolean operator ‘OR’. For the search component ‘feedback report’, we used the Mesh terms ‘Electronic Health Records’, Health Smart Cards’ and ‘Patient Portals’ and the entry terms ‘patient feedback report’ and ‘feedback report’ in the title and abstract and combined these by use of the Boolean operator ‘OR’. Both search components were combined by use of the Boolean operator ‘AND’ and filtered by language (German and English) and by studies concerning human subjects. We screened title and abstracts of the identified publications and reviewed full text of those meeting our inclusion criteria. We also screened general scientific literature on how to present data. In addition, members of the special interest group ‘Quality of Life in Clinical Practice’ of the International Society of Quality of Life Research (ISOQOL) were approached regarding any (personal) experience, articles, guidelines etc. on how to feedback PROs to patients.

In a second step, we developed two preliminary versions of a feedback report (a graphical display and a text-based version) relating to the PROMIS® domains of Physical Function and Pain Interference. Our research group included three experts experienced in quality of life and PRO research with an additional clinical background (a psychologist, a physician specialized in internal medicine and psychosomatic medicine and a nurse). As part of the development, we discussed what information to include in the reports based on the results of the literature review and previous experience with PRO feedback reports for health care professionals, followed by initial drafts and two revision rounds.

In a third step, the two preliminary report versions were discussed with patients in terms of the value of the information, comprehension and interpretation of the report using qualitative methods (semi-structured interviews including cognitive debriefing techniques). The methodological approaches are explained in more detail later.

Finally, we derived reasonable modifications for the PRO feedback report based on the results of the interviews and revised the PRO feedback report versions for patients according to the results of the interviews with patients. We discussed these in light of results of the literature search. An overview of the development process is provided in Fig. 1.

Fig. 1
figure 1

PRO feedback report development process – flow chart

Patient interviews

Orthopedic (primary knee arthroplasty) patients at a university hospital in Germany were invited to participate in our study (convenience sampling) 1 to 2 days after surgery. The sample size was not predetermined, as we were aiming to reach data and thematic saturation [24]. We therefore continued recruitment until no additional codes or categories emerged from the interviews (thematic/data saturation).

Patients were eligible to participate if they were over 18 years, had sufficient knowledge of German (spoken and written), were able to understand the purpose of the study and gave written informed consent. Patients with cognitive limitations, or insufficient knowledge of German were excluded from the study. Ethical approval was obtained from the ethics committee of the university hospital.

The first author conducted all semi-structured interviews with patients according to the interview guideline during the hospital stay. Patients received printed copies of both report versions for both health domains in random order, as the patient app does not currently include any feedback report for patients. They were initially invited to look at the paper copies of the feedback report on their own. Cognitive debriefing methods for probing such as think-aloud and comprehension were used [25]. To assess comprehension of the feedback report drafts, patients were engaged to explain what they see, how they interpret the different elements, and what information they get out of the report. If an element of the report was not addressed by a patient, we explicitly pointed it out. Further, patients were asked which report style they prefer and if a feedback report would be of value and why.

Interviews were tape recorded and transcribed verbatim. All interviews were analyzed using a content analysis approach. Based on the guideline developed for the interviews, we pre-specified a category structure (Table 4) that is ‘general value of the report’, ‘favored report style’ and ‘comprehension’ and ‘informative content’ of both report styles. Additional categories were created, if necessary. All interview transcripts were read multiple times and codes were extracted and assigned to one of the specified categories. We subsequently summarized and discussed the pre-defined and created categories and included codes in detail. The interview analysis was performed using MAXQDA10 software for Windows.


Development process of a PRO feedback report for patients

Review of the literature revealed only few studies focusing on PRO feedback reports in general, and on how to report PRO scores in particular. Results from previous studies showed that patients prefer short and condensed information [26] using simple language [27]. An efficient way to present longitudinal PRO scores is by use of graphs such as bar charts or line graphs [28,29,30,31,32,33,34,35,36,37,38,39], alternatively PRO results could be reported by use of tables [40, 41] or textual reports [31, 39]. A text-based report is the least preferred option for patients [31] but is less susceptible to misinterpretation [39]. For graphs, meaningful labelling of axes is recommended. For example, descriptive labeling, and harmonization of directionality, where higher indicates better facilitates interpretation of PRO scores [35,36,37, 41, 42]. Further, highlighting results of importance helps to focus attention and support the interpretation of results [30, 35, 36]. Color-shading, threshold lines or circles around important scores are suggested to increase readability and interpretability of graphical displays [14, 15, 29, 35, 36]. Additional information such as comparison to a reference population could be of value for some patients, but might be critical to others [34, 41, 43]. An overview of the results of the literature review is provided in Table 1.

Table 1 PRO feedback reports - literature overview

In the initial development phase, we decided to provide feedback reports on a domain level (PROMIS® health domains Physical Function and Pain Interference), rather than a combined report, to increase readability on mobile devices, which was in concordance with the results of the literature review. We were aiming to include the feedback reports in an existing patient app that already includes a message function. Thus, we decided to develop two different report styles, a text-based report, which could be sent like using the message function, and a graphical display, which would be shown after the PROM completion. The text-based report was considered a pragmatic solution and could be implemented without additional effort, whereas, reprogramming would be needed to incorporate the graphical display. Our research team determined the following information to include in the PRO feedback report: the domain T-score, reference to the norm population, score thresholds and the predicted disease trajectory, as well as the opportunity to share information with health care professionals by use of the text message function or by a push button.

The text-based version included general information regarding the assessment, estimation of the current individual level of symptoms or function (including examples of limitations), and reference to the norm population. Patients were informed to contact their health care team in case of unexpected PRO changes or related questions. We used the thresholds provided by the PROMIS® Health Organization [49] to categorize value ranges. We created four categories for the Pain Interference health domain (group 1: T-score < 55; group 2: T-score of 55–60; group 3: T-score of 61–70; group 4: T-score > 70) and five categories for Physical Function (group 1: T-score < =30; group 2: T-score of 31–40; group 3: T-score of 41–45; group 4: T-score of 45–65; group 5: T-score > 65). Text messages were adapted according to the respective category. Patients scoring within the range of a category would receive the corresponding text message.

The graphical display included information on the individual T-score, the reference to the norm population, score thresholds and information on the predicted disease trajectory. The information was included on a line graph, where scores are plotted over time. On the x-axis, the measurement date as well as the time passed since the knee arthroplasty are displayed. We included two y-axis. The right-hand side y-axis only included the numerical scale. Descriptive labels were added to the left-hand side y-axis, referring to norm population scores. In addition, we added a rainbow-colored background from red (bottom) to green (top) to the line chart to visualize the grading of the individual scores. Whereas higher scores of Physical Function are better outcomes, higher scores of Pain Interference are worse outcomes. In order to account for this, we reversed the scale on the right y-axis for Pain Interference to harmonize the direction of the line graphs, i.e. the higher the better. We overlaid the line graph with an additional shaded area indicating the projected trajectory of the health domain. A button to share the information with health care professionals was included. For Pain Interference, we added information on the relative frequency of pain medication according to measurement points and percentage of change in pain medication in reference to the previous value. For Physical Function, we added information on immune activity including a rating of inflammatory markers (low, medium or high). The preliminary versions of the PRO feedback reports for the domain Pain Interference are displayed in Fig. 2 and Table 2.

Fig. 2
figure 2

Preliminary version of the graphical PRO feedback reports for the health domain of Pain Interference

Table 2 Preliminary version of the text-based PRO feedback report for the health domain of Pain Interference according to grouped T-score categories

Patient interviews

We invited 15 adult patients, who have underwent knee arthroplasty to participate in our study between September 2018 and January 2019. Seven patients who declined to participate said they were not interested in the study or were already participating in another research study. Non-responders were predominantly male (n = 6). We interviewed eight patients after knee arthroplasty, five women (62.5%) and three men (37.5%) aged between 60 and 80. We continued interviews until generated data was repetitive and no new codes or categories emerged during the analyses. This was the case after five interviews. The interviews took 24 min and 33 s on average, ranging from 19:59 to 28:38 min. Sociodemographic details of the study participants are provided in Table 3.

Table 3 Sociodemographic details (n = 8)

To achieve the aim of the study and the purpose of the interviews, we predefined four categories and four sub-categories, which were general value of the report, favored feedback report style, graphical display, including the sub-categories comprehension and informative content, and text-based report including the sub-categories comprehension and informative content. During the coding process, the additional category aspects related to the individual person or situation emerged.

The general value of the reports was rated as moderate. While all patients thought such a report would be interesting and they would look at it, only one patient saw personal gain from using the report as a communication tool with health care professionals and as a self-monitoring tool. One participant stated that the majority of people would not understand what this report is about. Another participant stated that the idea of such a feedback report is good, but probably more suited to a younger generation. Another aspect mentioned by one participant was the course of treatment after knee surgery in Germany. This includes an extensive rehabilitation phase in a specialized clinic with daily treatments and regular examinations. The participant questioned the benefit of such a report given the continuous monitoring of the recovery process in Germany.

Participants had different opinions regarding the favored feedback report style. Three participants were in favor of the graphical display and two participants were undecided but tended toward the text-based version. Participants in favor of the graphical display argued that it is easy and quick to get the relevant information from the line graph. Those undecided stated that the graphical display looks more appealing, but the text-based version is easier to understand and most people are used to read short text messages.

The text-based report was easy to understand and none of the patients had questions regarding comprehension. Participants assessed the report as informative, but thought that some information was repetitive. Issues regarding the reference to the norm population were raised, as the percentage of people reporting similar symptoms or functional limitations would not be relevant. Difficulties in assigning a meaning to the number was also raised. One patient was even put off by it, feeling it would put certain pressure on someone to belong to the majority. Overall, the text-based version was considered to be too long, Participants therefore suggested shortening the text.

The graphical display of the reports was perceived to be complex due to the inclusion of a lot of information. Participants were concerned that this format may not be appropriate for an older generation. In terms of comprehension, all participants correctly understood the line graph and were able to interpret the scores. However, some patients needed some initial guidance on how to read a line graph. The rainbow-colored background was understood by all participants. With one exception, participants automatically attached value to the different shades of color. However, the shaded area indicating the projected trajectory of the health domains was not clear to patients. One participant did not ascribe any value to this area. Others understood this area as the range of possible scores, or as a way to draw attention to the area of the respective scores. Just two respondents interpreted the improvement over time as a prognosis for recovery after surgery. Only one participant understood the additional information regarding pain medication and related the information to the Pain Interference T-score. The additional information on immune activity was deemed to be too technical. Participants guessed that it might be related to the immune system, but could not explain what it meant, or relate this information to the Physical Function health domain. The function to share the information with health care professionals was clear to all patients.

During the interviews, five participants emphasized the relevance of aspects related to the individual person or situation. It was thought that a comparison to the norm population would not be of particular interest to patients after knee arthroplasty, as the individual before and after comparison of physical function or level of pain is of particular importance. Furthermore, reference to the knee replacement would help to relate the report to the individual patient’s situation. Statements like the activities which one might not be able to do due to the limitations of physical function might not reflect the individual’s activities. The report therefore loses the personal touch. One patient mentioned that the measurement time points should be flexible so that the patient can decide on the frequency of assessments.

An overview of the categories including the number of codes per category and examples of codes is presented in Table 4.

Table 4 Coding structure of the interviews, number of codes and coding example

Revisions of PRO feedback reports for patients

As a result of the interview analysis, we modified the PRO feedback reports for patients to increase comprehension and the value of the reports. Minimal adjustments were done to the text-based version. This version was comprehensible to all patients, but the text was too long and some of the information was not seen as valuable to patients. We deleted the reference to the norm population. A sentence stating what the score measured as well as a sentence linking the assessment to the patient’s recent knee arthroplasty were added.

The graphical display required more extensive revisions to decrease complexity and thereby increase comprehension and value. We excluded the additional information regarding immune activity and pain medication as this information was misunderstood or not understood by patients and was not interpreted in reference to the measured health domain. The button to share the information with clinicians was enlarged to improve user-friendliness. In addition, we added a green tick to confirm that data were transmitted successfully. We reduced clutter within the line graph to increase readability and comprehension. The reference curve of the expected disease trajectory became an optional setting. The rainbow-colored background was reduced to a bar on the left-hand side y-axis. The data points plotted on the line graph include the scores, as well as the color corresponding to the rainbow-colored bar added to the y-axis. In addition, we decided to enlarge the most recent score to draw attention to it. The revised graphical display for the Pain Interference domain is depicted in Fig. 3.

Fig. 3
figure 3

Final graphical display of the PRO feedback reports for the health domain of Pain Interference


This study aimed to develop a PRO feedback report for adult knee arthroplasty patients that is easy to understand while providing valuable information. Results showed that the group of knee arthroplasty patients is heterogeneous in terms of age and experience with mobile devices and so are their demands on a PRO feedback report.

Based on the patient interviews, it was not possible to appoint a preferred report style. In previous studies, patients preferred graphical displays over a text-based report [31, 39]. Patients reported that the graphical display looked more appealing but the text-based feedback report version was easy to understand, even if partly too lengthy. Corresponding with our finding, Fried and colleagues [26] reported that feedback reports for patients should contain short and useful information instead of lengthy explanations.

Presenting data over time allows users to evaluate their individual progress and to compare current and previous scores. Patients valued the line graph presenting scores over time as important information, something which is not provided in our text-based version. This finding corresponds to the results reported by Brundage et al. [30] that patients preferred the depiction of multiple time points.

Interestingly, participants considered the reference to the norm population in the text-based version to not be of value and hence not relevant. However, they viewed the labels added to the y-axis and the color-coding within the graphical display positively. This implicit reference to the norm population supports the evaluation of individual domain scores. Similar findings were reported in previous studies that the inclusion of information referring to a reference group, such as labels of axis or integration of thresholds or a norm line, is helpful in evaluating the results [15, 29, 33, 36].

During the interviews, initial guidance on how to read the line graph was helpful to patients, and they suggested that some guidance should be given to all users before receiving the feedback report. Baldwin and colleagues [27] outlined that information should be provided in a simple way, including a simple interface design. This means that if additional guidance is required, the interface design might be too complex. However, a brief introduction to the PRO feedback report, including the functionality of the graphical display, could be implemented in the mobile app. The user could refer back to this information when needed. This training on how to use and interpret the feedback report could help to increase the utilization of the report among users [50].

The graphical display of the feedback report included additional information that overwhelmed participants. Considering the conceptual framework ‘mHealth for older users’ described by Wildenbos et al. [51], vision problems, cognitive restriction as well as attitudes and health-technology literacy might have influenced the perceived overload of the graphical display. Further, the use of medical terms instead of layman’s terms and the presentation of the clinical information (pain medication and immune activity) was difficult for participants to understand and prone to misinterpretation. This information was removed from the report to ensure that appropriate information is provided according to the health literacy level of patients using simple language, crucial aspects for a user-friendly interface [27, 51].

Another aspect study participants addressed during the interviews was the penetration of smartphone use within this specific target group. Only a few study participants had a smartphone and only one patient stated that he uses his smartphone for purposes beyond communication (phone calls and messages). When designing PRO feedback for mobile app use, two major challenges have to be considered. Firstly, the penetration of smartphone use in a specific age group has to be evaluated in a national context [52, 53]. Secondly, experience and user behavior with mobile devices is closely linked to the health-technology literacy level of the target group [51, 54, 55]. The field of mobile applications offers technological opportunities such as customizations and user guidance. In this way, it is possible to adjust the default settings of the graphical design of the report so that the interested and experienced user can fade-in additional information, but the default design does not overwhelm the inexperienced user [33].


This study has some limitations that have to be considered with regard to the study results. There were some limitations to the study design. We did not include patient research partners in the study and discussed only the two developed feedback report styles with patients without testing alternative design formats including less or more information. The information included in the feedback report and the design would have been different by including patient research partners within the study. Furthermore, there would have been different formats to present the information, especially for the graphical display of the report. Thus, we are not able to make any statement in terms of the optimal design of PRO feedback reports for patients. However, the study’s goal was to develop a feedback report for mobile devices that provides valuable information to patients after knee arthroplasty, while being easy to understand and to evaluate. We have to test the developed feedback report to evaluate functioning and the practical value of sharing such PRO information with patients in future studies. A feasibility study to investigate how frequently patients look at the report and how the information facilitate communication between patients and health care professionals during the recovery period is needed.

We initially planned to conduct focus groups instead of semi-structured interviews to induce discussions among participants. We invited patients to take part in a focus group, but all patients preferred an individual interview. The coding was conducted by one researcher only. To increase reliability, all codes were discussed with a second researcher and if necessary, codes were removed from a category or assigned to another.

In addition, we were faced with a high non-responder rate in our study and recruited a small number of participants. However, we consider the number of patients to be acceptable given the narrow scope of the study and the saturation of data after five interviews. Data generated in the last interviews were repetitive and no new codes or categories emerged, which lead to the assumption of data and thematic saturation [24]. Nevertheless, the results of this study have to be interpreted cautiously. The findings are limited to the specific setting of the study and cannot be generalized.


The development of a PRO feedback report for knee arthroplasty patients for mobile app use is challenging due to the heterogeneous group of patients. The range of health-technology literacy levels within the target group has to be considered when developing an easy to understand and informative PRO feedback report for patients. The amount of information provided to patients has to be balanced to minimize the complexity of the report while maximizing the value for patients. A graphical display including short explanatory texts seems to be the most promising approach as a simple default version can be displayed, while additional information can be set as supplementary information for interested patients.

Availability of data and materials

The dataset used and analyzed during the current study is available from the corresponding author on reasonable request.



Patient-reported outcomes


Patient-reported outcomes measure


Patient-Reported Outcomes Measurement Information System


  1. Higginson, I. J., & Carr, A. J. (2001). Measuring quality of life - using quality of life measures in the clinical setting. Br Med J, 322(7297), 1297–1300.

    Article  CAS  Google Scholar 

  2. Greenhalgh, J. (2009). The applications of PROs in clinical practice: What are they, do they work, and why? Quality of life research : an international journal of quality of life aspects of treatment, care and rehabilitation, 18(1), 115–123.

    Article  Google Scholar 

  3. Greenhalgh, J., Dalkin, S., Gooding, K., Gibbons, E., Wright, J., Meads, D., Black, N., Valderas, J. M., & Pawson, R. (2017). Health services and delivery research. In: Functionality and feedback: A realist synthesis of the collation, interpretation and utilisation of patient-reported outcome measures data to improve patient care. NIHR Journals Library Copyright (c) Queen's Printer and Controller of HMSO 2017. This work was produced by Greenhalgh et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK., Southampton (UK).

  4. Snyder, C. F., Jensen, R. E., Segal, J. B., & Wu, A. W. (2013). Patient-reported outcomes (PROs): Putting the patient perspective in patient-centered outcomes research. Med Care, 51(8 Suppl 3), S73–S79.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Rose, M., & Bezjak, A. (2009). Logistics of collecting patient-reported outcomes (PROs) in clinical practice: An overview and practical examples. Quality of life research : an international journal of quality of life aspects of treatment, care and rehabilitation, 18(1), 125–136.

    Article  Google Scholar 

  6. Haverman, L., van Rossum, M. A. J., van Veenendaal, M., van den Berg, J. M., Dolman, K. M., Swart, J., Kuijpers, T. W., & Grootenhuis, M. A. (2013). Effectiveness of a web-based application to monitor health-related quality of life. Pediatrics, 131(2), E533–E543.

    Article  PubMed  Google Scholar 

  7. Malvey, D., & Solovnsky, D. (2014). mHealth: Transforming healthcare. New York: Springer Science+Business Media New York.

    Book  Google Scholar 

  8. Weinstein, R. S., Lopez, A. M., Joseph, B. A., Erps, K. A., Holcomb, M., Barker, G. P., & Krupinski, E. A. (2014). Telemedicine, Telehealth, and Mobile health applications that work: Opportunities and barriers. Am J Med, 127(3), 183–187.

    Article  PubMed  Google Scholar 

  9. Dobkin, B. H., & Dorsch, A. (2011). The promise of mHealth: Daily activity monitoring and outcome assessments by wearable sensors. Neurorehabil Neural Repair, 25(9), 788–798.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Hamine, S., Gerth-Guyette, E., Faulx, D., Green, B. B., & Ginsburg, A. S. (2015). Impact of mHealth chronic disease management on treatment adherence and patient outcomes: A systematic review. J Med Internet Res, 17(2), e52.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Jensen, R. E., Gummerson, S. P., & Chung, A. E. (2016). Overview of patient-facing Systems in Patient-Reported Outcomes Collection: Focus and Design in Cancer Care. Journal of oncology practice, 12(10), 873–875.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Jiang J (2018) Millenials stand out for their technology use, but older generations also embrace digital life. Accessed Sept 5 2019.

    Google Scholar 

  13. Joe, J., & Demiris, G. (2013). Older adults and mobile phones for health: A review. J Biomed Inform, 46(5), 947–954.

    Article  PubMed  Google Scholar 

  14. Barthel, D., Fischer, K. I., Nolte, S., Otto, C., Meyrose, A. K., Reisinger, S., Dabs, M., Thyen, U., Klein, M., Muehlan, H., Ankermann, T., Walter, O., Rose, M., & Ravens-Sieberer, U. (2016). Implementation of the Kids-CAT in clinical settings: a newly developed computer-adaptive test to facilitate the assessment of patient-reported outcomes of children and adolescents in clinical practice in Germany. Quality of life research : an international journal of quality of life aspects of treatment, care and rehabilitation, 25(3), 585–594.

    Article  CAS  Google Scholar 

  15. Rothrock, N. E., Bass, M., Blumenthal, A., Gershon, R. C., Hanson, B., Joeris, A., Kaat, A., Morrison, S., O'Toole, R. V., Patel, S., Stover, M., Weaver, M. J., White, R., Varela Diaz, M., & Vrahas, M. S. (2019). AO patient outcomes center: Design, implementation, and evaluation of a software application for the collection of patient-reported outcome measures in orthopedic outpatient clinics. JMIR formative research, 3(2), e10880.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Auyong, D. B., Allen, C. J., Pahang, J. A., Clabeaux, J. J., MacDonald, K. M., & Hanson, N. A. (2015). Reduced length of hospitalization in primary Total knee Arthroplasty patients using an updated enhanced recovery after orthopedic surgery (ERAS) pathway. J Arthroplast, 30(10), 1705–1709.

    Article  Google Scholar 

  17. Kumar, P. J., McPherson, E. J., Dorr, L. D., Wan, Z., & Baldwin, K. (1996). Rehabilitation after total knee arthroplasty: A comparison of 2 rehabilitation techniques. Clin Orthop Relat Res, 331, 93–101.

    Article  Google Scholar 

  18. Chapter 3: A practical guide to improving patient outcomes (2000). Orthopedic nursing 19 Suppl:22–28.

  19. Holman, H., & Lorig, K. (2004). Patient self-management: A key to effectiveness and efficiency in Care of Chronic Disease. Public Health Rep, 119(3), 239–243.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Cameron, A. (1998). Patient self-management. PharmacoEconomics & Outcomes News Weekly, 187(1), 3–5.

    Article  Google Scholar 

  21. Bodenheimer, T., Lorig, K., Holman, H., & Grumbach, K. (2002). Patient self-management of chronic disease in primary care. JAMA, 288(19), 2469–2475.

    Article  PubMed  Google Scholar 

  22. Kim, K., Pham, D., & Schwarzkopf, R. (2016). Mobile application use in monitoring patient adherence to perioperative Total knee Arthroplasty protocols. Surgical technology international, 28, 253–260.

    PubMed  Google Scholar 

  23. Santana, M. J., & Feeny, D. (2014). Framework to assess the effects of using patient-reported outcome measures in chronic care management. Quality of life research : an international journal of quality of life aspects of treatment, care and rehabilitation, 23(5), 1505–1513.

    Article  Google Scholar 

  24. Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018). Saturation in qualitative research: Exploring its conceptualization and operationalization. Qual Quant, 52(4), 1893–1907.

    Article  PubMed  Google Scholar 

  25. Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of life research : an international journal of quality of life aspects of treatment, care and rehabilitation, 12(3), 229–238.

    Article  Google Scholar 

  26. Fried, T. R., Redding, C. A., Robbins, M. L., Paiva, A. L., O'Leary, J. R., & Iannone, L. (2016). Development of personalized health messages to promote engagement in advance care planning. J Am Geriatr Soc, 64(2), 359–364.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Baldwin, J. L., Singh, H., Sittig, D. F., & Giardina, T. D. (2017). Patient portals and health apps: Pitfalls, promises, and what one might learn from the other. Healthcare (Amsterdam, Netherlands), 5(3), 81–85.

    Article  Google Scholar 

  28. Bantug, E. T., Coles, T., Smith, K. C., Snyder, C. F., Rouette, J., & Brundage, M. D. (2016). Graphical displays of patient-reported outcomes (PRO) for use in clinical practice: What makes a pro picture worth a thousand words? Patient Educ Couns, 99(4), 483–490.

    Article  PubMed  Google Scholar 

  29. Gilbert, A., Sebag-Montefiore, D., Davidson, S., & Velikova, G. (2015). Use of patient-reported outcomes to measure symptoms and health related quality of life in the clinic. Gynecol Oncol, 136(3), 429–439.

    Article  PubMed  Google Scholar 

  30. Brundage, M. D., Smith, K. C., Little, E. A., Bantug, E. T., & Snyder, C. F. (2015). Communicating patient-reported outcome scores using graphic formats: Results from a mixed-methods evaluation. Quality of life research : an international journal of quality of life aspects of treatment, care and rehabilitation, 24(10), 2457–2472.

    Article  Google Scholar 

  31. Cronin, R. M., Conway, D., Condon, D., Jerome, R. N., Byrne, D. W., & Harris, P. A. (2018). Patient and healthcare provider views on a patient-reported outcomes portal. Journal of the American Medical Informatics Association : JAMIA, 25(11), 1470–1480.

    Article  PubMed  Google Scholar 

  32. Demiris, G., & Thompson, H. (2011). Smart homes and ambient assisted living applications: From data to knowledge-empowering or overwhelming older adults? Contribution of the IMIA smart homes and Ambiant assisted living working group. Yearbook of medical informatics, 6, 51–57.

    CAS  PubMed  Google Scholar 

  33. Izard, J., Hartzler, A., Avery, D. I., Shih, C., Dalkin, B. L., & Gore, J. L. (2014). User-centered design of quality of life reports for clinical care of patients with prostate cancer. Surgery, 155(5), 789–796.

    Article  PubMed  Google Scholar 

  34. McNair, A. G., Brookes, S. T., Davis, C. R., Argyropoulos, M., & Blazeby, J. M. (2010). Communicating the results of randomized clinical trials: Do patients understand multidimensional patient-reported outcomes? Journal of clinical oncology : official journal of the American Society of Clinical Oncology, 28(5), 738–743.

    Article  Google Scholar 

  35. Smith, K. C., Brundage, M. D., Tolbert, E., Little, E. A., Bantug, E. T., & Snyder, C. F. (2016). Engaging stakeholders to improve presentation of patient-reported outcomes data in clinical practice. Supportive care in cancer : official journal of the Multinational Association of Supportive Care in Cancer, 24(10), 4149–4157.

    Article  Google Scholar 

  36. Snyder, C. F., Smith, K. C., Bantug, E. T., Tolbert, E. E., Blackford, A. L., & Brundage, M. D. (2017). What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability. Cancer, 123(10), 1848–1859.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Snyder, C., Smith, K., Holzner, B., Rivera, Y. M., Bantug, E., & Brundage, M. (2019). Making a picture worth a thousand numbers: Recommendations for graphically displaying patient-reported outcomes data. Quality of life research : an international journal of quality of life aspects of treatment, care and rehabilitation, 28(2), 345–356.

    Article  Google Scholar 

  38. Wu, A. W., Kharrazi, H., Boulware, L. E., & Snyder, C. F. (2013). Measure once, cut twice--adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. J Clin Epidemiol, 66(8 Suppl), S12–S20.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Brundage, M., Feldman-Stewart, D., Leis, A., Bezjak, A., Degner, L., Velji, K., Zetes-Zanatta, L., Tu, D., Ritvo, P., & Pater, J. (2005). Communicating quality of life information to cancer patients: A study of six presentation formats. Journal of clinical oncology : official journal of the American Society of Clinical Oncology, 23(28), 6949–6956.

    Article  CAS  Google Scholar 

  40. Fritz, F., Stander, S., Breil, B., Riek, M., & Dugas, M. (2011). CIS-based registration of quality of life in a single source approach. BMC medical informatics and decision making, 11, 26.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Wu, A. W., White, S. M., Blackford, A. L., Wolff, A. C., Carducci, M. A., Herman, J. M., & Snyder, C. F. (2016). Improving an electronic system for measuring PROs in routine oncology practice. Journal of cancer survivorship : research and practice, 10(3), 573–582.

    Article  Google Scholar 

  42. Krogstad, H., Sundt-Hansen, S. M., Hjermstad, M. J., Hagensen, L. A., Kaasa, S., Loge, J. H., Raj, S. X., Steinsbekk, A., & Sand, K. (2019). Usability testing of EirV3-a computer-based tool for patient-reported outcome measures in cancer. Supportive care in cancer : official journal of the Multinational Association of Supportive Care in Cancer, 27(5), 1835–1844.

    Article  Google Scholar 

  43. Grossman, L. V., & Mitchell, E. G. (2017). Visualizing the patient-reported outcomes measurement information system (PROMIS) measures for clinicians and patients. AMIA Annual Symposium proceedings AMIA Symposium, 2017, 2289–2293.

    PubMed  Google Scholar 

  44. Aldekhyyel, R. N., Melton, G. B., Lindgren, B., Wang, Y., & Pitt, M. B. (2018). Linking pediatrics patients and nurses with the pharmacy and electronic health record system through the inpatient television: A novel interactive pain-management tool. Hospital pediatrics, 8(9), 588–592.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Harle, C. A., Listhaus, A., Covarrubias, C. M., Schmidt, S. O., Mackey, S., Carek, P. J., Fillingim, R. B., & Hurley, R. W. (2016). Overcoming barriers to implementing patient-reported outcomes in an electronic health record: A case report. Journal of the American Medical Informatics Association : JAMIA, 23(1), 74–79.

    Article  PubMed  Google Scholar 

  46. Krogstad, H., Brunelli, C., Sand, K., Andersen, E., Garresori, H., Halvorsen, T., Haukland, E. C., Jordal, F., Kaasa, S., Loge, J. H., Lohre, E. T., Raj, S. X., & Hjermstad, M. J. (2017). Development of EirV3: A computer-based tool for patient-reported outcome measures in Cancer. JCO clinical cancer informatics, 1, 1–14.

    Article  PubMed  Google Scholar 

  47. Schwartzberg L (2016) Electronic patient-reported outcomes: The time is ripe for integration into patient care and clinical research. American Society of Clinical Oncology educational book American Society of Clinical Oncology Annual Meeting 35:e89-e96. doi: 10.1200/edbk_158749.

  48. Sokka, T. (2016). Go, go, GoTreatIT! Clin Exp Rheumatol, 34(5 Suppl 101), S91–s95.

    PubMed  Google Scholar 

  49. PROMIS Health Organisation (2019) PROMIS Score Cut Points - General guidelines for interpreting PROMIS scores Available via Health Measures. Accessed 15 May 2019.

    Google Scholar 

  50. Amante, D. J., Hogan, T. P., Pagoto, S. L., & English, T. M. (2014). A systematic review of electronic portal usage among patients with diabetes. Diabetes Technol Ther, 16(11), 784–793.

    Article  PubMed  Google Scholar 

  51. Wildenbos, G. A., Peute, L. W., & Jaspers, M. W. (2015). A framework for evaluating mHealth tools for older patients on usability. Studies in health technology and informatics, 210, 783–787.

    Article  CAS  PubMed  Google Scholar 

  52. Berenguer, A., Goncalves, J., Hosio, S., Ferreira, D., Anagnostopoulos, T., & Kostakos, V. (2017). Are smartphones ubiquitous?: An in-depth survey of smartphone adoption by seniors. IEEE Consumer Electronics Magazine, 6(1), 104–110.

    Article  Google Scholar 

  53. Sharareh, B., & Schwarzkopf, R. (2014). Effectiveness of telemedical applications in postoperative follow-up after total joint arthroplasty. The Journal of arthroplasty, 29(5), 918–922.e911.

    Article  PubMed  Google Scholar 

  54. Grindrod, K. A., Li, M., & Gates, A. (2014). Evaluating user perceptions of mobile medication management applications with older adults: A usability study. JMIR mHealth and uHealth, 2(1), e11.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Arnhold, M., Quade, M., & Kirch, W. (2014). Mobile applications for diabetics: A systematic review and expert-based usability evaluation considering the special requirements of diabetes patients age 50 years or older. J Med Internet Res, 16(4), e104.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


Our special thanks goes to all patients who participated in this study. We thank Dr. A. Figura for her input during the development phase and Dr. L. Renner and Dr. S. Kirschbaum for their support in patient recruitment. Further, we would like to thank Wellframe for their insights and cooperation. We thank members of the special interest group Clinical Practice of the International Society of Quality of Life Research for their helpful hints and sharing their experience and knowledge. Finally, we thank Smith & Nephew Orthopedics for funding this research project. We acknowledge support from the German Research Foundation (DFG) and the Open Access Publication Fund of Charité – Universitätsmedizin Berlin.


This study was funded by Smith & Nephew Orthopaedics.

Author information

Authors and Affiliations



KF, DDF and MR developed the idea and designed the study. KF collected the data. MR and KF analyzed the data. KF wrote the manuscript with support from MR. All authors reviewed and approved the final manuscript.

Corresponding author

Correspondence to Kathrin I. Fischer.

Ethics declarations

Ethics approval and consent to participate

The study was conducted adhering to the Declaration of Helsinki and its later amendments. Ethical approval was obtained from ethics committees at Charité Universitaetsmedizin Berlin (EA1/114/18), Germany. All participants gave written informed consent.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fischer, K.I., De Faoite, D. & Rose, M. Patient-reported outcomes feedback report for knee arthroplasty patients should present selective information in a simple design - findings of a qualitative study. J Patient Rep Outcomes 4, 6 (2020).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: