Author | Primary study goal | Study population | Presenting PROMs data | Visualization | ||||
---|---|---|---|---|---|---|---|---|
Type of PROMs | What is presented? | Graphic visualisation format | Comparator | Outcomes of included studies preferences | Interpretation accuracy | |||
Individual level PROMs data visualization, patient | ||||||||
Brundage [16] | To investigate the interpretability of current PRO data presentation formats | N = 50 patients with variety of cancer diagnoses; N = 20 clinicians in active practice, Johns Hopkins Clinical Research Network (JHCRN)* | EORTC-QLQ-C30 scores | Individual scores and group means | Line graphs of mean scores | Previous scores | Simple line graphs for overall ease-of-understanding and usefulness | Patients accuracy ranged from 64–96% (line graphs questions) |
Tabulated scores | 92% preferred formats displaying multiple time points | A graph trending down with better = higher scores was correctly interpreted by 96%. A graph trending down up with better = lower scores was correctly interpreted by 64% | ||||||
Bubble plots of scores at a point in time | ||||||||
Heat map | ||||||||
Damman [14] | To investigate: | Interviews: patients with Parkinson's disease (N = 13) and clinicians (N = 14) | Not specified | Individual scores | Line graph | Patients with the same age, gender and disease duration | Bar chart is preferred (57.2%) compared to line graphs (42.3%) | What hindered easy comprehension: the use of a “higher = worse” directionality and comparative information of patients that are similar in terms of age, gender and disease progression |
(a) How patients and clinicians think about using PROMs during consultations; | Survey: patients (N = 115), the Netherland | Bar graph | Line and bar charts were interpreted most often correctly, compared with more “evaluative” formats like smileys and colors | |||||
(b) For which purpose patients and clinicians use PROMs during consultations; | Line graph with comparative data over time (i.e. average scores of similar patients) | Individual PROMs scores over time were interpreted more often correctly when presented in a bar chart (87.8%) compared to a line graph (74.3%) | ||||||
(c) How patients interpret PROMs information presented in various formats | ||||||||
Fischer [1] | To develop a PRO feedback report for mobile devices that is comprehensible and provides valuable information for patients after knee arthroplasty | Orthopedic patients (N = 8), Germany | Multiple (literature) | Individual scores | Text-based report and a graphical display (line graph, where scores are plotted over time, over a rainbow-colored background from red (bottom) to green (top) to visualize the grading of the individual scores) | Norm population | Short and condensed information using simple language (literature) | A text-based report is the least preferred but less susceptible to misinterpretation (literature) |
PROMIS (development) | An efficient way to present longitudinal PRO scores: graphs such as bar or line graphs (literature) | All participants correctly understood the line graph and were able to interpret the scores. Some needed some initial guidance on how to read a line graph | ||||||
Those (n = 3) in favor of graphs: easy and quick to get the relevant information from the line graph | The rainbow-colored background was understood by all participants | |||||||
The text-based (n = 2) version is easier to understand and most people are used to read short text messages | ||||||||
Geerards [26] | To assess the impact of tailored multimodal feedback and computerized adapted testing (CAT) on user experience in HRQoL assessment using validated PROMs | N = 1386 participants from the general population, United Kingdom (UK) | World Health Organization Quality of Life assessment (WHOQOL) | Individual scores | Graphical only | N/A | Respondents thought the questionnaire with graphical and text-based feedback was more interesting compared with no feedback assessment, whereas providing only graphical feedback did not make the questionnaire more interesting | 82.4% of patients thought the graphical feedback was accurate |
Graphical and adaptive text-based feedback | 92.9% of patients thought the graphical feedback was clear | |||||||
Graphs: Separate horizontal bar charts for 4 domains | ||||||||
Text: What each domain reflects, how score corresponds to average scores, and what score might mean | ||||||||
Grossman [27] | To identify the design requirements for an interface that assists patients with PRO survey completion and interpretation; to build and evaluate the interface of PROMs feedback | Interview: N = 13 patients with heart failure and N = 11 clinicians, study location or country was not described | Health IT Usability Evaluation Scale (Health-ITUES) | Individual scores | Small cards: Short sentence describing a severe symptom, which when clicked on provides textual educational information | N/A | Perceiving the mockup as useful and easy-to-use | Half of the participants failed to interpret the bar chart correctly, and even participants who could read it often required multiple attempts |
Usability testing: N = 12 patients with heart failure | Large cards: | Patients preferred visualizations with brief text descriptions | ||||||
Symptom name and description, visual representation of its severity, and a link to textual educational information | ||||||||
Graph: Bar chart (lists patient’s symptoms from most to least severe, with symptom’s severity scores) | ||||||||
Hartzler [6] | To conduct a HCD to engage patients, providers, and interaction design experts in the development of visual “PRO dashboards” that illustrate personalized trends in patients’ HRQoL following prostate cancer treatment | Focus groups (N = 60 patients) | Not specified | Individual scores | Pictographs | The dashboard compares patients’ trends with trends from “men like you” matched by default by age and treatment derived from a previously published prostate cancer cohort | Pictographs less helpful than bar charts, line graphs, or tables (P < .001) | Pictographs might reach patients with limited literacy |
N = 50 prostate cancer patients and N = 50 clinicians, study location or country was not described | Bar charts | Bar charts and line graphs are most preferred | Some patients expressed concern over inclusion of comparison scores without representation of data variability (e.g., confidence intervals, error bars), while others preferred simpler charts and graphs | |||||
Line graphs | ||||||||
Hildon [2] | To explore patients’ views of different formats and content of data displays of PROMs | N = 45 patients undergone or planning knee surgery in six focus groups, UK | Oxford Hip Score (OHS) | Individual scores | Different formats (table, bar chart, caterpillar and funnel plot) | N/A | Numerical tables lacked visual clarity | Representations of uncertainty were mostly new to the audience (numbers facilitated interpretation of uncertainty) |
Content (uncertainty displays, volume of outcomes, color, icons, and ordering) | Bar charts were liked because they were considered visually clear and facilitated appraisal at a glance, since it was a known format. But they do not give enough information | Traffic light colors were described as universally recognized | ||||||
Caterpillar plots were seen as visually clearer and to give more information but you would need to learn how to read them | Using colors consistently was important, as this enabled understanding across formats | |||||||
Funnel plots were difficult to read, had to learn how to read them | Stars were described as universally recognized and their interpretation did not require the ability to read | |||||||
Tables with icons were seen as accessible to the average person | The use of red and amber was thought to cause undue alarm while icons based on thumbs was seen as trivializing the issue | |||||||
Words (these were ‘at average’ ‘better’, ‘worse’, etc.) were liked because they were perceived as needing no personal interpretation | ||||||||
Izard [3] | To develop graphic dashboards of questionnaire responses from patients with prostate cancer to facilitate clinical integration of HRQoL measurement | N = 50 prostate cancer patients and N = 50 providers, USA | Expanded Prostate Cancer Index | Individual scores | Bar chart | Previous scores; ‘patients like me’ | 44% ranked bar chart dashboards as most preferred vs line graphs vs tables and pictographs | High reading scores for the table format |
Line graph | 20% found pictograph too complicated (too many steps to interpret) | |||||||
Table that display HRQOL data in raw form | 18% had difficulty disentangling facial expressions. Felt to be ‘‘too similar’’ | |||||||
Facial expression pictograph | 16% felt table to be easy to understand, 18% felt this format made HRQoL trends difficult to interpret | |||||||
Kuijpers [4] | To investigate patients’ and clinicians’ understanding of and preferences for different graphical presentation styles for individual-level EORTC QLQC30 scores | N = 548 cancer patients in four European countries and N = 227 clinicians, the Netherlands | EORTC QLQ-C30 | Individual scores | Bar chart with color | The preferred comparison group was one’s own previous results (40.9%) | 39% preferred colored bar charts, over heat maps (20%) and colored line graphs (12%) | Objective understanding did not differ between graphical formats |
Bar chart without color | ||||||||
Line graph with color | ||||||||
Line graph without color | ||||||||
Heat map | ||||||||
Liu [28] | To develop Rheumatoid Arthritis (RA) ‘dashboard’ that could facilitate conversations about PROs and is acceptable to a wide range of patients, including English and Spanish speakers, with adequate or limited health literacy | N = 25 RA patients and N = 11 clinicians from two academic rheumatology clinics, California | (1) Clinical Disease Activity Index (CDAI) | Individual scores | Line graph | Previous scores | Preference for more detailed information and more complex design in the adequate health literacy groups, but this preference was expressed by some limited health literacy participants as well | Several, particularly in the limited health literacy groups, did not notice or understand the longitudinal nature of data from left to right nor the temporal connection between the different graphic elements |
(2) Patient-Reported Outcomes Measurement Information System (PROMIS)-physical function scale | A few patients misinterpreted the line drawn between two data points to mean information from between the visits | |||||||
(3) Pain score | ||||||||
Loth [28] | To investigate patients’ understanding of graphical presentations of longitudinal EORTC QLQ-C30 scores | N = 40 brain tumor patients, Austria | EORTC QLQ-C30 | Individual scores | Colored bar chart | Previous scores | N/A | Objective correct answers about overall change was between 74.4% (fatigue) and 90.0% (emotional functioning) |
x | Thresholds based on reference population | Difficulties with correct interpretation of different directionality of the symptom and functioning scales | ||||||
Values below/above a predefined threshold for clinical importance were given as green (clinically unimportant) or red (clinically important) bars. Thresholds for clinical importance were distribution-based | The meaning of color-coding to highlight clinically important problems was answered correctly by 100% of patients (physical function and pain), and 92.5% (emotional function and fatigue) | |||||||
90% of the patients reported that the graphs (overall change) were “very easy” or “rather easy” to understand (subjective understanding) | ||||||||
Oerlemans [5] | To investigate whether patients with lymphoma wished to receive PRO feedback, including the option to compare their scores with those of their peers, and how this feedback was evaluated | Lymphoma patients (N = 64), the Netherlands | EORTC-QLQ-C30 + item tingling in hands or feet | Individual scores | Bar chart | Previous scores | Respondents had a slight preference for bar charts | 1 patient had trouble understanding the colors of the PRO feedback at first, but after looking for a second time it became clear |
Hospital Anxiety and Depression Scale (HADS) | Line graph | Reference population: | Preferred dotted line over a solid line to indicate “your score” in the bar chart | |||||
Adapted Self-Administered Comorbidity Questionnaire | General population | |||||||
Scores other lymphoma patients | ||||||||
Patients: The vast majority (94%) compared their scores with those of the lymphoma reference cohort and 64% compared their scores with those of the normative population without cancer, whereas 6% viewed only their own scores | ||||||||
Ragouzeos [25] | To develop a “dashboard” for RA patients to display relevant PRO measures for discussion during a routine RA clinical visit | Patients with rheumatology (N = 45) and providers (N = 12), USA | Not specified | Individual scores | Prototype PRO dashboard (on paper) | N/A | Important to show progress over time | Adding simple iconography and brief definitions of terms to the design helped patients understand which information the measured represented |
A longitudinal line graph with coloring helped patients see their measures as a process instead of a moment in time | ||||||||
Smith [18] | To improve formats for presenting individual-level PRO data (for patient monitoring) and group-level PRO data (for reporting comparative clinical studies) | N = 40 clinicians in active practice and N = 39 patients diagnosed with cancer ≥ 6 months previously, not currently receiving chemotherapy/radiation or within 6 months of surgery, from JHCRN* | Not specified | Individual scores | Line graphs | Previous scores | N/A | Ease-of-understanding ratings were high for all formats, with median ranges from 9–10 |
Pie charts | ||||||||
Bar charts | ||||||||
Icon array | ||||||||
Snyder [34] | To test approaches for presenting PRO data to improve interpretability | N = 627 cancer patients/survivors, N = 236 oncology clinicians, and N = 250 PRO researchers for survey, from JHCRN* | Not specified | Individual scores | 3 line-graphs: | Previous scores | N/A | 82–99% correctly responded across directionality items |
N = 10 patients and N = 10 clinicians for interviews | (1) Green-shaded normal range | 74–83% correctly identified domains that changed > 10 points | ||||||
(2) Red-circled possibly concerning scores | 53–86% accurately identified possibly concerning scores | |||||||
(3) Red threshold-lines between normal and concerning scores | Red circles were interpreted more accurately than green shading | |||||||
Higher = better were interpreted more accurately versus higher = more | ||||||||
Threshold-line significantly more likely to be rated “very” clear and most useful compared with green shading and red circles | ||||||||
Group level/aggregated PROMs data visualization, patients | ||||||||
Brundage [12] | To explore patients' attitudes toward, and preferences for, 10 visual and written formats for communicating Health Related Quality of Life (HRQoL) information | N = 14 men and N = 19 women with variety of cancer diagnoses, post treatment ≥ 6 months earlier, Canada | PRO results from hypothetical clinical trial (cross-sectional, longitudinal) | Group mean scores | Mean HRQL scores: | Two treatments | Line graphs were preferred, because of their relative simplicity and straightforward layout | N/A |
Trends in text | Decrease in preferences for line graphs when error bars around the mean are presented | |||||||
Mean scores | ||||||||
Mean scores with SD | ||||||||
Text | ||||||||
Change mean > 6 months | ||||||||
Brundage [30] | To determine which formats for presenting HRQoL data are interpreted most accurately and are most preferred by patients | Patients with variety of cancer diagnosis, previously treated (N = 198), Canada | PRO results from hypothetical clinical trial (cross-sectional, longitudinal) | Group mean scores | Two treatments | N/A | Line graphs were preferred, due to high ease of interpretation and perceived helpfulness | Line graphs most often interpreted correctly (98%), most easy to understand, and most helpful (all p < 0.0001) |
Format type, participant age and education independent predictors of accuracy rates | ||||||||
Brundage [16] | To investigate the interpretability of current PRO data presentation formats | N = 50 patients with variety of cancer diagnoses; N = 20 clinicians in active practice, from JHCRN* | EORTC-QLQ-C30 scores | Individual scores and group means | Line graph means over time | Two treatments | Simple line graphs were preferred, since they have a high ease-of-understanding and usefulness | Accuracy ranged from 36% (cumulative distribution function question) to 100% (line graph with confidence intervals question) |
Line graph with norms | Line graphs are straightforward and clear | Patients tented to find normed scores, p-values and confidence intervals confusing | ||||||
Line graph with confidence intervals | 87% of patients preferred formats displaying multiple time-points | |||||||
Bar chart of average changes | ||||||||
Bar chart with | ||||||||
definition (improved, stable, worsened) | ||||||||
Cumulative distribution functions | ||||||||
Damman [14] | To investigate: | Interviews: patients with Parkinson's disease (N = 13) and clinicians (N = 14) | Not specified | Individual scores | Line graph with results of 2 treatment options | Patients with the same age, gender and disease duration | 56% of patients found line graphs most useful | Line graph showing results of two treatment options resulted in decisions reflecting adequate comprehension of information |
(a) How patients and clinicians think about using PROMs during consultations; | Survey: patients (N = 115), the Netherlands | Bar chart with results of 2 treatment options | 47% of patients found bar charts most useful | |||||
(b) For which purpose patients and clinicians use PROMs during consultations; | Bar chart with performance of 2 providers | 43% of patients found information with performance of two providers useful | ||||||
(c) How patients interpret PROMs information presented in various formats | ||||||||
McNair [32] | To assess patients’ understanding of multidimensional PROs in a graphical format | Patients with esophageal and gastric cancer (N = 132), UK | Semi-structured interviews | Mean scores | Line graphs: | Two treatments | N/A | 87% of patients accurately interpreted multidimensional graphical PROs from two treatments |
(1) Treatment changes in a single PRO over time | 81% of patients was able to interpret graph 4 correctly | |||||||
(2) Different PRO, reversed direction of treatment | 67% of patients was able to integrate information from two graphs together | |||||||
(3) Divergent and convergent PROs | ||||||||
(4) Divergent and convergent PROs over 18 months | ||||||||
Smith [18] | To improve formats for presenting individual-level PRO data (for patient monitoring) and group-level PRO data (for reporting comparative clinical studies) | N = 40 clinicians in active practice and N = 39 patients diagnosed with cancer ≥ 6 months previously, not currently receiving chemotherapy/radiation or within 6 months of surgery, from JHCRN* | Not specified | Individual scores, proportional data | Line graphs | Previous scores | 55% of patients preferred pie charts | N/A |
Pie charts | 25% of patients preferred icon arrays | |||||||
Bar charts | 20% of patients preferred bar charts | |||||||
Icon array | 45% of patients preferred formats with an asterisk indicating important differences | |||||||
Tolbert [29] | To identify the association of PRO score directionality and score norming on a) how accurately PRO scores are interpreted and b) how clearly they are rated by patients, clinicians, and PRO researchers | N = 629 patients (various oncologic diagnoses, N = 139 oncology clinicians, and N = 249 PRO researchers, conducted at the Johns Hopkins Clinical Research Network (JHCRN)* | Two treatments | Mean scores | Line graph 3 versions: | Two treatments | 84% of patients rated “Better” formatted line graphs most often as “very clear” or “somewhat clear” | 56% of patients answered questions correctly for “better” formatted lines, compared to 41% for “more” and 39% for “normed” graphs |
(1) Lines going up indicating “better” outcomes | The normed value confused patients | |||||||
(2) Lines going up indicating “more” (better for function domains, worse for symptoms). (3) Lines “normed” to a general population | ||||||||
Tolbert [20] | To identify best practices for presenting PRO results expressed as proportions of patients with changes from baseline (improved/ stable/ worsened) for use in patient educational materials and decision aids | N = 629 patients (various oncologic diagnoses, treated), N = 139 oncology clinicians, and N = 249 PRO researchers, conducted at the Johns Hopkins Clinical Research Network (JHCRN)* | Two treatments | Proportions | Pie chart | Two treatments | Preferred pie charts: these were easy to read and enabled obtaining information quickly. Rated the clearest for communicating proportions changed from baseline | Patient’s accuracy was highest for pie charts and icon arrays |
Bar chart | Noted helpful aspects of bar charts: “Side by side comparisons are much easier to read and comprehend” | Bar graphs were less accurately interpreted than pie charts and icon arrays | ||||||
Icon array | Icon arrays would be easy to understand for patients | |||||||
Van Overveld [19] | To investigate the preferences of receiving feedback between stakeholders | N = 37 patients, medical specialists, allied health professionals and health insurers in the Netherlands | Audit data on professional practice and health care outcomes | National average scores | Bar graph | National average scores | Patients preferred both a pie chart and a bar chart | Give feedback with average national scores |
Pie chart | Patients prefer a figure over plain text | National average scores on indicators of more interest for patient organizations and professionals | ||||||
Line graph | ||||||||
Point graph | ||||||||
Area graph | ||||||||
Box plot | ||||||||
Kaplan- Meier graph | ||||||||
Funnel plot |