Skip to main content

Table 1 Summary of data extraction: visualization strategies and preferences, interpretation accuracy, comparators; use of PRO data on individual and group level, in patients

From: Visualization formats of patient-reported outcome measures in clinical practice: a systematic review about preferences and interpretation accuracy

Author

Primary study goal

Study population

Presenting PROMs data

Visualization

   

Type of PROMs

What is presented?

Graphic visualisation format

Comparator

Outcomes of included studies preferences

Interpretation accuracy

Individual level PROMs data visualization, patient

Brundage [16]

To investigate the interpretability of current PRO data presentation formats

N = 50 patients with variety of cancer diagnoses; N = 20 clinicians in active practice, Johns Hopkins Clinical Research Network (JHCRN)*

EORTC-QLQ-C30 scores

Individual scores and group means

Line graphs of mean scores

Previous scores

Simple line graphs for overall ease-of-understanding and usefulness

Patients accuracy ranged from 64–96% (line graphs questions)

     

Tabulated scores

 

92% preferred formats displaying multiple time points

A graph trending down with better = higher scores was correctly interpreted by 96%. A graph trending down up with better = lower scores was correctly interpreted by 64%

     

Bubble plots of scores at a point in time

   
     

Heat map

   

Damman [14]

To investigate:

Interviews: patients with Parkinson's disease (N = 13) and clinicians (N = 14)

Not specified

Individual scores

Line graph

Patients with the same age, gender and disease duration

Bar chart is preferred (57.2%) compared to line graphs (42.3%)

What hindered easy comprehension: the use of a “higher = worse” directionality and comparative information of patients that are similar in terms of age, gender and disease progression

 

(a) How patients and clinicians think about using PROMs during consultations;

Survey: patients (N = 115), the Netherland

  

Bar graph

  

Line and bar charts were interpreted most often correctly, compared with more “evaluative” formats like smileys and colors

 

(b) For which purpose patients and clinicians use PROMs during consultations;

   

Line graph with comparative data over time (i.e. average scores of similar patients)

  

Individual PROMs scores over time were interpreted more often correctly when presented in a bar chart (87.8%) compared to a line graph (74.3%)

 

(c) How patients interpret PROMs information presented in various formats

       

Fischer [1]

To develop a PRO feedback report for mobile devices that is comprehensible and provides valuable information for patients after knee arthroplasty

Orthopedic patients (N = 8), Germany

Multiple (literature)

Individual scores

Text-based report and a graphical display (line graph, where scores are plotted over time, over a rainbow-colored background from red (bottom) to green (top) to visualize the grading of the individual scores)

Norm population

Short and condensed information using simple language (literature)

A text-based report is the least preferred but less susceptible to misinterpretation (literature)

   

PROMIS (development)

   

An efficient way to present longitudinal PRO scores: graphs such as bar or line graphs (literature)

All participants correctly understood the line graph and were able to interpret the scores. Some needed some initial guidance on how to read a line graph

       

Those (n = 3) in favor of graphs: easy and quick to get the relevant information from the line graph

The rainbow-colored background was understood by all participants

       

The text-based (n = 2) version is easier to understand and most people are used to read short text messages

 

Geerards [26]

To assess the impact of tailored multimodal feedback and computerized adapted testing (CAT) on user experience in HRQoL assessment using validated PROMs

N = 1386 participants from the general population, United Kingdom (UK)

World Health Organization Quality of Life assessment (WHOQOL)

Individual scores

Graphical only

N/A

Respondents thought the questionnaire with graphical and text-based feedback was more interesting compared with no feedback assessment, whereas providing only graphical feedback did not make the questionnaire more interesting

82.4% of patients thought the graphical feedback was accurate

     

Graphical and adaptive text-based feedback

  

92.9% of patients thought the graphical feedback was clear

     

Graphs: Separate horizontal bar charts for 4 domains

   
     

Text: What each domain reflects, how score corresponds to average scores, and what score might mean

   

Grossman [27]

To identify the design requirements for an interface that assists patients with PRO survey completion and interpretation; to build and evaluate the interface of PROMs feedback

Interview: N = 13 patients with heart failure and N = 11 clinicians, study location or country was not described

Health IT Usability Evaluation Scale (Health-ITUES)

Individual scores

Small cards: Short sentence describing a severe symptom, which when clicked on provides textual educational information

N/A

Perceiving the mockup as useful and easy-to-use

Half of the participants failed to interpret the bar chart correctly, and even participants who could read it often required multiple attempts

  

Usability testing: N = 12 patients with heart failure

  

Large cards:

 

Patients preferred visualizations with brief text descriptions

 
     

Symptom name and description, visual representation of its severity, and a link to textual educational information

   
     

Graph: Bar chart (lists patient’s symptoms from most to least severe, with symptom’s severity scores)

   

Hartzler [6]

To conduct a HCD to engage patients, providers, and interaction design experts in the development of visual “PRO dashboards” that illustrate personalized trends in patients’ HRQoL following prostate cancer treatment

Focus groups (N = 60 patients)

Not specified

Individual scores

Pictographs

The dashboard compares patients’ trends with trends from “men like you” matched by default by age and treatment derived from a previously published prostate cancer cohort

Pictographs less helpful than bar charts, line graphs, or tables (P < .001)

Pictographs might reach patients with limited literacy

  

N = 50 prostate cancer patients and N = 50 clinicians, study location or country was not described

  

Bar charts

 

Bar charts and line graphs are most preferred

Some patients expressed concern over inclusion of comparison scores without representation of data variability (e.g., confidence intervals, error bars), while others preferred simpler charts and graphs

     

Line graphs

   

Hildon [2]

To explore patients’ views of different formats and content of data displays of PROMs

N = 45 patients undergone or planning knee surgery in six focus groups, UK

Oxford Hip Score (OHS)

Individual scores

Different formats (table, bar chart, caterpillar and funnel plot)

N/A

Numerical tables lacked visual clarity

Representations of uncertainty were mostly new to the audience (numbers facilitated interpretation of uncertainty)

     

Content (uncertainty displays, volume of outcomes, color, icons, and ordering)

 

Bar charts were liked because they were considered visually clear and facilitated appraisal at a glance, since it was a known format. But they do not give enough information

Traffic light colors were described as universally recognized

       

Caterpillar plots were seen as visually clearer and to give more information but you would need to learn how to read them

Using colors consistently was important, as this enabled understanding across formats

       

Funnel plots were difficult to read, had to learn how to read them

Stars were described as universally recognized and their interpretation did not require the ability to read

       

Tables with icons were seen as accessible to the average person

The use of red and amber was thought to cause undue alarm while icons based on thumbs was seen as trivializing the issue

        

Words (these were ‘at average’ ‘better’, ‘worse’, etc.) were liked because they were perceived as needing no personal interpretation

Izard [3]

To develop graphic dashboards of questionnaire responses from patients with prostate cancer to facilitate clinical integration of HRQoL measurement

N = 50 prostate cancer patients and N = 50 providers, USA

Expanded Prostate Cancer Index

Individual scores

Bar chart

Previous scores; ‘patients like me’

44% ranked bar chart dashboards as most preferred vs line graphs vs tables and pictographs

High reading scores for the table format

     

Line graph

  

20% found pictograph too complicated (too many steps to interpret)

     

Table that display HRQOL data in raw form

  

18% had difficulty disentangling facial expressions. Felt to be ‘‘too similar’’

     

Facial expression pictograph

  

16% felt table to be easy to understand, 18% felt this format made HRQoL trends difficult to interpret

Kuijpers [4]

To investigate patients’ and clinicians’ understanding of and preferences for different graphical presentation styles for individual-level EORTC QLQC30 scores

N = 548 cancer patients in four European countries and N = 227 clinicians, the Netherlands

EORTC QLQ-C30

Individual scores

Bar chart with color

The preferred comparison group was one’s own previous results (40.9%)

39% preferred colored bar charts, over heat maps (20%) and colored line graphs (12%)

Objective understanding did not differ between graphical formats

     

Bar chart without color

   
     

Line graph with color

   
     

Line graph without color

   
     

Heat map

   

Liu [28]

To develop Rheumatoid Arthritis (RA) ‘dashboard’ that could facilitate conversations about PROs and is acceptable to a wide range of patients, including English and Spanish speakers, with adequate or limited health literacy

N = 25 RA patients and N = 11 clinicians from two academic rheumatology clinics, California

(1) Clinical Disease Activity Index (CDAI)

Individual scores

Line graph

Previous scores

Preference for more detailed information and more complex design in the adequate health literacy groups, but this preference was expressed by some limited health literacy participants as well

Several, particularly in the limited health literacy groups, did not notice or understand the longitudinal nature of data from left to right nor the temporal connection between the different graphic elements

   

(2) Patient-Reported Outcomes Measurement Information System (PROMIS)-physical function scale

    

A few patients misinterpreted the line drawn between two data points to mean information from between the visits

   

(3) Pain score

     

Loth [28]

To investigate patients’ understanding of graphical presentations of longitudinal EORTC QLQ-C30 scores

N = 40 brain tumor patients, Austria

EORTC QLQ-C30

Individual scores

Colored bar chart

Previous scores

N/A

Objective correct answers about overall change was between 74.4% (fatigue) and 90.0% (emotional functioning)

x

     

Thresholds based on reference population

 

Difficulties with correct interpretation of different directionality of the symptom and functioning scales

      

Values below/above a predefined threshold for clinical importance were given as green (clinically unimportant) or red (clinically important) bars. Thresholds for clinical importance were distribution-based

 

The meaning of color-coding to highlight clinically important problems was answered correctly by 100% of patients (physical function and pain), and 92.5% (emotional function and fatigue)

        

90% of the patients reported that the graphs (overall change) were “very easy” or “rather easy” to understand (subjective understanding)

Oerlemans [5]

To investigate whether patients with lymphoma wished to receive PRO feedback, including the option to compare their scores with those of their peers, and how this feedback was evaluated

Lymphoma patients (N = 64), the Netherlands

EORTC-QLQ-C30 + item tingling in hands or feet

Individual scores

Bar chart

Previous scores

Respondents had a slight preference for bar charts

1 patient had trouble understanding the colors of the PRO feedback at first, but after looking for a second time it became clear

   

Hospital Anxiety and Depression Scale (HADS)

 

Line graph

Reference population:

Preferred dotted line over a solid line to indicate “your score” in the bar chart

 
   

Adapted Self-Administered Comorbidity Questionnaire

  

General population

  
      

Scores other lymphoma patients

  
      

Patients: The vast majority (94%) compared their scores with those of the lymphoma reference cohort and 64% compared their scores with those of the normative population without cancer, whereas 6% viewed only their own scores

  

Ragouzeos [25]

To develop a “dashboard” for RA patients to display relevant PRO measures for discussion during a routine RA clinical visit

Patients with rheumatology (N = 45) and providers (N = 12), USA

Not specified

Individual scores

Prototype PRO dashboard (on paper)

N/A

Important to show progress over time

Adding simple iconography and brief definitions of terms to the design helped patients understand which information the measured represented

       

A longitudinal line graph with coloring helped patients see their measures as a process instead of a moment in time

 

Smith [18]

To improve formats for presenting individual-level PRO data (for patient monitoring) and group-level PRO data (for reporting comparative clinical studies)

N = 40 clinicians in active practice and N = 39 patients diagnosed with cancer ≥ 6 months previously, not currently receiving chemotherapy/radiation or within 6 months of surgery, from JHCRN*

Not specified

Individual scores

Line graphs

Previous scores

N/A

Ease-of-understanding ratings were high for all formats, with median ranges from 9–10

     

Pie charts

   
     

Bar charts

   
     

Icon array

   

Snyder [34]

To test approaches for presenting PRO data to improve interpretability

N = 627 cancer patients/survivors, N = 236 oncology clinicians, and N = 250 PRO researchers for survey, from JHCRN*

Not specified

Individual scores

3 line-graphs:

Previous scores

N/A

82–99% correctly responded across directionality items

  

N = 10 patients and N = 10 clinicians for interviews

  

(1) Green-shaded normal range

  

74–83% correctly identified domains that changed > 10 points

     

(2) Red-circled possibly concerning scores

  

53–86% accurately identified possibly concerning scores

     

(3) Red threshold-lines between normal and concerning scores

  

Red circles were interpreted more accurately than green shading

        

Higher = better were interpreted more accurately versus higher = more

        

Threshold-line significantly more likely to be rated “very” clear and most useful compared with green shading and red circles

Group level/aggregated PROMs data visualization, patients

Brundage [12]

To explore patients' attitudes toward, and preferences for, 10 visual and written formats for communicating Health Related Quality of Life (HRQoL) information

N = 14 men and N = 19 women with variety of cancer diagnoses, post treatment ≥ 6 months earlier, Canada

PRO results from hypothetical clinical trial (cross-sectional, longitudinal)

Group mean scores

Mean HRQL scores:

Two treatments

Line graphs were preferred, because of their relative simplicity and straightforward layout

N/A

     

Trends in text

 

Decrease in preferences for line graphs when error bars around the mean are presented

 
     

Mean scores

   
     

Mean scores with SD

   
     

Text

   
     

Change mean > 6 months

   

Brundage [30]

To determine which formats for presenting HRQoL data are interpreted most accurately and are most preferred by patients

Patients with variety of cancer diagnosis, previously treated (N = 198), Canada

PRO results from hypothetical clinical trial (cross-sectional, longitudinal)

Group mean scores

Two treatments

N/A

Line graphs were preferred, due to high ease of interpretation and perceived helpfulness

Line graphs most often interpreted correctly (98%), most easy to understand, and most helpful (all p < 0.0001)

        

Format type, participant age and education independent predictors of accuracy rates

Brundage [16]

To investigate the interpretability of current PRO data presentation formats

N = 50 patients with variety of cancer diagnoses; N = 20 clinicians in active practice, from JHCRN*

EORTC-QLQ-C30 scores

Individual scores and group means

Line graph means over time

Two treatments

Simple line graphs were preferred, since they have a high ease-of-understanding and usefulness

Accuracy ranged from 36% (cumulative distribution function question) to 100% (line graph with confidence intervals question)

     

Line graph with norms

 

Line graphs are straightforward and clear

Patients tented to find normed scores, p-values and confidence intervals confusing

     

Line graph with confidence intervals

 

87% of patients preferred formats displaying multiple time-points

 
     

Bar chart of average changes

   
     

Bar chart with

   
     

definition (improved, stable, worsened)

   
     

Cumulative distribution functions

   

Damman [14]

To investigate:

Interviews: patients with Parkinson's disease (N = 13) and clinicians (N = 14)

Not specified

Individual scores

Line graph with results of 2 treatment options

Patients with the same age, gender and disease duration

56% of patients found line graphs most useful

Line graph showing results of two treatment options resulted in decisions reflecting adequate comprehension of information

 

(a) How patients and clinicians think about using PROMs during consultations;

Survey: patients (N = 115), the Netherlands

  

Bar chart with results of 2 treatment options

 

47% of patients found bar charts most useful

 
 

(b) For which purpose patients and clinicians use PROMs during consultations;

   

Bar chart with performance of 2 providers

 

43% of patients found information with performance of two providers useful

 
 

(c) How patients interpret PROMs information presented in various formats

       

McNair [32]

To assess patients’ understanding of multidimensional PROs in a graphical format

Patients with esophageal and gastric cancer (N = 132), UK

Semi-structured interviews

Mean scores

Line graphs:

Two treatments

N/A

87% of patients accurately interpreted multidimensional graphical PROs from two treatments

     

(1) Treatment changes in a single PRO over time

  

81% of patients was able to interpret graph 4 correctly

     

(2) Different PRO, reversed direction of treatment

  

67% of patients was able to integrate information from two graphs together

     

(3) Divergent and convergent PROs

   
     

(4) Divergent and convergent PROs over 18 months

   

Smith [18]

To improve formats for presenting individual-level PRO data (for patient monitoring) and group-level PRO data (for reporting comparative clinical studies)

N = 40 clinicians in active practice and N = 39 patients diagnosed with cancer ≥ 6 months previously, not currently receiving chemotherapy/radiation or within 6 months of surgery, from JHCRN*

Not specified

Individual scores, proportional data

Line graphs

Previous scores

55% of patients preferred pie charts

N/A

     

Pie charts

 

25% of patients preferred icon arrays

 
     

Bar charts

 

20% of patients preferred bar charts

 
     

Icon array

 

45% of patients preferred formats with an asterisk indicating important differences

 

Tolbert [29]

To identify the association of PRO score directionality and score norming on a) how accurately PRO scores are interpreted and b) how clearly they are rated by patients, clinicians, and PRO researchers

N = 629 patients (various oncologic diagnoses, N = 139 oncology clinicians, and N = 249 PRO researchers, conducted at the Johns Hopkins Clinical Research Network (JHCRN)*

Two treatments

Mean scores

Line graph 3 versions:

Two treatments

84% of patients rated “Better” formatted line graphs most often as “very clear” or “somewhat clear”

56% of patients answered questions correctly for “better” formatted lines, compared to 41% for “more” and 39% for “normed” graphs

     

(1) Lines going up indicating “better” outcomes

  

The normed value confused patients

     

(2) Lines going up indicating “more” (better for function domains, worse for symptoms). (3) Lines “normed” to a general population

   

Tolbert [20]

To identify best practices for presenting PRO results expressed as proportions of patients with changes from baseline (improved/ stable/ worsened) for use in patient educational materials and decision aids

N = 629 patients (various oncologic diagnoses, treated), N = 139 oncology clinicians, and N = 249 PRO researchers, conducted at the Johns Hopkins Clinical Research Network (JHCRN)*

Two treatments

Proportions

Pie chart

Two treatments

Preferred pie charts: these were easy to read and enabled obtaining information quickly. Rated the clearest for communicating proportions changed from baseline

Patient’s accuracy was highest for pie charts and icon arrays

     

Bar chart

 

Noted helpful aspects of bar charts: “Side by side comparisons are much easier to read and comprehend”

Bar graphs were less accurately interpreted than pie charts and icon arrays

     

Icon array

  

Icon arrays would be easy to understand for patients

Van Overveld [19]

To investigate the preferences of receiving feedback between stakeholders

N = 37 patients, medical specialists, allied health professionals and health insurers in the Netherlands

Audit data on professional practice and health care outcomes

National average scores

Bar graph

National average scores

Patients preferred both a pie chart and a bar chart

Give feedback with average national scores

     

Pie chart

 

Patients prefer a figure over plain text

National average scores on indicators of more interest for patient organizations and professionals

     

Line graph

   
     

Point graph

   
     

Area graph

   
     

Box plot

   
     

Kaplan- Meier graph

   
     

Funnel plot

   
  1. Definitions: individual level PROMs data—The patient’s perspective on their health status; Group level PROMs data—Aggregated PROMs scores collected in clinical studies or trials
  2. CDAI—Clinical Disease Activity Index; EHR—Electronic Health Record; EORTC-QLQ-C30—European Organization for Research and Treatment of Cancer Quality of life questionnaire Core 30; HADS—Hospital Anxiety and Depression Scale; HCD—Human Centered Design; HRQoL—Health-Related Quality of Life; HUI—Health Utility Index; Health-ITUES—Health IT Usability Evaluation Scale; JHCRN—Johns Hopkins Clinical Research Network; N/A—Not Applicable; PRO(s) —Patient Reported Outcome(s); PROMs—Patient Reported Outcome Measurements; PROMIS—Patient-Reported Outcomes Measurement Information System; QoL—Quality of Life; REALM-SF—Rapid Estimate of Adult Literacy in Medicine Short Form; SD—Standard Deviation; WHOQOL—World Health Organization Quality of Life
  3. *JHCRN—Johns Hopkins Clinical Research Network: A consortium of academic and community health systems in the US mid-Atlantic with clinics outside the USA as well