Table 2 Summary of data extraction: visualization strategies and preferences, interpretation accuracy, comparators; use of PRO data on individual and group level, in clinicians

Individual level PROMs data visualization, clinicians

Brundage [16]

To investigate the interpretability of current PRO data presentation formats

N = 50 patients with variety of cancer diagnoses; N = 20 clinicians in active practice, from Johns Hopkins Clinical Research Network (JHCRN)*

EORTC-QLQ-C30 scores

Individual scores and group means

Two line graphs:

Previous scores

Preference for line graphs: overall ease-of-understanding and usefulness

Interpretation accuracy was high across formats

Higher = better functioning

90% of clinicians preferred formats displaying multiple time-points vs single time-points

Inconsistency between whether higher scores were better or worse contributes to incorrect accuracy (Uptrend lines intuitively signify improvement of understanding)

Higher = more symptoms

Tabulated scores were considered boring but straightforward

Bubble plot

Heat maps

Brundage [37]

To evaluate the interpretation accuracy and perceived clarity of various strategies for displaying clinical trial PRO findings

Oncology clinicians (N = 233) and PRO researchers (N = 248), from JHCRN*

PRO results from hypothetical clinical trial (cross-sectional, longitudinal)

Longitudinal individual data, proportions

Bar chart

Line graph also normed against general population

Not one approach for either longitudinal data or proportions changed is universally appealing, nor is free of misinterpretation errors

Line graph:

Pie chart

More likely to be interpreted correctly “better” vs “normed” graphs (p = 0.04)

3 Line graphs:

No differences between “better” and “more”

1) Higher = “better” functioning;

Regardless of graph type and version, adding asterisks for clinical significance and confidence limits did not contribute to better interpretation accuracy

2) Higher = “more” symptoms;

Bar chart/pie chart:

3) “Normed” against general population

Respondents less likely to make interpretation errors with pie vs bar charts (p < 0.001)

Odds of selecting an “incorrect” treatment significantly lower in pie charts compared to bar charts

Clarity ratings did not differ between formats

Damman [14]

To investigate:

Interviews: patients with Parkinson's disease (N = 13) and clinicians (N = 14)

Not specified

Individual scores

Line graph

Patients with the same age, gender and disease duration

Strong preference for individual PROMs data over time

Individual PROMs scores with comparative data of similar patients were found useful, some expressed doubts

(a) How patients and clinicians think about using PROMs during consultations;

Survey: patients (N = 115), the Netherlands

Bar graph

Line and bar graphs

(b) For which purpose patients and clinicians use PROMs during consultations;

Line graph with comparative data over time (i.e. average scores of similar patients)

Scores from repeated measurements over time

(c) How patients interpret PROMs information presented in various formats

Multiple individual quality of life, rather than one overall quality of life score

Identified the possibility to use aggregated PROMs scores as evidence for treatment options

Grossman [27]

To identify the design requirements for an interface that assists patients with PRO survey completion and interpretation; to build and evaluate the interface of PROMs feedback

Interview: N = 13 patients with heart failure and N = 11 clinicians, study location or country was not described

Health IT Usability Evaluation Scale (Health-ITUES)

Individual scores

Small cards: Contains a short sentence to describe a severe symptom, which when clicked on provides textual educational information

N/A

Perceiving the mockup as useful and easy-to-use

Two providers reported that PROs might reduce their cognitive load

Usability testing: N = 12 patients with heart failure

Graph:

Preference for tracking symptoms over time

Bar chart that lists the patient’s symptoms from most to least severe and displays each symptom’s severity score

Large cards:

Displays a symptom name, its two-sentence description, a visual representation of its severity, and a link to textual educational information

Hartzler [36]

To share lessons learned from engaging clinicians to inform design of visual dashboards

Clinicians: N = 12 for interviews, N = 40 for surveys and consensus meeting, N = 9 for user testing, study location or country was not described

Not specified

Individual scores

PRO data needs appear to differ for health care providers and administrative staff as key target users

N/A

Participants liked viewing trends over time

Value was found in developing meaningful ways to report on this new source of data

The functional prototype has 3 components:

Participants found the views to provide a useful basis for comparison

In addition to the information buttons provided on “how to interpret this chart,” clear labels are needed, such as on chart axes

(1) An “At a glance” screen providing a simple data overview of PROs data

(2) An “Analyze” screen providing a data view the user can filter

(3) A “Data quality” screen

Hartzler [6]

To conduct a HCD to engage patients, providers, and interaction design experts in the development of visual “PRO dashboards” that illustrate personalized trends in patients’ HRQoL following prostate cancer treatment

Focus groups (N = 60 patients)

Not specified

Individual scores

Pictographs

The dashboard compares patients’ trends with trends from “men like you” matched by default by age and treatment derived from a previously published prostate cancer cohort

Pictographs less helpful than bar charts, line graphs, or tables (P < .001)

Pictographs might reach patients with limited literacy

N = 50 prostate cancer patients and N = 50 clinicians, study location or country was not described

Bar charts

Preferred bar charts and line graphs

Some participants, both patients and providers, expressed concern over inclusion of comparison scores without representation of data variability (e.g., confidence intervals, error bars), while others preferred simpler charts and graphs

Line graphs

Izard [3]

To develop graphic dashboards of questionnaire responses from patients with prostate cancer to facilitate clinical integration of HRQoL measurement

N = 50 prostate cancer patients and N = 50 providers from Seattle, USA

Expanded Prostate Cancer Index

Individual scores

Bar chart

Previous scores; ‘patients like me’

No universally preferred dashboard format: 30% preferred tables, 34% preferred bar charts, and 34% preferred line graphs

Helpfulness and confidence ratings varied among dashboard format. Pictographs had the lowest helpfulness compared with table, bar, and line graph formats

Line graph

Table that display HRQOL data in raw form

Facial expression pictograph

Jagsi [35]

To investigate practicing oncologists view on incorporating routine collection of PROs into cancer care

N = 17 oncologists, USA

Edmonton Symptom Assessment System

Individual scores

Bar chart

Previous scores

Ability to track symptoms over time and effect of intervention

Keep it simple: limit number of symptoms

Link number scale to narrative

Kuijpers [4]

To investigate patients’ and clinicians’ understanding of and preferences for different graphical presentation styles for individual-level EORTC QLQC30 scores

N = 548 cancer patients in four European countries and N = 227 clinicians, the Netherlands

EORTC QLQ-C30

Individual scores

Bar chart with color

The preferred comparison group was one’s own previous results (40.9%)

Medical specialist:

Medical specialist:

Bar chart without color

Heat map (46%)

Objective understanding of 78%

Line graph with color

Nurses:

Nurses:

Line graph without color

Bar chart (non-colored) and heat map (32%)

Objective understanding of 74%

Heat map

85% of all HCP’s indicated that the graphs were (easy) to understand, not differing between professions or graphical presentation styles

Uniformity in scoring direction would be preferred

A significant difference for overall change scores, with the non-colored bar charts being interpreted correctly more often than the other graphical displays

Ragouzeos [25]

To develop a “dashboard” for RA patients to display relevant PRO measures for discussion during a routine RA clinical visit

Patients with rheumatology (N = 45) and providers (N = 12), USA

Not specified

Individual scores

Prototype PRO dashboard (on paper)

Previous scores

N/A

Information needs to be clearly focused on what is most essential

Important to show progress over time

Santana [33]

To describe the process, feasibility and acceptability of use of the Health Utilities Index (HUI) in routine clinical care

Pre- and post-heart and -lung transplant patients (N = 151), Canada

Health Utilities Index (HUI)

Individual scores

HUI score card, using a color-coded system

Pre- and post-treatment scores

N/A

Clinicians did not need much time to understand the use of the HUI score card

Clinicians developed their own way of using the information over time

Smith [18]

To improve formats for presenting individual-level PRO data (for patient monitoring) and group-level PRO data (for reporting comparative clinical studies)

N = 40 clinicians in active practice and N = 39 patients diagnosed with cancer ≥ 6 months previously, not currently receiving chemotherapy/radiation or within 6 months of surgery, from JHCRN*

Not specified

Individual scores, proportional data

Line graphs

Previous scores

75% preferred the line graph

Ease-of-understanding ratings were high for all formats

Pie charts

Directional inconsistency emerged as an interpretation challenge

Bar charts

Format interpretation challenges included explaining the meaning of scores (i.e., whether scores are good/bad, what normal is)

Icon array

Snyder [34]

To test approaches for presenting PRO data to improve interpretability

N = 627 cancer patients/survivors, N = 236 oncology clinicians, and N = 250 PRO researchers for survey, from JHCRN*

Not specified

Individual scores

3 line-graphs:

Previous scores

N/A

The proportion responding correctly across the 4 directionality items ranged from 80 to 100%

N = 10 patients and N = 10 clinicians for interviews

1) Green-shaded normal range

Red circles were interpreted more accurately than green shading

2) Red-circled possibly concerning scores

Higher = better were interpreted more accurately versus higher = more

3) Red threshold-lines between normal and concerning scores

Threshold-line significantly more likely to be rated “very” clear and most useful compared with green shading and red circles

Group level/aggregated PROMs data visualization, clinicians

Brundage [16]

To investigate the interpretability of current PRO data presentation formats

N = 50 patients with variety of cancer diagnoses; N = 20 clinicians in active practice, from JHCRN*

EORTC-QLQ-C30 scores

Individual scores and group means

Line graph means over time

Two treatments (study arms)

90% of clinicians preferred formats displaying multiple time-points

Line graphs contributed to overall ease-of-understanding and usefulness

Line graph with norms

Preference for line graphs of normed sores or with confidence intervals

Normed scores provided basis for comparison beyond two treatments, p-values and confidence intervals were particularly important for publications

Line graph with confidence intervals

Some preference for bar charts to compare treatments

Cumulative distributing function confusing and difficult to interpret

Bar chart of average changes

Inconsistency between whether higher scores were better or worse contributes to incorrect accuracy

Bar chart with definition (improved, stable, worsened)

•Cumulative distribution functions

Damman [14]

To investigate:

Interviews: patients with Parkinson's disease (N = 13) and clinicians (N = 14)

Not specified

Individual scores

Line graph with results of 2 treatment options

Patients with the same age, gender and disease duration

Identified the possibility to use aggregated PROMs scores as evidence for treatment options

N/A

(a) How patients and clinicians think about using PROMs during consultations;

Survey: patients (N = 115), the Netherlands

Bar chart with results of 2 treatment options

Aggregated PROMs scores for provider options could be useful, but would not be used much in clinical practice

(b) For which purpose patients and clinicians use PROMs during consultations;

Bar chart with performance of 2 providers

(c) How patients interpret PROMs information presented in various formats

Liu [28]

To develop Rheumatoid Arthritis (RA) ‘dashboard’ that could facilitate conversations about PROs and is acceptable to a wide range of patients, including English and Spanish speakers, with adequate or limited health literacy

N = 25 RA patients and N = 11 clinicians from two academic rheumatology clinics, California (USA)

(1) Clinical Disease Activity Index (CDAI)

Individual scores

Line graph

Aggregated clinical data

A dashboard is a potential method for aggregating data from various sources

A ‘snapshot’ of relevant information for a particular patient would make HCP’s own medical decisions easier

(2) Patient-Reported Outcomes Measurement Information System (PROMIS)-physical function scale

Clinicians were very interested in customizing the dashboard to their own needs and recommended that it can be designed to present information that is more detailed

(3) Pain score

Smith [18]

To improve formats for presenting individual-level PRO data (for patient monitoring) and group-level PRO data (for reporting comparative clinical studies)

N = 40 clinicians in active practice and N = 39 patients diagnosed with cancer ≥ 6 months previously, not currently receiving chemotherapy/radiation or within 6 months of surgery, from JHCRN*

Not specified

Individual scores, proportional data

Line graphs

Average changes

For proportional data formats: pie charts (70%)

Median ease-of-understanding ranged from 6.5 to 8

Pie charts

Few clinicians (10%) preferred bar charts

Mixed feelings about indications of clinical significance in terms of having p-values in addition to confidence intervals and asterisks indicating important differences

Bar charts

75% preferred the line graph

Directional inconsistency emerged as an interpretation challenge

Icon array

Format interpretation challenges included explaining the meaning of scores (i.e., whether scores are good/bad, what normal is)

Tolbert [29]

To identify the association of PRO score directionality and score norming on a) how accurately PRO scores are interpreted and b) how clearly they are rated by patients, clinicians, and PRO researchers

N = 629 patients (various oncologic diagnoses), N = 139 oncology clinicians, and N = 249 PRO researchers, conducted at the Johns Hopkins Clinical Research Network (JHCRN)*

Two treatments

Mean scores

Line graph 3 versions:

Two treatments

“Better” formatted line graphs were rated most often as “very clear” or “somewhat clear” by all three groups (84% by patients, 81% by clinicians, and 85% by researchers)

Answers correct for “better”: 70%; “more”: 65%; “normed”: 65%

(1) Lines going up indicating “better” outcomes

However, the range in the proportion rating each format “very clear” or “somewhat clear” was narrow: 77% to 85%

“More” line graph comments noted that up could mean different things, which could lead to errors

(2) Lines going up indicating “more” (better for function domains, worse for symptoms). 3) Lines “normed” to a general population average of 50

“Better” line graph comments pointed out how changing the scale could result in interpretation errors as one must orient to the direction of the scale each time

Tolbert [20]

To identify best practices for presenting PRO results expressed as proportions of patients with changes from baseline (improved/ stable/ worsened) for use in patient educational materials and decision aids

N = 629 patients (various oncologic diagnoses, treated), N = 139 oncology clinicians, and N = 249 PRO researchers, conducted at the Johns Hopkins Clinical Research Network (JHCRN)*

Two treatments

Proportions changed

Pie chart

Two treatments

Preferred pie charts: these were easy to read and enabled obtaining information quickly

Clinician and researchers scored pie charts as the most accurately interpreted

Bar chart

43% had positive feedback on icon arrays

In general, bar graphs were less accurately interpreted than pie charts and icon arrays

Icon array

38% had positive feedback on bar charts

Noted helpful aspects of bar charts: “Side

by side comparisons are much easier to read and comprehend”

van Overveld [19]

To investigate the preferences of receiving feedback between stakeholders

N = 37 patients, medical specialists, allied health professionals and health insurers in the Netherlands

Audit data on professional practice and health care outcomes

National average scores

Bar graph

National average scores

Preference for bar charts since they are easier to read

Box plots, Kaplan–Meier graphs and funnel plots give a less clear overview and are more difficult to interpret

Pie chart

For survival and process indicators: Kaplan–Meier graphs and box plots

Find a balance between giving feedback and giving too much information

Line graph

Give an overview of the results first, followed by the details

Point graph

Present it that one can easily understand without explanation

Area graph

Box plot

Kaplan- Meier graph

Funnel plot