Overview of the ISOQOL symposium
The symposium session included five presentations and was focused on using qualitative methods to explore and define estimates of clinically meaningful change and responder thresholds, including an outline of advantages, challenges and potential solutions. The first presentation provided an overview of qualitative techniques including potential benefits and limitations. See sections Semi structured interviews conducted externally to a clinical trial to Vignettes (bookmarking/standard setting) for an evaluation of each approach.
The second presentation summarized results from a qualitative exit interview study with psychologists and parents/caregivers to help inform clinically meaningful change thresholds on an observer-reported, clinician- administered measure and a performance outcome measure in Down syndrome. The third and fourth presentations focused on results from qualitative interviews and an associated analysis of clinical trial data in pediatric asthma. The overarching aim of this research was to establish clinically meaningful change in symptom-free days (SFDs), rescue free days (RFDs) and on the childhood Asthma Control Test (C-ACT) in pediatric asthma. The final presentation focused on the use of clinical trial exit interviews to generate anchors for incorporation in later psychometric evaluation. See sections Semi-structured interviews conducted externally to a clinical trial and Semi-structured exit interviews conducted as part of a clinical trial for further detail and elaboration on these examples.
Overview of qualitative research techniques to define meaningful change
Numerous qualitative methods have been used to capture patient, caregiver and clinician perspectives of important change. These include semi-structured interviews conducted externally to a clinical trial, semi-structured interviews conducted as part of a clinical trial (exit interviews), focus groups, the presentation of vignettes and the Delphi panel method. When considering the conduct of semi-structured interviews, Concept Elicitation (CE) or Cognitive Debriefing (CD) techniques can be used to explore a particular condition or COA measure. As outlined by Patrick et al., (2011) the concept elicitation process starts deliberately broad and open-ended in order to explore and define information, usually about a particular condition, from the perspective of the patient [14]. The cognitive debriefing process on the other hand (also called cognitive interviews) is more structured and usually focused upon a specific COA measure, with direct questions about understanding, relevance and comprehensiveness of the measure. Both of these interview techniques can be used to explore what would potentially be a meaningful change in a condition or on a specific COA measure. Moreover, these methods can be employed in interviews conducted independently of a clinical trial or embedded as part of a clinical trial in order to explore important change on target COAs.
CE questioning within a semi-structured interview can provide non-biased, spontaneous insights into what an important change in symptoms would be, based upon individual experiences. Such qualitative data, even when captured from a sample who are not clinical trial participants, can be used to augment quantitative findings from a clinical trial by providing in-depth qualitative descriptions of the patient experience, for example from patients at various levels of functioning or disease severity. The technique of CD within a semi-structured interview can also help to provide better understanding of thresholds for important improvements/decline in the context of a response scale or score change. This would involve asking patients about their understanding of the underlying scale and then establishing the level of change on the scale that they would consider meaningful.
Other approaches to exploring meaningful change include focus groups, the presentation of clinical vignettes and the Delphi panel method. Although there is a not a fixed procedure for focus groups, they typically involve a group discussion among six to ten participants, with high moderator involvement to guide the discussion and to ensure participants keep to topic and all share their views [15]. Again, CE or CD interview techniques can be used to discuss meaningful change in a focus group setting. The presentation of vignettes, specifically the bookmarking/standard setting method, is another approach which has been used to derive qualitative meaningful change insights. This typically involves presenting patients and/or experts with hypothetical clinical vignettes in order to reach a consensus on thresholds for meaningful change. Finally, the Delphi panel method is a technique which involves using a series of questionnaires or ‘rounds’ to gather perspectives, obtain feedback and reach consensus on a given topic [16].
The remainder of this article evaluates these qualitative techniques for establishing important score change on COAs and provides examples of each approach.
Semi-structured interviews conducted externally to a clinical trial
Semi-structured interviews conducted externally to a clinical trial can provide preliminary insights regarding meaningful concepts for a treatment to target, as well as anecdotal insights regarding what level of change in these concepts is important. Moreover, this approach can provide useful insights when it is not possible to incorporate an interview in a clinical trial for logistical reasons. Semi-structured interviews involve the interviewer leading the discussion based on a pre-determined interview guide. However the participant’s responses help guide the level of information generated about the predetermined topics and their relative importance [10]. One challenge of this approach is how to link the qualitative comments to potential score changes on the measure and reconciling heterogeneous qualitative insights, especially for complex, multi-domain COAs. In addition, the target sample’s experience of change in the concept of interest must be considered in the context of the severity of their disease; meaningful change thresholds may differ depending on severity (see Katz et al. 2015, for example) [17]. It is also important to ensure that the population interviewed matches the severity of the clinical trial population as the severity of the disease can potentially significantly affect patient’s perception of meaningful change. Moreover, discussing hypothetical change versus being able to reflect upon real changes that patients have experienced may lead to the generation of different evidence. Therefore, exploring change in the context of a real-world setting where actual change could be expected to occur is most beneficial in collecting these insights.
Example of an interview study conducted externally to a clinical trial
As discussed during the symposium, a recent study in pediatric asthma involved using both CE and CD techniques when interviewing patients and caregivers in an observational, non-interventional setting [18]. The primary objective of this research was to generate evidence to support estimates of the individual within-patient changes that constitute an important or meaningful change in symptom-free days (SFDs) and rescue-free days (RFDs) and updated estimates on the Childhood Asthma Control Test (C-ACT) in pediatric asthma populations aged 5–11 years.
Semi-structured, qualitative interviews were conducted with children (aged 8–11 years old) who had asthma and their parents/caregivers, as well as independent caregivers of 5–7 year olds. Participants were asked to complete diaries before their interview. During the interview the children and caregivers were asked to talk about symptoms they experienced on a ‘good’ day and a ‘bad’ day with their asthma (i.e., use of CE style questions) and whether the difference between ‘good’ and ‘bad’ days was meaningful. Additionally, a calendar task [ Fig. 1] was used to help the children think about the hypothetical number of days that would comprise a “very bad”, “little bad”, and “very good” week with asthma. Finally, using the diary data collected 4–9 days before the study as a starting point for discussion, the interviewer used CD interview techniques and a ‘think aloud’ process to understand changes experienced over the period and whether the participant considered those changes meaningful and important.
Children and parents/caregivers provided generally concordant estimates of the number of days with asthma symptoms in ‘very bad’, ‘little bad’ and ‘very good’ asthma weeks, with the findings suggesting that a change of ≥1–2 days per week is meaningful and important, which was broadly concordant with the findings for RFDs. In addition to being able to discuss “important” change, parents/caregivers were also able to talk about the concept of “minimal” change, suggesting that 1–2 additional SFDs or RFDs would be the smallest meaningful improvement. Both parents/caregivers and children were able to articulate what a meaningful level of change would be on the C-ACT at the item level. This qualitative study generated C-ACT item-level meaningful change estimates in the region of a 1–3 category change.
This study highlights that the use of CE /CD interview techniques in semi-structured interviews conducted independently of a clinical trial can provide valuable insights regarding meaningful change and can represent a useful alternative when embedded exit interviews are not possible. Both children with asthma and their parents/caregivers could qualitatively articulate meaningful change in SFDs and RFDs.
In addition, this qualitative data was triangulated with distribution-based analyses performed using independent clinical trial data. Anchor-based methods were not possible due to a lack of adequately correlated anchors. For SFDs and RFDs the results were similar to the qualitative results (i.e., suggesting changes of 1–2 additional symptom or rescue free days in a seven-day period would be above measurement error), providing support for qualitative data generating enhanced confidence in quantitative results. The C-ACT analyses were based on the total score and ranged from 1.67–4.37 on the total score (0–27). The qualitative data however is not comparable here because the interviews focused on item level meaningful change as discussing the total score was considered too abstract for participants. The challenge of discussing meaningful change on multi-item PROs is perhaps even more pronounced in stand-alone qualitative studies where participants are not familiar with the target COA in the same way they would be in a clinical trial.
Semi-structured exit interviews conducted as part of a clinical trial
Exit or embedded interviews incorporated as part of a clinical trial are being utilised as a means of obtaining patient perspectives on whether actual changes experienced during a trial are meaningful and important. An advantage of exit interviews is that patients are familiar with the target COA from the clinical trial period and at least some of the patients have likely experienced change of some description over the study course. However, patients may not have insight into how the measure is scored or how a change in the totality of the measure might be relevant. Therefore qualitative discussion may be focused on the concepts measured and small versus large changes, as opposed to dialog surrounding a specific score change. Exit interviews can also be used to obtain patient-derived anchors for change that can be used to supplement analyses using clinical trial anchors and distribution-based statistical techniques [19]. Being embedded within the clinical trial offers the unique opportunity to explore the patient perception of the same changes as those that are being evaluated by the clinical trial endpoints, thus increasing interpretability.
There are several recent examples in the literature of exit interviews being used to explore clinically important change across a variety of therapeutic areas [20,21,22]. Challenges in conducting exit interviews include the potential for feedback obtained in the interview to conflict with clinical trial data, which may make the clear interpretation of study findings more difficult, particularly if the exit interview data highlights a lack of meaningful change from a patient perspective in those where change is observed on COA measures. There are also logistical challenges to consider including adverse event reporting, maintaining data privacy which will vary depending on whether the exit interviews are incorporated into the main trial protocol versus collecting the data in a follow-up study. The optimal timing of the interview in order to minimize recall bias is imperative. The interview should either be embedded in the trial when meaningful change is anticipated to have occurred or be conducted as close to a patient exiting the trial as possible. The discussion during the interview should be explicitly focused upon the trial period to avoid confusion with any other changes either before or after participation in the trial (for example, since discontinuing treatment). Additionally, recall bias can be minimized by prospectively incorporating entry questions/interviews at the start of the trial intervention to act as a ‘reliable’ anchor for areas of focus during the exit interview, and to reduce the bias of retrospective assessment [23].
Examples of exit interview studies
As discussed during the symposium, a recent example in Down syndrome (DS) involved using both CE and CD interview techniques with caregivers and expert clinicians in an exit interview context to explore clinically important change [24]. This study aimed to establish, via qualitative exit interviews, the level of change that is clinically important on the Vineland Adaptive Behavior Scales, Second Edition (Vineland™-II) [25] and the Repeatable Battery for the Assessment of Neuropsychological status (RBANS) [26] instruments in a DS population. Psychologists (n = 10) and caregivers (n = 8) who participated in a clinical trial were interviewed after they had exited the trial.
Overall, the study generated rich data and caregivers and psychologists were able to provide detailed insights into which domains/concepts are most important to improve. The communication domain of the Vineland™-II was rated as the most important domain to improve by the majority of participants asked (n = 5 caregivers and n = 5 psychologists) [ Fig. 2]. While such meaningful change insights were not grounded in score change on the target COA, understanding which concepts are most important to improve from a patient and caregiver perspective can be highly valuable and informative when considering endpoint selection where it is imperative that conceptually relevant and important aspects of a disease are being measured.
Notably, during this study, caregivers generally considered any change to be important (e.g. from not being able to perform an activity at all to doing something sometimes or with assistance) while psychologists provided larger meaningful change estimates. Despite the creative use of CE/CD interview techniques, the caregivers (and psychologists) also sometimes struggled to respond to questions focussed on complex and abstract scales. Therefore, despite caregivers and psychologists being familiar with this scale from the clinical trial, given its complexity, participants still struggled to articulate a meaningful score change on the instrument and this would likely be the case regardless of the qualitative method employed to discuss meaningful change. Future studies may benefit from a domain-level focus or use of vignettes to anchor qualitative responses in order to give respondents something tangible to consider.
Another example is an exit interview study that was conducted following the Xermelo clinical trial with a subset of 35 of the 135 trial participants. The findings indicated that 17 of the exit interview participants identified diarrhea as the single most important symptom to treat, and frequency and urgency was most often selected as the single most important aspect of diarrhea they wished to improve [27]. Moreover, anchor-based data generated during the exit interviews were evaluated alongside anchors included in the full trial sample to determine that the differences between the treatment and placebo groups in the change in frequency of bowel movements observed during the trial were meaningful and important. Thus, the exit interview sub-study provided confirmation that the primary endpoint was important to patients and provided additional anchors to aid interpretation of that endpoint. Exit interviews can therefore be used in multiple ways to better understand the meaningfulness of clinical trial data. More recently there have been a number of exit interview studies published and the field continues to learn about the most optimal approach to conducting and generating valuable insights using this data type [28, 29].
Focus groups
Focus groups can be a useful forum to discuss important change, where CE or CD techniques can be used. The advantages of focus groups include having the potential to reconcile conflicting data and reach consensus on key aspects of meaningful change that are important to the participants, while also reducing costs and timelines. However, individual differences may mean reaching consensus is challenging and, as in any focus group study, social desirability bias may influence the comments that some participants make. The participant(s) with the strongest opinions may unduly influence other participants and the findings. An alternative that potentially attenuates some of the challenges of face-to-face focus groups is to conduct the group discussion via an online discussion panel. This approach is likely to reduce bias, and also gives participants more time to consider their responses as well as enabling participants to contribute at a convenient time, despite having the potential to increase dropout rates and timelines. A further challenge (as outlined in section Semi structured interviews conducted externally to a clinical trial ) is that if participants have not experienced change, reflection on hypothetical change can be a challenge for participants and potentially less reliable. In addition, lack of familiarity with the target COA can be a barrier to informed discussion.
Examples of focus group studies
The use of focus groups to discuss clinically meaningful change is not commonplace in the literature due to a preference for individual semi-structured interviews. However, two studies in rare diseases have used focus groups with success to discuss this topic. The first study in spinal muscular atrophy adopted face-to-face focus groups to explore meaningful change in Type II and non-ambulant type III patient populations, associated with treatment of this condition [30]. Given the progressive nature of the condition, small changes in motor function were considered to have an important impact on quality of life and maintenance of current ability was considered to be a meaningful outcome. The second study, in Duchenne and Becker Muscular Dystrophy involved an online focus group (n = 12) as well as interviews (n = 4) with individuals and caregivers and an expert committee regarding meaningful outcomes that extend beyond function, using the Pediatric Outcomes Data Collection Instrument (PODCI) as a starting point [31]. Vitality/fatigue; mental health; peer relationships; self-efficacy and autonomy; and a sense of hope and purpose were identified as key themes. The authors concluded that such outcomes measured using Health Related Quality of Life (HRQoL) measures may not be responsive to change. Overall, both of these studies provided very important insights regarding concepts of importance to measure and the ability of outcome measures to detect such changes.
Vignettes (bookmarking/standard setting)
Vignettes are valuable for exploring meaningful change on a target COA as they provide a starting point for patients to evaluate meaningful change. The benefits of this approach have been noted primarily in relation to the modified bookmarking/standard setting approach [3]. The method does not require an interventional study and ties the threshold directly to the patient perspective, while giving respondents a starting point from which to consider their own symptoms. The rigorous approach advocated by Cook and colleagues requires a relatively large sample to support item response theory (IRT) based quantitative analyses and reaching a consensus on the threshold is not guaranteed [32]. While less robust, if logistical considerations do not allow for such an involved approach there is still value in taking a similar approach but with a smaller sample, using vignettes derived from qualitative sources, and placing more emphasis on the qualitative insights generated. However, irrespective of how the vignette is designed, the approach can be cognitively challenging for patients as it is asking patients to reflect on scenarios where there may be subtle differences between vignettes.
Examples of studies employing vignettes
With regards to examples in the literature, clinical vignettes for different severity levels on the NeuroQOL have previously been created from an IRT calibrated bank [32]. In this study patients with multiple sclerosis and experts were recruited. For severity levels associated with lower extremity function and sleep disturbance, patients and experts derived the same threshold. However, for upper extremity function and fatigue there were differences, with patients setting higher thresholds for more severe classifications of symptoms. This approach can also be extended to ask patients to consider their symptoms in context of the vignette and whether they are experiencing symptoms that are greater, the same, or lesser than the vignette description, and to then hypothetically consider a meaningful or important change in the context of the vignette [33]. In order to estimate clinically important improvement/decrement values, differences endorsed as enough to make a difference by > 50% were calculated. This approach was used by Cook and colleagues to establish meaningful change on the NeuroQoL fatigue short form [33]. The use of qualitative vignettes has also been employed in a recent study evaluating clinically meaningful change on a self-reported assessment of ‘worst itch’ in Atopic Dermatitis [34]. During semi-structured interviews three vignettes (as well as qualitative questioning) depicting mild, moderate and severe symptoms were used as a basis for patients to discuss meaningful change. Patients were asked if the itch in each vignette was the same, worse or better than their own current itch and if any differences were meaningful. In addition, patients were asked if there were meaningful differences among the vignettes. The smallest level of change that each patient considered meaningful could then be established. The majority of participants suggested either a two- or a three-point change was the smallest change that would be meaningful. Overall, patients found the vignette task easier to understand than the questions about meaningful change that used qualitative questioning, based on the patient’s own reported score on the numerical response scale.
Delphi panel method
Benefits and points for consideration of the Delphi panel method have been outlined in the literature and are generally consistent with a remote focus-group approach [16]. Advantages include the ability to reach a large number of individuals across a variety of locations, and by conducting this anonymously a single individual is less likely to bias the results. However, variability in execution of the Delphi procedure exists, including defining when sufficient consensus has been reached and attrition rates can be high. However, guidance to improve the optimal use and reporting of the Delphi method has been developed [16].
Examples of studies employing the Delphi panel method
There are numerous examples of using the Delphi panel method to define meaningful change in the context of clinical trials, most commonly with groups of expert clinicians. [35,36,37] The Delphi panel method has also been highlighted as a way to reach a consensus on a single MCID, where multiple estimates have been generated [38]. The method has also recently been used to establish meaningful or important change thresholds for PRO measures in the literature with an example in chronic obstructive pulmonary disease [35]. Here physicians were sent material on clinically meaningful change estimates from published literature/patient change scenarios and were asked to outline what they considered to be a minimal, moderate or large amount of clinically meaningful change. During subsequent rounds, additional literature was added and results synthesized. In general, the expert panel meaningful change estimates were larger than previous studies based on patient-reported clinically important differences.