Re-examining the factor structure of the Insomnia Severity Index (ISI) and defining the meaningful within-individual change (MWIC) for subjects with insomnia disorder in two phase III clinical trials of the efficacy of lemborexant

Abstract

Background

The Insomnia Severity Index (ISI) is a widely used measure of insomnia severity. Various ISI research findings suggest different factor solutions and meaningful within-individual change (MWIC) to detect treatment response in patients with insomnia. This study examined an ISI factor solution and psychometric indices to define MWIC in a robust patient sample from clinical trial settings.

Methods

We endeavored to improve upon previous validation of ISI by examining structural components of confirmatory factor analysis (CFA) models using two large, placebo-controlled clinical trials of lemborexant for insomnia. Using the best-fitting two-factor solution, we evaluated anchor-based, distribution-based and receiver operating characteristic (ROC) curve methods to derive an estimate of the MWIC.

Results

The model structure for the 7-item scale proposed in other research did not fit the observed data from our two lemborexant clinical trials (N = 1956) as well as a two-factor solution based on 6 items did. Using triangulation of anchor-based, distribution-based, and ROC methods, we determined that a 5-point reduction using 6 items best represented a clinically meaningful improvement in individuals with insomnia in our patient sample.

Conclusions

A 6-item two-factor scale had better psychometric properties than the 7-item scale in this patient sample. On the 6-item scale, a reduction of 5 points in the ISI total score represented the MWIC. Generalizability of the proposed MWIC may be limited to patient populations with similar demographic and clinical characteristics.

Background

Insomnia is a widespread problem, with studies estimating a global prevalence of chronic insomnia between 10% and 15% of the population, with an additional 25–35% suffering from transient insomnia [1]. It is characterized by persistent difficulties with initiating sleep at bedtime, frequent or prolonged awakenings, or early-morning awakenings with inability to return to sleep, despite adequate opportunities to sleep, or some combination of these problems [2]. Such sleep disruptions result in clinically significant impairment of daytime function, including fatigue, decreased energy, mood disturbances, and reduced cognitive function.

The Insomnia Severity Index (ISI) is a well-known and widely used patient-reported outcome (PRO) measure used to assess insomnia symptoms and daily functioning in clinical trials. The reliability and validity of the ISI have previously been established, however, the factorial structure remains a topic of active interest [3,4,5,6,7]. For example, one study of adult insomnia patients in Taiwan, China, and Canada found the ISI to have three factors, while another study in Chinese adolescent patients found a structure of two factors [4, 5]. Other findings suggest that the ISI items may not all similarly contribute to overall insomnia severity, at least in some selected patient populations [6].

In addition to variability in the proposed factor structures and resulting ambiguity about scoring of domains, there is no consensus on the threshold(s) for clinically meaningful change. The few studies that have attempted to define a threshold for the clinical meaningfulness of changes on the 7-item ISI report estimates ranging between 6-point and 8.4-point reductions [6, 7]. Further research on the interpretability of the ISI scores is also important for clinical research on insomnia treatment.

The primary objectives of the current analysis were to evaluate the ISI factor structure using confirmatory factor analysis (CFA), and then to evaluate the psychometric properties of the ISI based on the factor structure that best fitted our patient sample. A secondary objective of this analysis was to evaluate thresholds for the clinically meaningful within-individual change (MWIC) of the scores derived from the revised ISI.

Methods

Study design

We used data from two Eisai pivotal trials of lemborexant (LEM) for insomnia disorder. SUNRISE-1 (NCT02783729; E2006-G000-304) was a 1-month, double-blind, randomized, placebo (PBO)- and active-controlled, parallel-group study [8]. SUNRISE-2 (NCT02952820; E2006-G000-303) was a 12month (6-month PBO-controlled, followed by 6-month active treatment only) double-blind study [9]. In both clinical trials, the ISI was completed at baseline, at the end of Month 1, and for SUNRISE-2, at Months 3, 6, and 12. The baseline ISI values were derived just prior to the mid-run-in polysomnogram in SUNRISE-1 and from the scores at the end of the PBO run-in for subjects in SUNRISE-2.

Study population

Participants included in this analysis were 1006 participants (ages: women ≥ 55 years and men ≥ 65 years) in SUNRISE-1 and 950 participants (age ≥ 18 years) in SUNRISE-2. Participants met Diagnostic and Statistical Manual of Mental Disorders, 5th edition criteria for insomnia disorder. Individuals diagnosed with comorbid sleep disorders including sleep apnea, periodic limb movement disorder, restless leg syndrome, circadian rhythm sleep disorder or narcolepsy, and individuals with a history of complex sleep -related behavior, were not eligible for the trials. However, subjects with stable medical or psychiatric conditions that would not interfere with participating in the study were eligible to participate.

Measures

The ISI is a patient-reported 7-item questionnaire assessing insomnia symptoms and its impact on daytime functioning. Patients are asked to think about their current (i.e. last two weeks) insomnia symptoms and impacts. The items are focused on difficulties related to sleep onset and maintenance, problems related to early-morning awakening, satisfaction with sleep pattern, noticeability of sleep problems by others, degree of distress or concern caused by sleep difficulties, and interference of sleep difficulties on daily functioning [6]. The items can be rated from 0 to 4 (0 = no problem; 4 = very severe problem), with a total score ranging from 0 to 28. The total score can be interpreted as follows: absence of insomnia (0–7), subthreshold insomnia (8–14), moderate insomnia (15–21), and severe insomnia (22–28). Previous research has identified at least two clinically important thresholds for the ISI total score: Morin et al. proposed a change of −8.4 points as a moderate improvement, while Yang et al. proposed a change of −6 points as a minimally clinically important difference [6, 7].

Patient Global Impression–Insomnia (PGI-I) provides a self-reported questionnaire assessing the patients’ perception of a medication’s effects on sleep after treatment compared with their sleep prior to treatment initiation [10]. Hence, the three PGI-I items are related to the benefit of study medication as perceived by patients: (i) Helped/Worsened Sleep, (ii) Increased/Decreased Time to Fall Asleep, and (iii) Increased/Decreased Total Sleep Time (TST). Each of these items are rated on a 3-point categorical scale for medication effect: positive, neutral, and negative. PGI-I scores at Month 1 were utilized for the analyses presented here.

Statistical analysis

Demographic and clinical characteristics of the patient sample at baseline were calculated using mean, standard deviation (SD), and range for continuous variables, and percentages and frequencies for categorical variables.

We examined the distribution of the responses on the 7 items of the ISI as well as the mean and median for the ISI total score, Insomnia Symptoms, and Daytime Functioning scores at baseline. Additionally, the change in ISI total score from baseline to Month 1 was assessed in each treatment group.

A CFA is used in psychometric research to test if data fit a hypothesized model [11, 12]. We hypothesized that a two-factor solution would best fit the ISI model based on an evaluation of the content of the scale: the first three items measure different insomnia patterns (difficulty falling asleep, remaining asleep, and early-morning awakening), while the remaining four measure daytime functioning. We conducted a CFA to assess the best-fitting factor structure of the ISI and then used the scoring algorithm that was defined based on the results of the CFA. To evaluate the factor structure, the comparative fit index (CFI, minimum threshold of 0.9) and root mean square error of approximation (RMSEA, maximum acceptable threshold of 0.09) were examined. In our approach, we also used modification indices (MIs) to demonstrate an improvement in model fit when the suggested modifications to the model specifications were introduced. The MI indicated how much the model could be improved if a particular path were added to the model or a constraint removed; a value of > 3.84 is considered to represent a statistically significant improvement [11]. In this approach, by removing an item with residual correlations, we increased the model precision by reducing measurement error. This statistical approach was largely driven by the US Food and Drug Administration (FDA) guidance for PRO, which suggests that a PRO instrument should be “fit for purpose” for the target population [13].

To define a threshold for a MWIC, a multistep triangulation approach was used. Anchor-based methods were utilized to determine the possible range, then receiver operating characteristic (ROC) methods were used to refine the range to a single value. Distribution-based methods were evaluated as supplementary for the MWIC. The anchor-based method was based on three items from the PGI-I. The anchor-based method is a preferential method according to the FDA [13] given that it directly considers the patient’s voice to determine within-individual change. Within-individual change is the average amount of change experienced by patients within the same group or treatment arm.

The distribution-based methods included half the SD at baseline and the standard error of measurement (SEM). Although these are group-based methods, they can still be used as supportive in establishing measures of change within individuals.

The ROC method compared the predictive power for different ISI cutoffs defined using the three PGI-I items (Helped/Worsened Sleep, Increased/Decreased Time to Fall Asleep, and Increased/Decreased Total Sleep Time). For each ROC analysis, the dependent variable “Responder” was defined as “positive medication effect” on the PGI-I. The optimal cutoffs were selected using the data on sensitivity, specificity, and the Youden index. The Youden index is often used as a criterion to select optimal cutoff values across varying levels of sensitivity and specificity. Technically, it gives equal weights to false-positive and false-negative values and thus provides information on the proportion of the total misclassified for a specific cutoff, which can then be compared across different cutoffs. This index ranges from 0 to 1, and the highest value serves as an indicator of an optimal test performance. The probability density function (PDF) by treatment arms has also been used as supporting evidence to establish the threshold for clinical meaningfulness (Fig. 2).

The main objectives of the study were to re-examine the factor structure of the ISI instrument and to define the MWIC for the targeted population—individuals with insomnia disorder from the two clinical trials, SUNRISE-1 and SUNRISE-2.

Results

Patient demographics and clinical characteristics

The mean age of the pooled study population was 59.3 (SD 11.8), and the majority were female (77.6%) and White (71.9%) (Table 1). In the two trials, the mean ages and range were as follows: SUNRISE-1: mean age: 63.93 (SD 6.81) and range: 55–88; SUNRISE-2: mean age: 54.49 (SD 13.8) and range: 18–88. At baseline, the mean ISI score was 18.8 (SD 3.7) in the pooled trials, indicating that, on average, participants of both studies had moderately severe insomnia, based on their self-perception (Table 2). In both SUNRISE-1 and SUNRISE-2, patients’ responses to individual item scores within Factor 1 indicated that the majority of patients also had moderate to very severe insomnia according to the three patterns of sleep disturbance: sleep-onset problems (90.6%); sleep maintenance difficulties (98%); and early-morning awakening problems (88.2%) (Supplementry Table 1). The impact of insomnia on patients was large at baseline according to the Factor 2 items: 94.0–97.7% of subjects across the two studies reported to be dissatisfied or very dissatisfied with current sleep pattern (item 4); 29.5–37.8% thought their sleep problem was much or very much noticeable by others (item 5); 65.9–73.0% had much or very much distress or concern caused by sleep difficulties (item 6); and 45.0–56.9% had much or very much interference of sleep difficulties with daily functioning (item 7) (Supplementry Table 1).

Table 1 Demographic characteristics and sleep metrics at baseline

Table 2 Description of baseline ISI total and daytime functioning scores in SUNRISE-1 and SUNRISE-2 (N = 1956)

Change from baseline at Day 31/Month 1

After 1 month of treatment, the active treatment groups showed a higher proportion of patients dropping two or three categories of insomnia severity compared with PBO (a shift down by one category of severity is defined as a clinically significant benefit from treatment) (Table 3). For the pooled trial, at Day 31/Month 1, the percentage of subjects shifted to the moderate category (ISI total score 15–21) were 45.4% for PBO, 32.9% for lemborexant 5 mg (LEM5), 23.0% for lemborexant 10 mg (LEM10), and 31.7% for zolpidem tartrate (ZOL). A smaller but still considerable number of subjects jumped over two categories and shifted to “subthreshold insomnia” (ISI total score 8–14): 23.1% for PBO, 27.3% for LEM5, 31.1% for LEM10, and 31.7% for ZOL. A few subjects jumped over three categories and shifted to the “No clinically significant insomnia” (ISI total score 0–7): 12.0% for PBO, 24.5% for LEM5, 32.8% for LEM10, and 31.7% for ZOL.

Table 3 Change in ISI total score from baseline to Month 1/Day 31

Confirmatory factor analysis

A preliminary CFA was conducted on the pooled trial dataset using the prespecified seven ISI items in our two-factor measurement model (Factor 1, items 1–3; Factor 2, items 4–7) (Table 4 and Fig. 1A). Figure 1A shows the hypothesized two-factor structure along with the item loadings on each factor. For this model, goodness of fit indices of the 7-item model indicated weak structure with a CFI of 0.862 and excessive model error with an RMSEA of 0.126 (90% confidence interval [CI]: 0.116–0.137; Table 4). Examination of the highest MIs for this model revealed the presence of very highly correlated residuals between ISI items 5 (noticeability of sleep problem to others) and 7 (extent to which current sleep problem interferes with daily functioning) (MI = 348.8, which was much larger than the cut-off for significance of 3.84) (Table 5).

Table 4 Confirmatory factor analysis of the ISI with and without item 5

Fig. 1

Two-factor confirmatory factor analysis models (N = 1956) A with 7 items; B with item 5 removed. The figures show the latent constructs of sleep problems and daytime functioning (Factors 1 and 2), and the item factor loadings on each item of the ISI

Table 5 Confirmatory factor analysis of ISI within-factor item–item modification indices, FAS in pooled trial (N = 1956)

As a result of this analysis, it seemed appropriate to drop item 5 (0.660) instead of item 7 (0.787) to improve model fit, given the lower factor loading associated with item 5 (Table 6). It also made sense conceptually, as item 5 is the only item that does not ask for a direct self-assessment compared with the other items; rather, it asks the respondent to rate how they believe others perceive them. This is potentially more prone to inaccuracy than a self-assessment might be, decreases precision of the instrument, and in return “weakens” the outcome summary measure [14].

Table 6 Anchor-based definition of meaningful change from baseline to Month 1/Day 31

The reduced 6-item model without item 5 demonstrated an acceptable model fit. The CFI improved substantially to 0.956 and exceeded the minimum threshold value of 0.9; the RMSEA was below the maximum acceptable threshold (< 0.09) at 0.075 (90% CI: 0.062–0.089). All loadings were above 0.4 (Table 4 and Fig. 1B). The 6-item model included a two-factor structure, with a factor loading on the overall general factor of 0.904 for the Insomnia Symptoms domain (items 1–3) and 0.871 for the Daytime Functioning domain (items 4, 6, 7).

Threshold for clinically meaningful change

Based on the findings from the CFA, the results below are provided for the 6-item ISI total score, omitting item 5.

Anchor-based methods

The PGI-I, which was utilized for the anchor-based methods, had moderate to strong correlations with the PGI-I items at Day 31/Month 1 (r = 0.43 to 0.60; p < 0.0001).

The anchor-based method with the PGI-I item “Helped/Worsened Sleep” demonstrated that the mean changes (SD) in ISI total score for medication effect at Day 31/Month 1 for the pooled trials were as follows: positive, −8.15 (4.98); neutral, −3.59 (3.37); and negative, −1.67 (2.91) (Table 6). When used with the PGI-I item “Reduced Time to Fall Asleep,” the anchor-based method demonstrated mean changes in ISI total score of −7.49 (5.10) for positive, −4.54 (4.13) for neutral, and −2.94 (4.04) for negative medication effects at Day 31/Month 1. The estimates for “Increased Total Sleep Time” were −8.15 (5.04) for positive, −3.74 (3.23) for neutral, and −2.40 (3.49) for negative medication effects.

The mean changes in ISI total scores for the neutral and positive medication effects responses suggested that the MWIC for the 6-item total ISI, should be in the range of −4 to −8. This wide range is associated with the fact that the anchors from the PGI-I item have only three levels, which include negative, neutral, and positive medication effects. Given that the positive effect is an average of all “positive medication effect” responses, a positive effect would include both large positive and minimal positive effects, which suggests the MWIC value should be between the neutral and the positive range.

Distribution-based methods

Distribution-based methods were applied to provide evidence about measurement variability per FDA recommendations [13]. The SEM for the 6-item total ISI score was determined to have a value of 1.70. Additionally, the 0.5 SD at baseline was 1.55 for the pooled trials (Table 7). The distribution-based methods gave a much more restricted estimate of the MWIC, suggesting an MWIC of between 0 and −1.5.

Table 7 Distribution-based methods to identify change thresholds from baseline to Month 1/Day 31

ROC method

The ROC analysis showed that the cutoff at the ISI total score up to −5 had the highest Youden’s index (0.49), compared with the other cutoffs, to predict the individuals with the positive medication effect on the PGI-I item “Helped/Worsened Sleep” using the data for the pooled trial at Day 31/Month 1 (Table 8). The sensitivity and specificity were 0.76 and 0.74, respectively. An MWIC of −5 falls within the range observed for PGI-I items “Reduced Time to Fall Asleep” and “Increased Total Sleep Time.” Thus, these approaches provided strong evidence for an MWIC of −5 for the ISI total score.

Table 8 ROC analysis of the change in 6-item ISI total score predicting PGI-I Item 1 at Month 1

Evaluating treatment differences in the context of the proposed MWIC

The differences in change in the ISI total score between the treatment arms and PBO were examined in PDF plots to provide supporting evidence for the results above (Fig. 2). At Day 31/Month 1, the mean changes in ISI total score were −4.5 for PBO, −6.2 for LEM5, −6.3 for LEM10, and −6.6 for ZOL (Fig. 2), supporting an MWIC of −5 estimated by the triangulation approach. For example, the range of −3.6 to −4.5 represents an average change for PBO. An MWIC of −5 exceeds the average PBO response, which is appropriate, as the MWIC should not fall within the expected range of a PBO response. Using the MWIC of −5, the distribution of the responders was as follows: 119 (40.2%) for PBO, 179 (59.5%) for LEM5, and 163 (56.8%) for LEM10 for SUNRISE-2 at Month 1; and 86 (43.2%) for PBO, 156 (60.7%) for LEM5, 140 (55.6%) for LEM10 and 156 (63.9%) for ZOL for SUNRISE-1 at Day 31.

Fig. 2

PDF plot of the change in 6-item ISI total score from baseline to Month 1/Day 31 by treatment arm. FAS (N = 1956). FAS Full Analysis Set, ISI Insomnia Severity Index, LEM5 lemborexant 5 mg, LEM10 lemborexant 10 mg, PBO placebo, PDF probability density function, Q quartile, ZOL zolpidem tartrate

Summary: clinically meaningful within-individual change

The average of the MWIC estimated through triangulation was −5.0 (Fig. 3), indicating that a reduction of ≥ 5 points in total ISI score represents a clinically meaningful change in the SUNRISE-1 and SUNRISE-2 patient sample. The distribution-based method as a secondary approach demonstrated that the MWIC was on the conservative side based on this method.

Fig. 3

Plot of 6-item ISI total score MWIC estimates. ISI Insomnia Severity Index, PGI-I Patient Global Impression, ROC receiver operating characteristic, SD standard deviation, SEM standard error of measurement

Discussion

In this study, we identified a two-factor structure without item 5 (“How noticeable to others do you think your sleep problem is in terms of impairing the quality of your life?”) as the optimal ISI structure in the context of the lemborexant clinical trial population. An MWIC of −5 was determined as the clinically meaningful change threshold for the total ISI score for this particular patient sample based on the triangulation approach.

Model structure

The FDA guidance on PROs emphasizes the importance of having an internally consistent structure reflected in the scoring [15]. This is important because a less-than-optimal factor structure could generate additional error thus reducing the ability of a scale to detect treatment effects. We conducted this analysis to further examine the best factor structure for the ISI, as there currently is some disagreement in the literature [3,4,5].

We used MIs to specify structural modifications to measurement models influenced by kurtotic (or skewed) non-normal data in addition to extraction and rotation methods, as well as fit statistics commonly used in a CFA [16]. This method was used to rectify the structurally weaker 7-item solution (CFI < 0.90 and RMSEA > 0.12) associated with high MI statistics for item 5. The resulting two-factor model provided modification indices and expected parameter change (EPC) that pointed to a high residual correlation between item 5 and item 7 (MI 348.84; EPC 0.376). Removal of item 5 also removed the largest source of structural error in the resulting 6-item ISI model, which then consisted of a stronger, internally consistent daytime function scale and a symptom factor (items 1–3) that differentially characterizes the subtypes of insomnia.

We then explored effects of correlated residuals between daytime functioning and among the insomnia subtypes (identified by items 1–3). Interestingly, we found that the correlated residuals between items 5 and 7 did not affect the ISI measurement model for those with sleep-onset problems, while correlated residuals between items 5 and 6 did affect the ISI model but to a much lesser extent (MI 5.49; EPC 0.111). Once again, when item 5 was removed, the model associated with sleep-onset problems also improved. At this time, the reasons for this difference are not well understood and the findings will need further investigation.

These analyses were also repeated in the SUNRISE-1 and SUNRISE-2 trials separately (Supplementry Figs. 1 and 2). In both trials, there was a high residual correlation between item 5 and item 7, and the model fit improved with the removal of this item. Furthermore, the two-factor solution based on 6 items was supported in both trials.

In addition to the structural issues, it seemed logical to drop item 5 and keep item 7 (“To what extent do you consider your sleep problem to interfere with your daily functioning currently?”), as item 5 asks for others’ perception of the impact of insomnia on the survey responder’s quality of life, whereas item 7 asks the responder directly for their own self-perception.

The issues we noted with item 5 might explain the diversity of findings in the literature pertaining to the factor structure of the ISI.

Clinically meaningful change threshold

In a recent FDA guidance [13], the advantages of using anchor-based methods to establish clinical meaningfulness thresholds have been reiterated. The guidance emphasizes the use of clinically relevant benchmarks and takes into account within-patient meaningful change. In this study, the PGI-I items were used as anchors to cover the multidimensional insomnia spaces and seemed to work well.

A slight drawback for the implementation of this method was too few response options. There was only one response option for the “positive” medication effect, which thus may not have included a minimal “meaningful change” but rather a range of positive responses. For example, for the PGI-I item “Helped/Worsened Sleep,” the median for the positive medication effect was −8 (interquartile range −12, −5). Thus, we additionally applied the ROC method, which is based upon modeling with the PGI-I items. In contrast to the anchor-based method, the ROC method has not been recommended by the regulators as a primary method; but it is often used to define the best performing responder threshold in terms of sensitivity and specificity. Although a distribution-based method was also used, it does not include information incorporating the patient voice and thus has not been considered a primary method for defining thresholds. As a result of a careful application of the three methods, triangulation has supported a threshold MWIC of −5 in this study. The results are not presented here for the sake of parsimony, but similar methods (anchor-based, distribution-based, and ROC methods) were used to define a meaningful threshold of -3 for each of the ISI domains.

The MWIC identified in our study using the 6-item model was different from the thresholds identified by others. Morin et al. proposed a change of −8.4 points as a moderate improvement, while Yang et al. proposed a change of −6 points as a minimally clinically important difference [6, 7]. This discrepancy might be explained by differences in the number of items included in the model and the demographics of the patient sample. First, both studies deduced the MWIC based on a 7-item ISI model, whereas we used a 6-item model. Second, the patients in our sample were on average at least 10 years older (mean age 59.3 years) than those in the Morin et al. study (mean age < 50 years) and those in the Yang et al. study (mean age 45.6 years). Furthermore, there was a higher percentage of females in our patient sample (77.6%, compared with 61.2% in the Morin et al. study).

The results of our study suggest that the factor structure, and therefore the MWIC based on this analysis, may be more robust than has been observed in the literature to date. Rather than hastily recommending that researchers switch to this version of the ISI, we suggest that further research may be valuable. This would be easy to accomplish, as the 6-item version is derived simply from dropping item 5 from the 7-item version.

Conclusions

Our hypothesized two-factor solution had the best fit when item 5 was dropped. The final ISI model we present in this study included the following two domains: Insomnia Symptoms (items 1–3) and Daytime Functioning (items 4, 6, 7). The reduced ISI score is reliable and valid for the studied population. These results are applicable to the specific setting of the SUNRISE-1 and SUNRISE-2 clinical trial populations and demonstrate that a 6-item ISI is a viable option but is not intended to replace the 7-item ISI at this time. Further research is warranted to determine whether the factor structure we have proposed could be more sensitive to treatment effects than the legacy scale. It could also be useful to conduct additional qualitative research to see whether the rewording of selected items may improve the scale performance of the full scale and clarify the scale structure.

The best estimate of MWIC observed in the current study for the 6-item ISI total scale is −5. This estimate was derived using three common methods for establishing meaningful change thresholds. Identifying the level of change associated with a meaningful within-individual change help to stimulate further research into the clinical meaningfulness of changes associated with insomnia treatments and eventually to assess the impact of meaningful change in insomnia on the consequences of insomnia, such as cognitive deficits, emotional distress, and mood disorders.