The content validity of the ANMS GCSI-DD in patients with idiopathic or diabetic gastroparesis

Background The American Neurogastroenterology and Motility Society Gastroparesis Cardinal Symptom Index-Daily Diary (ANMS GCSI-DD) was developed to meet Food and Drug Administration (FDA) recommendations for patient-reported outcome (PRO) endpoints in gastroparesis studies, including therapeutic trials. The current version of the ANMS GCSI-DD contains five items pertaining to nausea, early satiety, post-prandial fullness, upper abdominal pain, and vomiting. The specific aims of this study were to determine if the appropriate symptoms are included in the ANMS GCSI-DD and to assess the content validity in patients with idiopathic (IG) and diabetic gastroparesis (DG). Methods Patients diagnosed with IG or DG were recruited by five clinical sites in the United States for a cross-sectional, qualitative study involving one-on-one in-person concept elicitation and cognitive debriefing interviews. Concept elicitation included open-ended questions to elicit patients’ symptoms and impacts of gastroparesis, while cognitive debriefing was designed to assess the comprehensiveness of the ANMS GCSI-DD and clarity of the instructions, items, and response scales. The interviews were audio-recorded and transcribed. Transcripts were analyzed using a content analysis approach with ATLAS.ti. Results Of 25 patients interviewed, 15 (60%) had IG and 10 (40%) DG. Mean age of the sample was 42.3 years (range: 20–70 years), and most patients were female (n = 19, 76%) and white (n = 19, 76%). During concept elicitation, patients endorsed the following signs and symptoms as relevant and important to their condition: early satiety (n = 25, 100%), post-prandial fullness (n = 25, 100%), nausea (n = 22, 88%), upper abdominal pain (n = 18, 72%), vomiting (n = 15, 60%), and bloating (n = 11, 44%). Many patients (n = 20, 80%) experienced day-to-day symptom change. During cognitive debriefing, patients confirmed the ANMS GCSI-DD content was comprehensive and reflective of their gastroparesis experience. Patients could easily select a response option and describe how they arrived at their answers. Overall, patients found the instrument’s instructions, recall period, items, and response options clear and understandable. Conclusions The ANMS GCSI-DD was easily understood, found to contain the most important symptoms for patients with IG and DG, and no changes were recommended. Results support the content validity of the ANMS GCSI-DD for clinical trials and clinical care among IG or DG patients. Electronic supplementary material The online version of this article (10.1186/s41687-018-0081-2) contains supplementary material, which is available to authorized users.


Background
Gastroparesis is a symptomatic chronic disorder of the stomach characterized by delayed gastric emptying when no mechanical obstruction is present [3,22]. Patients with gastroparesis often experience a variety of symptoms including early satiety, postprandial fullness, nausea, vomiting, and abdominal pain. Gastroparesis of unknown etiology (idiopathic gastroparesis; IG) accounts for the largest number of cases, but gastroparesis is also frequently associated with conditions such as diabetes (diabetic gastroparesis; DG). Gastroparesis may occur after various types of gastrointestinal surgeries, and is less frequently associated with other etiologies such as Parkinson's disease, collagen vascular disorders, and internal pseudo-obstruction. Gastroparesis is associated with reduced quality of life and impaired day-to-day functioning [38]. Data from the Rochester Epidemiology Project indicate that the prevalence of diagnosed gastroparesis is 24.2/100,000 individuals [12], but the prevalence is likely underestimated given that a large group of people with gastroparesis-like symptoms have never had a gastric emptying (GE) test to confirm their diagnosis. A study using data from 450 patients showed that delayed GE was estimated to occur in 1.8% of community subjects [33]. If this is correct, the prevalence of gastroparesis in the United States (US) is estimated at 5 million to 10 million individuals (approximately 3% of the total population) and is approximately four times higher in women compared to men [13].
A symptom questionnaire, the Gastroparesis Cardinal Symptom Index (GCSI), was originally developed through patient interviews and was tested and validated in university-based clinical practices for quantifying symptoms in gastroparesis [28,29]. The nine-symptom GCSI is based on three subscales (postprandial fullness/ early satiety, nausea/vomiting, and bloating) and represents a subset of the longer 20-symptom Patient Assessment of Upper Gastrointestinal Disorders-Symptoms (PAGI-SYM). The GCSI and PAGI-SYM both ask patients to describe the severity of their symptoms over the last two weeks. The GCSI is the most frequently used symptom severity measure in gastroparesis clinical trials and other clinical studies [1, 2, 4-6, 9, 14, 16, 19, 21, 35, 36], including the NIH Gastroparesis Clinical Research Consortium Gastroparesis Registry study [10,24].
The Food and Drug Administration (FDA) has been reevaluating symptom endpoints used in clinical trials of pharmaceutical agents for all disorders, including gastroparesis, and suggests using patient-reported outcome (PRO) measures to assess symptom endpoints. In July 2015, the FDA Gastroparesis Guidance for Industry was released; the guidance document contained recommendations for endpoints for gastroparesis studies [7]. A daily diary version of the GCSI, the American Neurogastroenterology and Motility Society Gastroparesis Cardinal Symptom Index-Daily Diary (ANMS GCSI-DD) was developed to meet these recommendations and is intended as a PRO measure used to assess symptom-based endpoints for new treatments in gastroparesis clinical trials [30,31].
The ANMS GCSI-DD was developed by the ANMS GCSI-DD PRO Committee, consisting of representatives of ANMS interested in gastroparesis and the creator of the original GCSI questionnaire. The initial ANMS GCSI-DD version was based on the existing two-week GCSI questionnaire. The development of the diary has involved changing the instructions, recall period, and response scales; however, the item stems remain the same. In response to suggestions from clinical experts and the FDA, the diary was revised to inquire about symptoms of upper abdominal pain, which was not present in the original GCSI. In some patients, abdominal pain and/or discomfort can be an important symptom [10,11], and has been used in other severity scores for gastroparesis [17,27]. The upper abdominal pain item was based on the PAGI-SYM questionnaire, which has been previously tested and validated in patients with different gastrointestinal disorders [29,32]. Suggestions from the FDA were also incorporated into the ANMS GCSI-DD including changing the vomiting assessment from severity to frequency. The ANMS GCSI-DD version used in the study reported herein is paper-based, uses a 24-h recall period, and consists of five symptoms: nausea, early satiety, post-prandial fullness, upper abdominal pain, and vomiting. Nausea, early satiety, post-prandial fullness, and upper abdominal pain are rated using a 5-point verbal rating scale, and the number of vomiting episodes over the last 24 h is recorded by the patient (see Additional file 1 for copy of the ANMS GCSI-DD).
Based on the FDA PRO guidance [8], recent FDA guidance on gastroparesis [7], and the FDA feedback to date during the qualification review of the ANMS GCSI-DD, it was necessary to conduct further evaluation of the content validity of the revised ANMS GCSI-DD. The FDA PRO guidance defines content validity by the empiric evidence demonstrating that items and domains of an instrument are appropriate and comprehensive relative to its intended measurement concept, population, and use [8]. Content validity is determined by demonstrating that the structure and content (i.e., items) capture the linkage between the intended measurement concept and the way patients from the population of interest comprehend and discuss the concept [25]. Collecting additional qualitative research provides further support for content validity of the ANMS GCSI-DD as a PRO tool to support a symptom-based endpoint for clinical trials involving diabetic (DG) or idiopathic (IG) gastroparesis patients. Therefore, the primary objective of the current study was to assess the content validity of the ANMS GCSI-DD with patients with DG or IG. The specific aims of the study were to determine if the appropriate symptoms were included in the ANMS GCSI-DD and to evaluate patients' understanding of the instrument's instructions, items, response options, and recall period among patients diagnosed with IG or DG.

Study design
To meet the study aims, a cross-sectional qualitative study was conducted that included concept elicitation and cognitive debriefing interviewing components. The study was approved by a central institutional review board (IRB) (Chesapeake; reference number: Pro00022109) and local IRBs at two of the five clinical sites. Regarding sample size, there are no quantitative statistical tests for identifying the minimum numbers of subjects to target for enrollment in qualitative interview studies. In order to determine if sufficient numbers of interviews have taken place to support the results, the goal is to achieve "saturation of concept," or reaching a point where no new information is forthcoming from the qualitative interview process. Generally, in samples that are relatively homogenous, saturation can be reached with small sample sizes (i.e., 10 patients). Therefore, it was anticipated the target of 25 gastroparesis patients would be sufficient to reach saturation of concept [15,25,26,37].

Recruitment procedures
Staff members at five clinical sites based in the US were trained by study investigators to identify, screen, recruit, and schedule eligible patients for interviews using a standardized screening script. Site staff identified potential patients through patient databases, medical records, and/or daily appointment schedules. Eligible patients included those who had: clinician-confirmed diagnosis of gastroparesis with documented delay in gastric emptying according to local criteria within the past two years, gastroparesis from either idiopathic or diabetic etiologies, and 18 years of age or older. Patients with gastroparesis from postsurgical causes and/or with lower gastrointestinal disease were excluded. All patients provided informed consent prior to participating in the interview and were remunerated a nominal amount for the time associated with their participation in the study.

Interview procedures
All interviews were conducted in person and took place at the clinical site locations. The interviews were conducted by three study staff trained in qualitative interviewing methodology, with a combined eight years of experience in conducting qualitative interviews. All interviews were conducted using a semi-structured interview guide with concept elicitation and cognitive debriefing components (Fig. 1).
First, the concept elicitation portion of the interview was conducted to elicit spontaneous descriptions of patients' symptoms of gastroparesis, and impact(s) of symptoms on their daily lives. Next, patients completed the paper ANMS GCSI-DD and were subsequently engaged in a cognitive debriefing discussion to evaluate the patients' interpretation of the instructions and items included in the diary, and to ensure that the response options and recall period were appropriate for capturing experiences of patients with IG or DG. Patients were also asked questions about the comprehensiveness, format, and relevance of the content of the ANMS GCSI-DD.
All interviews were conducted in English, took approximately 90 min to complete, and were audiorecorded and subsequently transcribed. Interviewers took detailed notes for each interview. Upon completion of the interview, each patient completed two case report forms (CRFs) that asked questions about patient sociodemographic characteristics and gastrointestinal symptoms in order to characterize the sample of patients in the study. Each clinical site completed a clinical CRF to collect information on relevant medical history of each patient to assist with describing the sample. The total sample size was divided into two rounds of interviews to allow for an interim data analysis in order to determine if changes to the ANMS GCSI-DD were deemed necessary.

Data analysis
Interview transcripts were assessed with a content analysis approach using ATLAS.ti version 7.5.18 qualitative data analysis software [20]. A coding dictionary was developed, based on the qualitative interview guide, and was supplemented by themes and concepts that emerged from the patient interviews. Using the coding dictionary, two researchers, including those who conducted the interviews, coded the transcripts to identify symptom concepts of gastroparesis and to examine the comprehension of the ANMS GCSI-DD instructions, recall period, items, and response options. A senior team member performed a quality check on a portion of the coded transcripts to ensure that all relevant responses were captured and to ensure that team members were coding consistently. Coded data were organized and summarized into an analysis grid with frequencies and percentages as appropriate. DG and IG patients were analyzed together and separately to determine any similarities and/or differences in their experience of gastroparesis symptoms or interpretation of concepts in the ANMS GCSI-DD.
All quantitative CRF data were entered into a validated study database. Descriptive statistics (mean, standard deviation, frequency, range using SAS version 9.4 [34]) were used to characterize the study sample in terms of sociodemographic and clinical characteristics.

Demographic and clinical characteristics
Twenty-five patients were interviewed for this crosssectional qualitative study conducted from August 2017 -November 2017 across five clinical sites located in California (n = 2), New York (n = 1), Pennsylvania (n = 1), and Massachusetts (n = 1). The sites consisted of both small medical clinics and large hospital-based centers.
Clinical characteristics of the sample are displayed in Table 2. On average, patients had been diagnosed with gastroparesis for 1.7 years (range < 1 y to 7 y). Most patients were rated by their clinician as having mild (40%) or moderate (36%) gastroparesis severity levels. All patients (n = 25, 100%) were currently receiving treatment for gastroparesis, with 44% on a gastric anti-secretory agent, 36% on a prokinetic agent, and 36% on an antiemetic agent. Most patients (n = 21, 84%) had no history of prior GI-specific surgery; the remaining four patients had GI-surgeries unrelated to their gastroparesis.

ANMS GCSI-DD cognitive debriefing results
Patients were interviewed in two rounds. Eleven patients were interviewed in round one. Based on a review of the interviewer notes from round one patients, no changes to the diary content were deemed necessary. The 14 round two patients were interviewed as a confirmatory cohort. Overall patient feedback on the ANMS GCSI-DD was positive. Patients reported that the diary was quick to complete, contained a reasonable number of questions, and was easy and straightforward to comprehend.

Instructions
Overall, patients demonstrated a clear understanding of the instructions for completing the nausea, early satiety, post-prandial fullness, and upper abdominal pain severity

Concept Elicitation
Cognitive Debriefing Please be sure to answer each question." When 23 patients were asked specifically if they thought about the worst experience, 19 patients (83%) correctly reported that they considered each symptom at its worst throughout the past 24 h. The remaining four patients (17%) responded that they thought about their symptoms as an average of the past 24 h, but noted that their answers would not change if they thought instead about their worst experience.
Item 5 (vomiting) had a separate set of instructions specific for vomiting that read: "The next question asks you to record the number of times vomiting occurred in the last 24 hours. Please record the number of vomits (throwing up with food or liquid coming out) that occurred in the last 24 hours. Record zero, if you have not vomited during the past 24 hours. If you vomited, write down the number of all vomits. If you vomited once, record one. If you vomited three times during the day, record three. If you vomited three times, whether it was during the same trip to the bathroom or three separate trips, record three as the number of episodes of vomiting." The majority of patients (n = 23, 92%) demonstrated a clear understanding of the instructions and correctly explained how they should be counting the number of vomiting episodes. The remaining patients (n = 2, 8%) stated that they should count the number of trips to the bathroom instead of counting the number of vomiting episodes. However, these two patients reported 30 or more vomiting episodes per day.

Recall period
There were no comprehension issues related to the 24-h recall period, and all 25 patients (100%) correctly reported that they considered their experiences over the past 24 h when answering the questions. Most patients stated that the most meaningful timeframe to think about the severity of their symptoms was the past 24 h (n = 15/24, 63%). Other patients suggested that the diary could ask about the past seven days (n = 4/24, 17%), previous night into the next morning (n = 1/24, 4%), only at the end of the day (n = 1/24, 4%), only in the morning (n = 1/24, 4%), the past month (n = 1/24, 4%), or before a gastroparesis procedure (n = 1/24, 4%).

Response options
The ANMS GCSI-DD utilizes a 5-point verbal rating scale for the nausea, early-satiety, post-prandial fullness, and upper abdominal pain items. The response options for the symptom severity items include none, mild, moderate, severe, and very severe. The vomiting frequency item has an open-ended field that allows for numerical responses of vomiting episodes, with a lower range of 0. All the patients had a clear understanding of both response scales and were able to easily select a response and describe how they arrived at their answers. Additionally, most patients (n = 19/25, 76%) endorsed meaningful changes between each response option. However, three patients (12%) found "very severe" and "severe" to mean the same thing, and an additional three patients (12%) reported no meaningful difference between "mild" and "moderate" response options.

Item level feedback
The item level feedback is displayed in Table 4. Specifically, the results on patient definition and understanding of the items, their description of the items, alternative or qualifying feedback on the items, and additional comments are reported.

Discussion
Overall, patients provided positive feedback on the ANMS GCSI-DD, and reported that the instrument was straightforward, comprehensive, and relevant to their experience with gastroparesis. In addition, the instructions, item wording, and response options were noted to be  well-understood by the majority of patients. Patients demonstrated that they could easily select a response option and describe how they arrived at their answers. The symptoms included in the ANMS GCSI-DD reflected the symptom experiences of patients with gastroparesis. Additionally, no significant differences in symptom experience were observed between patients diagnosed with DG or IG.
Some suggestions and recommendations were made by patients that require some reflection and discussion. First, bloating was endorsed by 11 (44%) patients as a relevant symptom of gastroparesis, four of which (16%) suggested adding this symptom to the diary. It should be noted that bloating was part of the original GCSI measure but was subsequently removed following FDA recommendations and previous research that indicated overlap in postprandial fullness and bloating concepts by gastroparesis patients [30]. Furthermore, the current study showed that less than half of patients endorsed bloating. Future clinical studies may include the bloating item as an exploratory symptom for gastroparesis.
Additionally, seven patients (28%) suggested that the ANMS GSCI-DD could be improved upon by adding a question about bowel movements (i.e., frequency, consistency, diarrhea, constipation). These recommendations are consistent with previous qualitative research on the GCSI and GCSI-DD [30]. However, bowel movement-related symptoms are not directly associated with the clinical manifestations of gastroparesis and may reflect other gastrointestinal comorbidities concurrent with gastroparesis in the study patients [18].
Regarding the overall instructions of the ANMS GCSI-DD, there were four patients (17%) who inadvertently considered the average of their symptom severity over the past 24 h as opposed to the worst severity. However, these patients mentioned that their answers would not have changed had they considered the worst severity instead of the average rating. The majority of patients (83%) understood the instruction meaning as intended, therefore no content changes are recommended to the instructions.
For Item 4, Upper Abdominal Pain (above the navel), patients were asked which term, "upper abdominal pain (above the navel)" or "abdominal pain" best captured their gastroparesis symptoms. Most patients (n = 12, 48%) endorsed "upper abdominal pain (above the navel)", while others preferred the term "abdominal pain" (n = 9, 36%) or had no preference (n = 2, 8%). Since there was no consensus, and since the majority of the sample (96%) demonstrated that they were able to differentiate among upper, lower, and overall abdominal pain, no content changes to the item are supported.
For Item 5, Vomiting, an episode of vomiting was well understood by the majority of patients (n = 23, 92%). However, two patients (8%) found it difficult to count the number of vomiting episodes in 24 h due to experiencing a high frequency of episodes. From a scoring and weighting standpoint, there are no major issues since the ceiling for the scale is four or more episodes of vomiting and adequately captures this increased frequency of vomiting. It should also be noted that 48% of the patient  sample did not experience issues with vomiting from gastroparesis, which may be related to the severity of their current gastroparesis condition [23]. The severity response scale used for the ANMS GCSI-DD is consistent with the recommended response scale in the FDA guidance on gastroparesis outcomes and endpoints [7]. Regarding the severity response options (None/Mild/Moderate/Severe/Very Severe), the majority of patients demonstrated a good understanding of and endorsed a meaningful difference between each response option. A few patients expressed that some of the response options were the same or very similar (e.g., mild and moderate, severe and very severe) and/or that they would consider a 2-point jump as a meaningful improvement (e.g., very severe to moderate) as opposed to a 1-point jump. However, no changes to the scale are recommended since the scale was well understood by the majority of patients and most patients interpreted a 1-point jump as meaningful improvement on the scale. In addition, previous item response theory analyses of the ANMS GCSI-DD with this response scale demonstrated good model fit and ordered response categories [31]. Guidelines for cognitive interviewing for new PROs recommend evaluating respondent understanding of the Patient "feeling sick to their stomach," "feeling like they will throw up," "sickness in the stomach," "upset stomach," and "queasy." One patient (4%) explained how she experiences different types of nausea and at times can experience nausea without feeling like she will throw up.
All patients thought the phrase in parentheses ("feeling sick to your stomach as if you were going to vomit or throw up") was a clear and appropriate definition for nausea.
Item 2: Not able to finish a normal-sized meal (for a healthy person) n = 24, 96% Patient unable to eat a normalsized meal, unable to finish a normal plate of food, a remature loss of appetite, or "getting full quickly." Patients explained that a regular meal for them would entail a half-sized portion of a "normal" meal such as half a cup of soup, half a sandwich, or half of a burger.
Patients correctly defined a normal-sized meal "for a healthy person," explaining that it would entail a "wellbalanced meal" consisting of "multiple servings or courses" or the "right" amount of protein, carbohydrates, vegetables, and fruit or dessert.
Item 3: Feeling excessively full after meals n = 25, 100% Patient terms such as "stuffed" or feeling like there is "too much food" in their stomach.

No other comments
Item 4: Upper abdominal pain (above the navel) n = 25, 100% Patient pain above the navel/ belly button or near the middle of the stomach and further described the feeling of pain as "sharp," "deep," "dull," "burning," and "uncomfortable." Most patients (n = 12, 48%) endorsed "upper abdominal pain (above the navel)", as appropriate to capture their symptoms, while others preferred the term "abdominal pain" (n = 9, 36%) or did not have a preference (n = 2, 8%).
All patients expressed understanding of the term "navel." Item 5: During the past 24 h, how many episodes of vomiting did you have? n = 25, 100% Patient "episode" of vomiting as food or liquid being expelled from the mouth.
Most patients (n = 15, 60%) agreed that it was important to assess the frequency of vomiting, while others suggested that the item could assess the severity of vomiting (n = 3, 12%) or ask about retching and dry heaving (n = 3, 12%) as an indicator of symptom severity.
All patients demonstrated they could answer the vomiting item using the written response option space, and most patients (n = 23, 92%) provided a clear explanation for how to count their episodes of vomiting over the past 24 h. Two patients (8%) reported that they would experience some difficulty with counting their episodes of vomiting over the past 24 h due to high frequency of vomiting episodes.

Funding
This work was supported by funding from Takeda Pharmaceuticals. All aspects of the study design, interpretation, and decision to submit for publication were determined by the authors.

Availability of data and materials
The data collected during this study is kept in a locked, secure facility and is unavailable to the public to ensure that confidentiality is maintained. Reasonable requests to review the data for scientific and/or research purposes will be considered. Ethics approval and consent to participate All methods and materials, including the study protocol, were approved by an Institutional Review Board (Chesapeake IRB, Columbia, MD 21046 [now Advarra]), reference number: Pro00022109. Written informed consent was obtained from all patients prior to initiating the interview.

Consent for publication
Not applicable.
Competing interests JP and CA are employees of Takeda Pharmaceuticals, the study sponsor who provided funding for this study. The study sponsor was involved in the design of the study, interpretation of the data, and preparation, review and approval of the manuscript. DAR, RMS, and SL are employed by Evidera, a company that received funding from Takeda for time spent on this research. As Evidera employees, they work with a variety of companies and organizations and are expressly prohibited from receiving any payment or honoraria directly from these organizations for services rendered. MC, HPP, and BK serve as advisors to Takeda Pharmaceuticals and conduct research supported by Takeda Pharmaceuticals. They receive no personal financial remuneration.

Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.