Development and content validation of the Symptoms Evolution of COVID-19: a patient-reported electronic daily diary in clinical and real-world studies

Abstract

Background

At the onset of the COVID-19 pandemic, there was limited understanding of symptom experience and disease progression. We developed and validated a fit-for-purpose disease-specific instrument to assess symptoms in patients with COVID-19 to inform endpoints in an interventional trial for non-hospitalized patients.

Methods

The initial drafting of the 23-item Symptoms Evolution of COVID-19 (SE-C19) Instrument was developed based on the Centers for Disease Control and Prevention symptom list and available published literature specific to patients with COVID-19 as of Spring 2020. The measurement principles outlined in the Food and Drug Administration (FDA) Patient-Reported Outcomes (PRO) guidance and the FDA's series of four methodological Patient-Focused Drug Development guidance documents were also considered. Following initial development, semi-structured qualitative interviews were conducted with a purposive sample of 30 non-hospitalized COVID-19 patients. Interviews involved two stages: (1) concept elicitation, to obtain information about the symptoms experienced as a result of COVID-19 in the patients’ own words, and (2) cognitive debriefing, for patients to describe their understanding of the SE-C19 instructions, specific symptoms, response options, and recall period to ensure the content of the SE-C19 is relevant and comprehensive. Five clinicians treating COVID-19 outpatients were also interviewed to obtain their insights on symptoms experienced by patients and provide input on the SE-C19.

Results

Patients reported no issues regarding the relevance or appropriateness of the SE-C19 instructions, including the 24-h recall period. The comprehensiveness of the SE-C19 was confirmed against the conceptualization of the patient experience of symptoms developed in the qualitative research. Minor conceptual gaps were revealed to capture nuances in the experience of nasal and gustatory symptoms and systemic manifestations of sickness. Almost all items were endorsed by patients as being appropriate, well understood, and easy to respond to. The clinicians largely approved all items, response options, and recall period.

Conclusions

The qualitative research provided supportive evidence of the content validity of the SE-C19 to assess the symptoms of outpatients with COVID-19, and its use in clinical trials to evaluate the benefit of treatment. Minor changes may be considered to improve conceptual clarity and ease of responding.

Background

Coronavirus disease 2019 (COVID-19) is a disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and was declared a pandemic by the World Health Organization (WHO) in March 2020 [1]. Initial reports of common COVID-19 symptoms included fever, cough, sputum, and fatigue [2]. By May 2020, the Centers for Disease Control (CDC) proposed additional symptoms indicative of possible COVID-19 infection: shortness of breath or difficulty breathing, muscle or body aches, headache, loss of taste or smell, sore throat, congestion or runny nose, nausea or vomiting, and diarrhea. These guidelines are consistent with other national guidelines, such as the National Health Service (NHS) [3], and recent studies have confirmed that these symptoms can be key predictors of a positive COVID-19 test [4].

To date, studies and guidelines have focused predominantly on hospitalizations and deaths; however, it is equally important to understand the emotional and physical effects of COVID-19 on outpatients' lives [5]. Furthermore, patient-reported outcomes (PROs) of health status are needed to support accurate measurement of COVID-19 symptoms, document their severity, and assess resolution [6], and are central to COVID-19 response and recovery [7]. This would be particularly useful in clinical trials [7, 8] and real-world studies/registries in outpatients [9]. At the time of study inception (May 2020), a fit-for-purpose, disease-specific symptom PRO measure did not exist. We therefore sought to develop a PRO measure specific to COVID-19. The COVID-Q [10] and the COVID Symptom Index (CSI) [11] have since been identified as novel instruments which have recently been developed to evaluate patient-reported symptoms of COVID-19. Such instruments potentially play an important role in assessing and monitoring the symptoms experienced by patients. However, patient input in the development of these instruments is lacking. While the CSI included limited patient input to detect language misunderstandings, the content validity, that is, the extent to which an instrument measures an individual’s clinical, biological, physical, or functional state or experience that it is intended to capture [12] was not assessed. For a PRO measure to adequately capture the patient’s experience, input from the target patient population should be evidenced throughout its development and evaluation [12]. Furthermore, the CSI and COVID-Q were not developed or validated in the English language. Finally, the importance of administering electronic PRO measures in COVID-19 has been identified due to the nature of this disease and the rapid deterioration that can occur in patients with mild symptoms [7].

We developed and sought to validate the content of a new PRO measure to be completed as a daily electronic diary: The Symptoms Evolution of COVID-19 (SE-C19). This paper summarizes the work that was conducted to (1) develop the SE-C19 and (2) evaluate its content validity in line with US Food and Drug Administration (FDA) guidance and good research practices [12]. This paper is part of a larger body of work addressing the humanistic impact of COVID-19 on outpatients, focusing on both symptoms and the impacts on their daily lives [13].

Methods

Stage 1: development of the SE-C19 questions

In early 2020, there was a limited understanding of the symptoms experienced by outpatients with COVID-19. The SE-C19 questions were developed based on the CDC symptom list [14] and published literature available at the time [15]. Item generation and evaluation of content validity of the SE-C19 were informed by the good measurement principles outlined in the FDA PRO guidance [12] and the Patient-Focused Drug Development guidance [16].

Symptoms identified in the literature and on the CDC website were used to develop 23 questions concerning the following: feeling feverish, chills, sore throat, cough, shortness of breath or difficulty breathing, nausea, vomiting, diarrhea, headache, red or watery eyes, body and muscle aches, loss of taste or smell, fatigue, loss of appetite, confusion, dizziness, pressure or tight chest, chest pain, stomachache, rash, sneezing, sputum/phlegm, and runny nose. Patients were asked to select which of the symptoms they had experienced in the past 24 h in an electronic diary, then rate the severity of each selected symptom at its worst in that 24-h period on a severity scale (mild, moderate, or severe). To aid interpretation of the SE-C19, additional questions were developed to assess patients’ global impressions of overall symptom severity and return to usual health/daily activities, in line with FDA recommendations [6]. The electronic app could be accessed using a smartphone, tablet, or computer.

Stage 2: content validation of the SE-C19

A cross-sectional, applied qualitative research study comprising in-depth, open-ended interviews in patients with a confirmed diagnosis of COVID-19 was conducted. Clinicians provided insights on the clinical experience and evolving symptomatic profile of COVID-19. Interviews were conducted in September and October 2020. This study is reported in accordance with the “Consolidated Criteria for Reporting Qualitative Research” guidelines [17].

Participants

We purposively recruited 30 adult (aged ≥ 18 years) outpatients through a healthcare research firm. All patients were recruited in the US and had a confirmed diagnosis of COVID-19, through a positive polymerase chain reaction test, up to 21 days before the interview. Patients self-reported fever (≥ 37.8 °C or feeling feverish or chills), cough, shortness of breath/difficulty breathing, change or loss of taste or smell, vomiting or diarrhea, or body/muscle aches. Patients who had been hospitalized within the past 30 days, were not fluent in English, or who were residing in nursing homes were excluded from the sample. Consistent with recommendations for sample sizes in qualitative healthcare research [18], a target of 30 patients was considered to be sufficient for achieving conceptual saturation, that is, the point at which no new information emerges from additional interviews [19, 20]. Five independent clinicians who regularly treated patients with COVID-19 were also interviewed. These clinicians treated at least five patients per week, both in person and remotely, and were recruited through the same healthcare research firm and Regeneron Pharmaceuticals, Inc.

Interview conduct

Both patient and clinician interviews (~ 60 min) were conducted virtually by three experienced qualitative researchers (NM, KP, AR) with backgrounds in health psychology and clinical research who received study-specific training in conducting the interviews using open-ended questions and the “think aloud” process to meet the study objectives as described further below. The interviewers were unknown to the patients prior to the conduct of the interview and were not involved in any aspect of the patients’ healthcare. The semi-structured patient interview guide was divided into (1) concept elicitation and (2) cognitive debriefing.

The concept elicitation comprised open-ended questions about patients’ COVID-19 symptoms (“Can you tell me about the symptoms you have experienced with COVID-19?”). Interviewers also used structured prompts for all common symptoms of COVID-19 [14] not spontaneously mentioned. The focus of this exercise was to spontaneously elicit proximal and distal concepts associated with the COVID-19 experience, prior to cognitive debriefing of the SE-C19. This was partly in aim to assess conceptual coverage of the SE-C19, see (submitted for publication) for full account of methods and results.

During cognitive debriefing, patients used a “think aloud” method to respond to each of the questions in the SE-C19. Patients were asked to read each question aloud and verbalize their thought process while responding. When necessary, interviewers asked open-ended questions to elicit additional information about the patients’ experience responding to each question (“Tell me what you are thinking about…” or “What are you considering when you respond to this question?”) [21,22,23]. Once patients were given an opportunity to spontaneously provide feedback on the SE-C19, structured probes were used to elicit patients’ understanding of the instructions, clarity, and appropriateness of the instrument. Each question was reviewed to understand its relevance, ease of selecting a response, and whether the 24-h recall period was appropriate. The objective was to ensure the intended meaning of each question of the SE-C19 was consistent with the patient’s interpretation. Patients were also asked about missing symptoms or if any questions should be removed.

Clinicians were presented with a paper copy of the SE-C19 and were asked to provide feedback on its questions (specifically any overlapping symptoms or groups of symptoms), as well as the relevance and appropriateness of the instructions, response options, and recall period.

Data analysis

Interviews were audio-recorded, transcribed, and anonymized. Transcripts were coded by NM, KP, AR in ATLAS.ti software [24] using an open, inductive coding approach [24,25,26,27]. Investigator triangulation [28] was used during analysis; the first transcript was independently coded by three researchers, and minor inconsistencies in codes were discussed and reconciled. Researchers met regularly to discuss and adjust coding guidelines, when necessary (for example, to ensure the codes were capturing the granularity reported by patients). The adequacy of the sample size was assessed by saturation analysis [19, 20] performed on quintiles of transcripts.

The conceptual coverage and comprehensiveness of the SE-C19 was evaluated by mapping its questions to the symptoms elicited from the patient experience. Cognitive debriefing analysis consisted of categorizing issues spontaneously reported by patients or found by probes. Structured, multi-level coding was used to report patient feedback on each question, corresponding response, instructions for completion, issues of clarity/interpretation, and any suggested changes.

Results

Sample characteristics

All 30 outpatient interviews were included in the analysis. Patient ages ranged from 18 to 76 years, with an average age of 45 (standard deviation ± 19.4). Patients were 60% female and 87% white. The interview was conducted an average of 19 days after symptoms began and an average of 12 days after COVID-19 diagnosis. Of the outpatients experiencing symptoms at the time of the interview (83%), 68% described the overall severity of their symptoms in the past 24 h as “mild” and 32% as “moderate”. See Table 1 for full patient demographic and health information.

Table 1 Overview of patient sample characteristics

Five clinicians were interviewed, one in Korea—where all patients with COVID-19 are hospitalized—and four in the US. The Korean clinician was informed of the focus of this study and asked to comment specifically on their experience treating mild to moderate cases. See Table 2 for full clinician backgrounds.

Table 2 Overview of clinician sample characteristics

Conceptual coverage of the SE-C19 instrument

The comprehensive list of underlying symptoms emerging from open-ended questions in the patient interviews were used for this analysis (Table 3). A comprehensive conceptualization of the patient experience is reported elsewhere [13].  Conceptual saturation of the symptoms presented in the table was achieved after 25 interviews; the final five interviews yielded no new additional symptoms. The symptomatic experiences of outpatients were covered by SE-C19 questions: upper and lower respiratory tract, systemic, gastrointestinal, smell and taste, and other (skin, ocular) symptoms. Minor conceptual gaps were described by one to two outpatients regarding nuances in the experience of already captured symptoms: nasal/congestion symptoms (head congestion, stuffy nose, post-nasal drip, sinus pressure, swollen glands, earache), systemic manifestations of sickness (sweating, not feeling like their usual self, weakness, heart palpitations), and a few other rare symptoms (constipation, skin redness, numb feet), see Table 3.

Table 3 Concept-to-item mapping

Most symptoms were explicitly mapped to one SE-C19 question, meaning the question (e.g., feverish) was directly associated with a patient-reported symptom (e.g., fever). One question in the SE-C19 (“confusion”) did not directly map onto one experience described by outpatients; however, outpatients did report feeling groggy, brain fog, not feeling as mentally sharp as they previously were, and mental fatigue. Additionally, the SE-C19 addresses both smell and taste in one question, but outpatients described alterations in these senses as different experiences.

Cognitive debriefing of the SE-C19 instrument

The instructions of the SE-C19, including the 24-h recall period, were relevant and easy for patients to understand. Very few issues in understanding were reported for the questions. The most frequent issue concerned “sputum/phlegm” (N = 11); outpatients were unfamiliar with the term “sputum”, but easily understood the term “phlegm”.

I didn’t really know what sputum meant, but I saw phlegm, so that’s why I checked it. (Male, 21)

Four outpatients felt that it was difficult to provide an accurate answer to the question “loss of smell/taste” if they were only experiencing a loss of smell or taste, but not both, or if they experienced an alteration in their senses as opposed to complete loss.

I thought they would go together, but I couldn’t smell first and I could still taste. (Female, 18)

However, other outpatients experienced these at the time same time and thought they belonged together conceptually (N = 7).

I don’t suppose there’s a difference. They both are blocked, both of them at the same time, so I feel that they’re connected. (Female, 28)

Differences between “runny nose” (included in the SE-C19) and “stuffy nose” (not included in SE-C19) were probed. Five outpatients believed these were different, but four other outpatients saw them as connected experiences.

Usually stuffy nose starts from runny nose. So, they are connected together. (Female, 65)

“Nausea” and “vomiting” were confirmed as conceptually different by all outpatients who endorsed these symptoms (N = 22). Outpatients also believed that “pressure/tightness in chest” and “chest pain” were conceptually different experiences and agreed that they should remain as separate questions (N = 18).

I personally feel like they’re different because I do have a lot of chest pressure, but, really, the only time I experienced pain in my chest was just from the constant coughing. (Female, 29)

Overall, the severity response scale (mild, moderate, and severe) was considered appropriate for capturing symptom severity at its worst. Minor instances of difficulty in choosing a response option were reported by patients. Two outpatients reported a response problem for the question “body aches such as muscle pain or joint pain” due to their aching experience not being specific to muscle or joint pain. One patient described difficulty with choosing between “moderate” and “severe” for the question “shortness of breath/difficulty breathing”; this patient indicated that their symptom was generally moderate but when specifically thinking of it within the context of the instructions (symptom severity at its worst) as they considered it to be “severe”.

Almost all outpatients (N = 28) thought the symptom list was comprehensive in reflection of their own experience, and the only symptoms listed as possible additions were “earache” and “voice loss”, each only reported by one outpatient.

Clinician interviews

The clinicians supported the importance and relevance of all symptoms that were currently combined into a singular question. However, one indicated that “body aches such as muscle pain and joint pain” could be separated into two questions if the scale were to be administered long-term, as joint pain can linger more than other body pains. Furthermore, clinicians felt that the term “body aches” did not add any specificity to the question and recommended removing it. Regarding “sputum/phlegm”, clinicians agreed that sputum and phlegm are not clinically different but stated that patients are not often familiar with the term “sputum”. Finally, “shortness of breath/difficulty breathing” was not considered clinically different in mild cases, but clinicians felt that “shortness of breath” was a more relevant general term than “difficulty breathing”.

From the clinicians’ perspectives, some questions in the SE-C19 were considered less common in outpatients with COVID-19: runny nose, sneezing, confusion, dizziness, stomachache, rash, and red/watery eyes. Clinicians reported that “confusion” was typically seen in hospitalized or older patients with severe cases of COVID-19. Minor conceptual gaps that were identified in the concept-to-item analysis based on outpatient results were not identified by clinicians as core to the outpatient experience of COVID-19.

The clinicians endorsed the response options regarding the severity level of each question (mild, moderate, or severe). Four of the five clinicians approved a 24-h recall period for completing the SE-C19; the fifth clinician did not provide specific feedback on the recall period. One clinician felt that if the diary is successfully administered daily, it will likely be able to capture change in patients’ symptomatic experiences. If administered less often, the symptom experience of COVID-19 would not be captured adequately.

Discussion

The purpose of this study was to develop and evaluate the content validity of the SE-C19, a new PRO instrument that is appropriate, easy to understand, and comprehensive for measuring symptom evolution in outpatients with COVID-19. The SE-C19 was developed in English with the involvement of English-speaking outpatients with a positive diagnosis of COVID-19. The development of the SE-C19 addresses the important unmet need for a standardized method for evaluating daily COVID-19 symptoms in outpatients using an electronic mode of data collection. As specified in the FDA guidance for assessing COVID-19 in clinical trials [6], it is important that data related to the patient experience of symptoms is collected daily to evaluate treatment benefit. To the best of our knowledge this is the first disease-specific instrument designed for daily assessment of symptoms.

The questions of the SE-C19 were developed based on the symptoms reported in clinical literature and listed on the CDC website at the time (May 2020), which provides face validity in tracking symptom onset and recovery. The current paper provides further empirical evidence to support clinical guidelines, based on data from both outpatients and clinicians. In addition to the face validity, this research was important to establish the content validity of the SE-C19 in two ways: first, to confirm the relevance and coverage of the symptomatic experience of outpatients based on a comprehensive list of concepts elicited before outpatients saw the SE-C19; and second, to confirm the questions were appropriate and understood through the cognitive debriefing.

The SE-C19 covered the symptoms that outpatients described as part of their experience. Minor conceptual gaps described by one to two outpatients revealed nuances in the experience of nasal/congestion or systemic manifestations of sickness. Once outpatients reviewed all questions in the SE-C19, they agreed that no important symptoms were missing, supporting the comprehensiveness and relevance of the SE-C19 in capturing the patient experience of COVID-19 symptoms.

The symptoms proposed in the FDA guidance for COVID-19 clinical trials released after the development of the SE-C19 [16], as well as the symptoms listed by the CDC [14], WHO [1], and NHS [3], are all measured by the SE-C19. The 24-h recall period and use of a four-point scale are also supported by the FDA guidance [6].

The cognitive debriefing of the SE-C19 confirmed that most questions were appropriate, well understood, and easy to answer. The two major findings of the cognitive debriefing were: (1) the word “sputum” in the “sputum/phlegm” question was not understood by some outpatients, but this did not impact their ability to respond, as “phlegm” was understood; (2) some outpatients reported difficulty answering the question about “loss of taste/smell” since they considered these to be different symptoms. Some outpatients also reported “altered” taste to be a conceptually different experience from “lost” taste.

Future studies could consider minor adjustments to the SE-C19 to improve the conceptual coverage of nuances in the symptomatic experience and improve the understanding of the following questions: adding one question related to congestion/stuffy nose, separating “loss of taste/smell” into two questions that further evaluate altered and lost concepts separately, rewording “confusion” to “brain fog” to more accurately reflect the outpatient experience, rewording “body aches such as muscle pain and joint pain” to “body or muscle aches”, and removing “sputum” from “sputum/phlegm”.

The current study has a few limitations. Some outpatients were no longer experiencing symptoms at the time of the interviews, as they were completed an average of 12 days after their positive diagnosis. However, outpatients had vivid memories of their recent symptoms and were able to provide detailed feedback for the SE-C19. Additionally, the study sample is not representative of the wider population with respect to race and ethnicity. Racial and ethnic disparities in the severity of COVID-19 symptoms have been identified [29, 30]. Future research is warranted to explore the ability of the SE-C19 in capturing the patient experience in a more diverse population. While our research provides unique contributions to the scientific and clinical literature, we must acknowledge, as with other similar qualitative studies, potential researcher bias. Specifically, the objectives of this study were primarily focused on the patients’ experiences of symptoms and the relevance of the SE-C19 in capturing those symptoms to inform treatment benefit in clinical research. We appreciate that there are other factors that may impact the patient experience of COVID-19, including but not limited to health beliefs, changes in identity, and social stigma. Future research could further explore broader patient experiences of this novel disease, including the potential longer-term impacts [31].

Due to the small sample size of the current study, further research should be conducted to confirm the validity and reliability of all the SE-C19 questions, using classical and modern test theory to support the scoring of the measure and define responders. This will also provide evidence for the minor adjustments suggested based on the current sample’s feedback. Upon completion of further analyses, this provides a valid, evidence-based PRO assessment of COVID-19 that can be considered for future use by patients and healthcare professionals. While the current study focused on outpatients in the acute phase of COVID-19, additional research could further explore the symptom trajectory and impact on daily lives in long-term COVID [32]. In this context, the SE-C19 could be a useful tool to better understand persistent symptoms.

Conclusions

The current study provides supportive evidence for the content validity of the SE-C19 to assess the symptoms of outpatients with COVID-19 and its use in clinical trials to evaluate treatment benefit, with suggestions for minor adjustments. The SE-C19 addresses the need for a comprehensive PRO assessing the symptoms of outpatients with COVID-19, their severity, and degrees of recovery, especially with the growing number of genetic variants of SARS-CoV-2 [33]. It can also be used to assess patient burden in those experiencing “long COVID” [32].

To conclude that the amended version of the SE-C19 is fit for purpose in outpatient clinical research, the next phase will involve quantitative evaluation of its psychometric performance, including a scoring algorithm, a strategy to derive endpoints from daily data, and the determination of within-patient meaningful change.