Skip to main content
Journal of Patient-Reported Outcomes
Download PDF
Download ePub

Psychometric evaluation of the Symptoms of Infection with Coronavirus-19 (SIC): results from a cross-sectional study and a phase 3 clinical trial

Journal of Patient-Reported Outcomes volume 7, Article number: 45 (2023) Cite this article

Abstract

Background

The Symptoms of Infection with Coronavirus-19 (SIC) is a 30-item patient-reported outcome (PRO) measure scored by body system composites to assess signs/symptoms of coronavirus disease 2019 (COVID-19). In addition to cross-sectional and longitudinal psychometric evaluations, qualitative exit interviews were conducted to support the content validity of the SIC.

Methods

In a cross-sectional study, adults diagnosed with COVID-19 in the United States completed the web-based SIC and additional PRO measures. A subset was invited to participate in phone-based exit interviews. Longitudinal psychometric properties were assessed in ENSEMBLE2, a multinational, randomized, double-blind, placebo-controlled, phase 3 trial of the Ad26.COV2.S COVID-19 vaccine. Psychometric properties evaluated included structure, scoring, reliability, construct validity, discriminating ability, responsiveness, and meaningful change thresholds of SIC items and composite scores.

Results

In the cross-sectional study, 152 participants completed the SIC (mean age, 51.0 ± 18.6 years) and 20 completed follow-up interviews. Fatigue (77.6%), feeling unwell (65.8%), and cough (60.5%) were symptoms most frequently reported. SIC inter-item correlations were all positive and mostly moderate (r ≥ 0.3) and statistically significant. SIC items and Patient-Reported Outcomes Measurement Information System-29 (PROMIS-29) scores correlated as hypothesized (all r ≥ 0.32). Internal consistency reliabilities of all SIC composite scores were satisfactory (Cronbach’s alpha, 0.69–0.91). SIC composite scores correlated moderately (r = 0.30–0.49) to strongly (r ≥ 0.50) with PROMIS-29 scores and Patient Global Impression of Severity (PGIS) ratings (all P < 0.01). A variety of signs/symptoms were cited in exit interviews, and participants considered the SIC straightforward, comprehensive, and easy to use. From ENSEMBLE2, 183 participants with laboratory-confirmed moderate to severe/critical COVID-19 were included (51.5 ± 14.8 years). Strong test-retest reliabilities were observed for most SIC composite scores (intraclass correlations ≥ 0.60). Statistically significant differences across PGIS severity levels were found for all but 1 composite score, supporting known-groups validity. All SIC composite scores demonstrated responsiveness based on changes in PGIS.

Conclusions

The psychometric evaluations provided strong evidence for the reliability and validity of the SIC for measuring COVID-19 symptoms, supporting its use in vaccine and treatment trials. In exit interviews, participants described a broad range of signs/symptoms consistent with previous research, further supporting the content validity and format of the SIC.

Plain english summary

Coronavirus disease 2019 (COVID-19) is a serious disease that continues to evolve globally. Researchers developed the Symptoms of Infection with Coronavirus-19 (SIC), a 30-item questionnaire designed for patients to report signs and symptoms of COVID-19. In this study, the researchers formally analyzed how well the SIC measures the patient experience with COVID-19, using survey and clinical trial data as well as telephone interviews. Adults with COVID-19 and at least 2 bothersome symptoms completed the web-based survey, and some of these individuals also participated in in-depth interviews. Participants in a clinical trial for a COVID-19 vaccine also completed the SIC measure. The SIC was compared with other commonly used questionnaires that evaluate patient experience. The most commonly reported symptoms of COVID-19 were fatigue, feeling unwell, cough, weakness, and headache. The items for individual symptoms (e.g., “cough”) and combined scores for body systems (e.g., “respiratory system”) performed well in statistical analyses. Participants found the SIC to be straightforward, comprehensive, and easy to use. The SIC may prove useful in the future for vaccine and treatment trials for COVID-19.

Background

According to the Centers for Disease Control and Prevention (CDC), as of September 2022, there were > 95 million total cases of coronavirus disease 2019 (COVID-19) in the United States and 1 million deaths [1]. As of September 2022, 68% of the eligible US population was fully vaccinated, including 92% of individuals aged ≥ 65 years and 72% of the population aged ≥ 5 years [1]. Booster doses lag behind, with only 49% of eligible people having received 1 booster dose. Worldwide, COVID-19 has resulted in 610 million cases and caused > 6.5 million deaths [2]. The emergence of new and increasingly transmissible variants [3,4,5] highlights the continued need for both vaccination and characterization of COVID-19 signs and symptoms as part of the global effort against the pandemic.

Some of the symptoms associated with COVID-19 include fever or chills, dry cough, shortness of breath, fatigue, muscle or body aches, headache, loss of taste or smell, sore throat, congestion or runny nose, red eyes, skin rash, discoloration of fingers or toes, nausea or vomiting, and diarrhea [6, 7]. Some of these symptoms can resemble those of other respiratory diseases, particularly influenza [6, 8]. Certain distinct symptoms, such as sudden loss of smell and taste, may indicate COVID-19 infection [9, 10]. Some populations are at increased risk of severe COVID-19, with key risk factors including age ≥ 65 years; male sex; and comorbidities such as heart disease, diabetes, immunocompromised states, or chronic lung, liver, or kidney disease.

The patient experience with COVID-19 varies across individuals and differs from other viral infections in terms of types of symptoms and their severity, with patients experiencing a range of respiratory, neurologic, gastrointestinal, circulatory, and musculoskeletal symptoms [6, 9, 11]. In addition to the variety of acute symptoms, many patients experience long-term symptoms [12]. Prospective studies of signs and symptoms of COVID-19 are needed to characterize disease presentation and progression, especially in light of new variants, and to evaluate diagnostic tests, treatments, and vaccines [9]. Patient-reported outcome (PRO) instruments provide a standardized method to assess the disease experience and impact from the patient perspective, and can be used to monitor disease progression and response to treatment [13, 14]. In the context of COVID-19, PRO measures may facilitate diagnosis, contact tracing and prevention strategies, long-term follow-up, and may inform clinical trial design [13]. COVID-19–specific PRO measures are needed to help accurately measure COVID-19 signs and symptoms and capture changes in disease severity.

At the time of this study, no COVID-19–specific PRO measures were published, and few have been developed to date [13]. PRO instruments developed for the diagnosis of COVID-19 include the Italian EPICOVID19 screen for 11 patient-reported symptoms [15] and the UK COVID-19 Symptom Study app based on 1 to 3 days of self-reported symptoms (among a total of 19 possible symptoms) [16]. The US CDC Coronavirus Self-Checker is an online tool that offers recommendations based on the user’s responses [17]. The US Occupational Safety and Health Administration employee COVID-19 health screening questionnaire includes 11 self-reported symptoms [18]. Other health surveys have been developed by workplaces, universities, laboratories, federal agencies, and other entities [19]. Many of the prominent symptoms reported in the SIC, such as fatigue and cough, have been reported in other PRO studies [11]. Newer instruments assess health-related quality of life in current/former COVID-19 patients [20] and/or post–acute sequelae of SARS-CoV-2 infection (“long COVID”) [21]. The SIC is unique in that it is a PRO measure designed to capture presence and severity of acute COVID-19 symptoms, rather than serving as an epidemiological or surveillance tool.

The purpose of this study was to perform a psychometric evaluation of the Symptoms of Infection with Coronavirus-19 (SIC), a PRO measure designed to assess the presence and severity of signs and symptoms of COVID-19. To be appropriate for use in both vaccine and treatment clinical studies, we sought to create a measure that was sensitive to both disease onset and changes in key signs and symptoms of COVID-19 using best practices in alignment with current regulatory guidance [22,23,24,25,26].

Methods

Description of the SIC

The SIC includes 30 sign/symptom items grouped into composite scores by body system, including respiratory (9 items), constitutional (7 items), gastrointestinal (5 items), neurologic (5 items), musculoskeletal (3 items), and vascular (1 item) (Fig. 1) [27]. Participants rate the signs and symptoms as present or absent (yes or no) during the past 24 h. Of the 30 signs and symptoms, 25 are rated on a severity scale of 0 (none) through 10 (worst possible), while 5 additional items are rated only as present or absent (Table S1). The SIC composite scores are constructed as the average of the symptom severity ratings for each set of items. The development and content validation of the SIC were based on a targeted review of the literature, interviews with patients with COVID-19, experts in infectious diseases, and clinicians who treat patients with COVID-19 (Figure S1). In the initial development of the SIC, 3 iterative rounds of combined concept elicitation and cognitive debriefing interviews were conducted with a total of 31 patients, caregivers, and healthy volunteers. Using data collected from a cross-sectional observational study, we examined the measurement properties of each SIC item, as well as composite scores based on these items. In the cross-sectional study, qualitative exit interviews were conducted in a subgroup of participants to strengthen the content validity of the SIC. Psychometric properties of the SIC, including test-retest reliability, known-groups validity, responsiveness, and meaningful change thresholds of composite scores, were also assessed using a subset of data from a phase 3 trial of the Ad26.COV2.S COVID-19 vaccine (ENSEMBLE2; ClinicalTrials.gov Identifier: NCT04614948) [28].

Fig. 1
figure 1

SIC conceptual framework [27]. SIC, Symptoms of Infection with Coronavirus-19; COVID-19, coronavirus disease 2019

Full size image

Cross-sectional study design

In the cross-sectional, observational study, SIC data were collected via a web-based survey from US adults with COVID-19 based on a positive test and current symptoms. Inclusion criteria were the following: adults (aged ≥ 18 years) with a self-reported positive COVID-19 test within 2 weeks of screening; ≥ 2 current, bothersome symptoms of COVID-19; and willingness to provide medical and demographic information and to complete the survey. Recruitment targets for the survey were set to ensure a diverse sample for age (up to 50% < 65 years old), hospitalization status (≥ 35 patients hospitalized because of COVID-19), preexisting comorbidities (causing higher risk for severe disease), race/ethnicity (up to 50% non-White), and disease severity (mild, moderate, and severe). Recruitment targets for the exit interviews also aimed for a diverse population with ≥ 50% minority status, with highest priority given to race/ethnicity other than White, education level less than high school/General Educational Development and variable access to health care. Participants received 100 USD in appreciation of their time for completing the web-based survey and an additional 100 USD for completion of the exit interview.

Participants in the cross-sectional study completed the SIC, a study-specific Overall Symptom Checklist for the 30 SIC items (to identify symptoms that had been experienced but resolved; yes or no for each item), and an open-ended response option for any symptoms they experienced that were not captured in the checklist. Participants were asked 2 additional questions about what symptoms had to improve or resolve before they felt they could return to their usual activities and 4 questions about their access to and receipt of medical care related to COVID-19 (Table S2).

Psychometric performance of the SIC was evaluated using other PRO measures, including the Patient Global Impression of Severity (PGIS) [29, 30], Patient Global Impression of Change (PGIC) [29], Patient-Reported Outcomes Measurement Information System-29 (PROMIS-29) [31], and the Overall Symptom Checklist. The PGIS is a single-item, patient-reported, 4-point rating scale that assesses the severity of COVID-19 symptoms over a 24-h period as follows: 0 = none, 1 = mild, 2 = moderate, and 3 = severe. The PGIC is a single-item, patient-reported, 7-point rating scale that evaluates change in severity of COVID-19 symptoms since onset as follows: 0 = much better, 1 = moderately better, 2 = a little better, 3 = no change, 4 = a little worse, 5 = moderately worse, and 6 = much worse. The PROMIS-29 is a PRO measure that assesses 7 domains: depression, anxiety, physical function, pain interference, fatigue, sleep disturbance, and ability to participate in social roles and activities. There are 29 items in total, with all 7 domains including 4 items each (rated from 1 to 5), plus an item assessing pain intensity (rated from 0 to 10). Higher scores indicate more of the concept being measured (i.e., more depression or more physical function).

The EQ-5D-5L was used on an exploratory basis to rate current health status. The EQ-5D-5L evaluates 5 dimensions: mobility, daily activities, self-care, pain/discomfort, and anxiety/depression [32]. Dimensions are rated as: 1 = no problems, 2 = slight problems, 3 = moderate problems, 4 = severe problems, and 5 = extreme problems. Global health is also measured on a visual analog scale from 0 (worst imaginable health) to 100 (best imaginable health).

After completing the web-based survey, participants were asked to participate in an in-depth exit interview using a semistructured guide to provide further details regarding their COVID-19 experience. Analysis of the exit interview data used a thematic qualitative approach with information from field notes and transcripts to identify dominant trends and generate themes or patterns in respondents’ symptom reports. Common themes were identified via constant comparative analysis, which also allowed confirmation of content validity and patient comprehension of the SIC.

Phase 3 trial study design

ENSEMBLE2 was a multinational, randomized, double-blind, placebo-controlled, phase 3 trial assessing the safety and efficacy of the Ad26.COV2.S vaccine for the prevention of COVID-19 in adults [28]. Participants from Europe, the United States, Latin America, South Africa, and the Philippines were randomized 1:1 to receive a primary dose of Ad26.COV2.S or placebo, plus a homologous booster dose or placebo dose at a 2-month interval. The study consisted of a 60-week study period and a follow-up period of 1 year. Eligible participants were adults (aged ≥ 18 years), who were healthy or with stable and well-controlled comorbidities, without prior receipt of a COVID-19 vaccine. Additional inclusion criteria for the current study of the SIC used in ENSEMBLE2 were: polymerase chain reaction (PCR)–confirmed COVID-19 from a central laboratory (University of Washington); COVID-19 occurring from Days 15 to 56 (prior to booster vaccination); moderate to severe/critical COVID-19, assessed by a blinded Clinical Severity Adjudication Committee; and SIC data collected within 7 days of PCR confirmation. ENSEMBLE2 recruitment targeted a diverse representation of race/ethnicity and sex.

In the ENSEMBLE2 trial, 31,300 participants were enrolled and randomized to receive Ad26.COV2.S (n = 15,708) or placebo (n = 15,592); a subset of participants was included in the present analysis. All participants in the ENSEMBLE2 trial were provided an electronic clinical outcome assessment tool, completed the SIC prior to first vaccine administration, and were monitored for the development of COVID-19 symptoms throughout the study. Participants reporting symptoms that suggested possible COVID-19 and those reporting any non-study positive-PCR tests were asked to complete 2 PRO measures daily throughout each suspected COVID-19 episode: the SIC was used to evaluate COVID-19 signs and symptoms, and the PGIS was used for validation. Additionally, participants reported blood oxygen saturation and pulse rate 3 times per day and body temperature daily during a COVID-19 episode. Nasal swabs were collected for detection of SARS-CoV-2 infection by reverse transcription (RT)–PCR within 1 to 2 days of symptom onset, on Days 3 to 5, and then every 2 days. Participants with ≥ 1 positive SARS-CoV-2 nasal swab returned to the study site 28 days (± 7 days) after symptom onset for clinical assessment and seroconfirmation of SARS-CoV-2 infection. Resolution of a COVID-19 episode was established by 2 consecutive, negative, SARS-CoV-2 nasal swabs and 2 consecutive days without COVID-19–related signs or symptoms.

Psychometric evaluation

Psychometric analyses were conducted using data from participants who completed the SIC in the cross-sectional study and a subset of participants from ENSEMBLE2. The psychometric properties evaluated in each study for all SIC items and candidate composite scores are summarized in Table 1. All statistical tests were 2-tailed (α = 0.05), and P ≤ 0.05 was considered statistically significant.

Table 1 Psychometric Properties Assessed in the Cross-sectional Study and ENSEMBLE2 Phase 3 Trial
Full size table

Descriptive statistics for the SIC were computed in the cross-sectional study for COVID-19 signs and symptoms, including for subgroups based on age (< 65 or ≥ 65 years) and comorbidities (0 vs. ≥ 1); response frequency distributions were calculated for each SIC item. Inter-item correlations for the SIC items were computed and examined for expected patterns of relationships.

Construct validity of the SIC was assessed in the cross-sectional study by correlational analyses, with the strength and sign of the predicted correlation depending on the expected pattern of relationship for measures addressing similar constructs. SIC fatigue scores were hypothesized to correlate at least moderately with PROMIS-29 fatigue scores, SIC musculoskeletal items to correlate at least moderately with PROMIS-29 physical function scores, and SIC pain items to correlate at least moderately with PROMIS-29 pain scores. Correlation coefficients were interpreted as follows: r ≥ 0.50, large; 0.30 to 0.49, moderate; 0.10 to 0.29, small; and < 0.10, trivial.

Known-groups validity (distinguishing among hypothetically different groups) was evaluated in the cross-sectional study and in ENSEMBLE2 via analysis of variance (ANOVA) to compare mean differences in SIC item scores between subgroups: participants reporting no symptoms or mild compared to those reporting moderate or severe on the PGIS; participants with scores in the top one-third of the PROMIS-29 physical function score distribution versus participants in the bottom one-third; and respondents with PROMIS-29 fatigue scores in the top one-third of the distribution versus those in the bottom one-third.

A set of candidate SIC composite scores representing different body systems was analyzed in both studies. Candidate scoring algorithms were evaluated with respect to internal consistency reliability and composite-level analyses that paralleled the item-level validity analyses. Although not ideal for the evaluation of symptom measures such as the SIC, Cronbach’s coefficient alpha was calculated to explore the internal consistency of the candidate SIC composite scores.

In the ENSEMBLE2 trial, test-retest reliability (reproduction of identical measurements when no change has occurred between testing) was assessed by calculating intraclass correlation coefficients in the subgroup of participants identified as stable by PGIS response (“no symptom,” “mild,” or same response) at Day 1 and Day 2 of their COVID-19 episode [33]. The intraclass correlation was estimated using a mixed effects model, calculated as estimated within-subject variance over estimated total variance. The responsiveness of the SIC was evaluated in ENSEMBLE2 using the PGIS as an anchor variable via change in SIC composite scores from Days 1 to 2 (ANOVA) to assess the ability to detect changes in symptoms over time. Meaningful change thresholds (i.e., the smallest change that patients perceive as beneficial) of the SIC were estimated in ENSEMBLE2 using ANOVAs and mean change scores (1- or 2-point improvement in PGIS).

Ethics

All participants provided informed consent. The cross-sectional study was reviewed on ethical grounds by the RTI International institutional review board. The ENSEMBLE2 trial adhered to the International Council for Harmonisation guidelines on Good Clinical Practice and the principles of the Declaration of Helsinki.

Results

Demographics

In the cross-sectional study, 152 participants completed the PRO measures from November 17, 2020, to January 4, 2021 (mean ± standard deviation [SD] age, 51.0 ± 18.6 years [range, 18–90 years]; female, 62.5%; Table 2). Respondents were racially and ethnically diverse, and approximately half reported ≥ 1 comorbid condition. Most participants reported their worst symptoms as moderate (47.4%) versus mild (26.3%) or severe (26.3%); patients had experienced symptoms for a mean duration of 14.9 ± 7.3 days. Among those who had been hospitalized (27.6%), the mean stay was 3.2 ± 2.4 days.

Table 2 Participant Characteristics at Screening (Cross-sectional Study)
Full size table

Twenty of the 152 participants who completed the SIC also completed follow-up interviews from January 7 to February 17, 2021 (mean ± SD age, 51.5 ± 16.8 years [range, 22–73 years]; female, 65%). Interview participants were racially diverse, with 45.0% African American, 25.0% White, 5.0% American Indian, 10.0% mixed race, and 15.0% other race, and had a range of educational levels, employment statuses, and types of health insurance coverage. Approximately half of the exit interview participants (45.0%; 9/20) had a preexisting risk factor increasing the likelihood of severe COVID-19 outcomes. Preexisting conditions included asthma (45.0%), obesity (35.0%), hypertension (35.0%), heart disease (25.0%), chronic obstructive pulmonary disease (10.0%), and type 1 diabetes (5.0%); some participants reported > 1 condition. COVID-19 was reported as severe in 45.0%, moderate in 35.0%, and mild in 20.0% of interview participants, and 45.0% (9/20) had been hospitalized for COVID-19; of those, 4 had received oxygen and 2 were admitted to the intensive care unit.

From ENSEMBLE2 (November 16, 2020, to June 25, 2021), 183 participants with confirmed moderate to severe/critical COVID-19 occurring from Days 15 to 56 post–primary vaccination, but before planned booster vaccination, were included in this analysis (mean ± SD age, 51.5 ± 14.8 years [range, 18–85]; female, 43.7%). Participants were 65.0% White, 18.0% Asian, 13.7% American Indian or Alaska Native, and 2.7% Black. Participants came from Europe (32.2%), the United States (27.9%), Latin America (23.0%), the Philippines (13.1%), and South Africa (3.8%). Of these 183 participants, 130 completed the SIC.

Psychometric analyses

Item-level descriptive statistics

The most frequently endorsed SIC symptoms in the cross-sectional study were fatigue (77.6%), feeling unwell (65.8%), cough (60.5%), physical weakness (59.9%), and headache (59.9%) (Table 3). Mean overall severity scores on the scale of 0 to 10 ranged from 2.95 for cough to 4.68 for fatigue. For participants aged < 65 years (n = 88), fatigue had the highest severity rating (mean, 4.59) followed by headache (3.78), feeling unwell (3.43), and physical weakness (3.36). For participants aged ≥ 65 years (n = 64), fatigue (4.80) had the highest severity rating followed by feeling unwell (4.23), loss of appetite (3.61), and physical weakness (3.44). In comorbidity subgroups, grouped as 0 (n = 79) versus ≥ 1 (n = 73), fatigue had the worst severity scores in both groups (4.70 for no comorbidities vs. 4.66 for those with comorbidities) followed by feeling unwell (3.82 vs. 3.71); participants without comorbidities generally had lower mean scores on the SIC items. In ENSEMBLE2, the most frequently endorsed SIC symptoms were similar to those reported in the cross-sectional study and included feeling unwell, cough, headache, fatigue, and sore throat (Table S3). Most respondents reported their overall symptoms as mild (n = 66, 43.4%) or moderate (n = 54, 35.5%) on the PGIS and rated their change in symptoms since symptom onset as much better (n = 56, 36.8%) or moderately better (n = 40, 26.3%) on the PGIC (Table S4).

Table 3 Frequently Endorsed SIC Item-level Responses and Severity Score (Cross-sectional Study)
Full size table

Item-level analyses

Results from the cross-sectional study showed that all SIC inter-item correlations were positive in sign and correlations within each composite score were generally statistically significant and at least moderate in size (r ≥ 0.30).

In the cross-sectional study, construct validity correlations between SIC items and PROMIS-29 scores followed the hypothesized patterns. As expected, the SIC fatigue score correlated positively with PROMIS-29 fatigue score (r = 0.64) and negatively with the physical function score (r = − 0.65); SIC physical weakness, muscle aches/pains, and joint aches/pains scores correlated moderately to strongly with PROMIS-29 physical function scores (r = − 0.67, r = − 0.52, and r = − 0.42, respectively); and the 5 SIC pain items correlated at least moderately with PROMIS-29 pain scores (all r ≥ 0.32).

Respondents with mild or no symptoms on the PGIS had better item-level SIC scores compared with the moderate or severe PGIS subgroups. Most differences between PGIS subgroups were statistically significant, with the exception of the chest congestion, eye irritation, abdominal pain, and red/bruised feet items. Respondents with better PROMIS-29 physical function scores and better PROMIS-29 fatigue scores also had better SIC scores, with statistically significant differences observed except for the vomiting item (with PROMIS-29 physical function) and skin rash, uncontrollable shaking/shivering, and red/bruised feet (with PROMIS-29 fatigue). As expected, the largest mean difference between PROMIS-29 physical function subgroups was for the SIC physical weakness item and the largest difference between PROMIS-29 fatigue subgroups was for the SIC fatigue and physical weakness items. These item-level known-group analyses provided strong support for the discriminating ability of the SIC items.

Composite-level analyses

SIC composite scores were formed by body system and included constitutional (7 items), gastrointestinal (5 items), musculoskeletal (3 items), neurologic (3 items), sensory (2 items), and respiratory (9 items) composites. A single vascular item was scored on its own (0 = no and 1 = yes). SIC scores were also computed separately for upper and lower respiratory symptoms.

Internal consistency reliabilities indicated that most of the SIC composite scores comprised items that were strongly related, with Cronbach’s alpha > 0.70 for all but 1 SIC composite score in either the cross-sectional study (gastrointestinal) and ENSEMBLE2 (neurologic) (Table 4).

Table 4 SIC Internal Consistency Reliabilities (Cross-sectional Study, N = 152; ENSEMBLE2 Trial, N = 130a)
Full size table

In the cross-sectional study, SIC composite scores correlated moderately (r = 0.30–0.49) to strongly (r ≥ 0.50) with PROMIS-29 scores, and all were statistically significant (construct validity; Table 5). Further, most EQ-5D-5L scores correlated moderately to strongly with SIC composite scores; EQ-5D-5L scores for pain/discomfort, self-care, and usual activities correlated most strongly with SIC constitutional and musculoskeletal scores (r = 0.58–0.65; Table 6). SIC composite scores correlated moderately to strongly and positively with the PGIS, with correlations ranging from 0.41 for the upper respiratory score to 0.73 for the constitutional score (Table 6).

Table 5 SIC Construct Validity Correlations With PROMIS-29 Scores (Cross-sectional Study, N = 152)
Full size table
Table 6 SIC Correlations With the PGIS and EQ-5D-5L (Cross-sectional Study, N = 152)
Full size table

In the cross-sectional study, the known-groups analyses showed that all composite-level differences were in the expected direction and all were statistically significant. The no symptoms/mild PGIS subgroup achieved better SIC scores than the moderate/severe PGIS subgroup. Respondents in the top one-third of the distribution of PROMIS-29 physical function scores achieved better SIC scores than those in the bottom one-third, and respondents in the top one-third of the distribution of PROMIS-29 fatigue scores had better SIC scores than the bottom one-third. In ENSEMBLE2, statistically significant differences (P < 0.05) were observed for the composite scores for the different PGIS severity levels except for the sensory composite (Fig. 2), supporting known-groups validity.

Fig. 2
figure 2

SIC composite score known-group validity by Day 1 PGIS response (ENSEMBLE2 trial; N = 130a). SIC, Symptoms of Infection with Coronavirus-19; PGIS, Patient Global Impression of Severity; PCR, polymerase chain reaction; COVID-19, coronavirus disease 2019. aOf 183 participants with PCR-confirmed moderate to severe/critical COVID-19 occurring from Days 15 to 56 post–primary vaccination, 130 completed the SIC.

*P < 0.05

Full size image

In ENSEMBLE2, test-retest reliabilities were strong for most composite scores and were moderate for neurologic and constitutional (Table S5). The responsiveness of the SIC was apparent in all SIC composite scores, and improvement and worsening of PGIS ratings and SIC composite scores moved in the same direction, supporting the ability of the SIC to detect COVID-19 sign and symptom changes over time (Fig. 3). The estimated meaningful change thresholds for the SIC composite scores are based on 1- or 2-point improvements on the PGIS and ranged from − 0.21 for gastrointestinal at Day 3 to − 2.11 for musculoskeletal at Day 5 (Table S6).

Fig. 3
figure 3

SIC composite scorea change by Day 2 PGIS status (ENSEMBLE2 trial; N = 130b). SIC, Symptoms of Infection with Coronavirus-19; PGIS, Patient Global Impression of Severity; PCR, polymerase chain reaction; COVID-19, coronavirus disease 2019. aChange in composite score is the composite score at Day 2 minus the composite score at Day 1. A score < 0 indicates improvement, a score of 0 indicates no change, and a score > 0 indicates worsening. bOf 183 participants with PCR-confirmed moderate to severe/critical COVID-19 occurring from Days 15 to 56 post–primary vaccination, 130 completed the SIC.

Full size image

Exit interviews (cross-sectional study)

In exit interviews, participants reported a broad range of signs and symptoms of COVID-19; they did not describe a single, consistent constellation of symptoms. The top 7 symptoms reported among the 20 patients were fatigue (80.0%), cough and shortness of breath (both 65.0%), decreased sense of smell and taste (both 60.0%), headache (55.0%), and nasal congestion (50.0%). Loss of smell or taste prompted some participants to believe they had COVID-19 instead of a cold or flu. The progression and ultimate resolution of symptoms varied among patients and by the symptom in question. The most bothersome symptoms (identified as the greatest barriers preventing a return to usual activities) were shortness of breath (25.0%), fatigue (20.0%), and headache (15.0%). Most participants (85.0%) reported being negatively affected by needing to self-isolate from family and other people, and that they experienced anxiety regarding what would happen to them and family members. Participants seemed hesitant to label their COVID-19 experience as severe disease if they were not hospitalized or receiving supplemental oxygen, even though almost half (45.0%) reported symptoms consistent with severe disease. Table S7 presents representative quotations from the exit interviews.

Participants provided insight into meaningful change and recovery by describing which COVID-19 symptoms required complete resolution before resuming normal activities versus those that only required some improvement. Fatigue, shortness of breath, and vomiting were the top 3 symptoms participants described as needing to be resolved before they could resume activities. Other symptoms frequently cited as needing to be completely resolved were physical weakness, fever, chest pain, uncontrollable shivering, feeling generally unwell, diarrhea, and headache. Cough and congestion would need improvement, but not necessarily resolution, before resuming activities. Estimates of the degree of symptom improvement that could enable participants to return to everyday activities varied by symptom.

Respondents also answered 4 questions about health care access for their COVID-19 symptoms. Most respondents considered it “very easy” to access care (49.0%) and to get their most recent COVID-19 test (46.0%). More than half cited no barriers to receiving care (57.0%), although most did not seek medical care beyond COVID-19 testing (60.0%).

Participants’ evaluation of the SIC (cross-sectional study)

When asked to review the format and content of the SIC, all participants perceived the SIC to be straightforward, comprehensive, and easy to use, even when they were moderately ill. Several participants indicated that self-report on the SIC would be challenging during severe illness or intubation, although measures could be (or were) taken with assistance from a caregiver or using fingers to indicate symptom severity. Overall, participants felt the SIC was an accurate measure to report and track COVID-19 symptom onset, emergence of new signs/symptoms, and changes in symptoms over time.

Discussion

This study reports the psychometric properties of the SIC, a PRO measure with potential application in future vaccine and treatment trials for COVID-19. The results support the reliability and validity of the SIC items and composite scores as appropriate and useful measures of COVID-19 symptom severity. Results of the qualitative exit interviews in a diverse and potentially vulnerable population described a broad range of signs and symptoms of COVID-19 and were consistent with previous qualitative research, further supporting the content validity and format of the SIC. Exit interview participants further endorsed the SIC, noting that it was an accurate and easy method to describe COVID-19 symptom onset and changes over time. In the ENSEMBLE2 clinical trial, longitudinal psychometric data collected in a global population supported the test-retest reliability, known-groups validity, responsiveness, and meaningful change thresholds of the SIC composite scores.

Effective survey instruments for health care research should be easy and quick to administer, inexpensive, accessible, and convenient [34]. Exit interviews from the current study and an earlier qualitative study support the ease of use of the SIC by patients and caregivers and across varying disease severity [27]. The SIC was developed to align with current US Food and Drug Administration and patient-focused drug development guidance regarding the use of PRO measures in clinical trials and characterization of the patient experience of COVID-19. This guidance stipulates that any PRO measure used to collect patient experiences must be developed with extensive input from patients in the population of interest and must undergo thorough psychometric evaluation in the target population. The results of the psychometric evaluations of the current study, along with the patient interviews, support its application in vaccine and treatment trials. Moreover, the SIC may have potential application in assessing the impact of variant-driven or vaccination-mediated changes in COVID-19 disease severity or symptoms.

Our goal in developing the SIC was to include, as much as possible and to the best of our knowledge, all symptoms experienced by COVID-19 patients based on literature reviews, patient input, and expert opinion. Although the cross-sectional study was limited to US patients, analyses were also conducted on a subset of data from ENSEMBLE2, a multinational, phase 3, COVID-19 vaccine trial. This dataset supports the validity of the SIC to assess signs and symptoms of COVID-19 in a variety of regions and settings. Because a vaccine trial is not optimal to establish meaningful change thresholds, further studies will be needed to more definitively establish these thresholds. Furthermore, symptoms experienced by COVID-19 patients are heterogeneous, with variability among patients in both type and severity of symptoms. SARS-CoV-2 is also changing rapidly; for example, with the continuous emergence of new variants [3,4,5] (most recently Omicron XBB.1.5, as of February 2023), additional qualitative studies will be needed to assess how new variants affect the symptomatic experience of COVID-19. It is possible the patient experience of COVID-19 as captured in this analysis could evolve as the COVID-19 variant landscape continues to change. Although ENSEMBLE2 data present change over time, long-term symptoms and effects of COVID-19 also remain to be investigated. However, the additional evidence obtained from the exit interviews may support a potential role for the SIC in diagnosis and evaluation of post–acute sequelae of COVID-19.

Conclusion

The results of the psychometric analyses and exit interviews from the cross-sectional study along with the longitudinal psychometric analyses from ENSEMBLE2 support use of the SIC in both treatment and vaccine clinical trials, pending future work to evaluate the responsiveness of the SIC.

Data Availability

The data sharing policy of Janssen Pharmaceutical Companies of Johnson &

Johnson is available at https://www.janssen.com/clinical-trials/transparency. These data were made available by RTI Health Solutions for the current study and are not publicly available. Other researchers should contact https://www.rtihs.org/.

Abbreviations

ANOVA:

Analysis of variance

CDC:

Centers for Disease Control and Prevention

COVID-19:

Coronavirus disease 2019

GED:

General Education Development

PCR:

Polymerase chain reaction

PGIC:

Patient Global Impression of Change

PGIS:

Patient Global Impression of Severity

PRO:

Patient-reported outcome

PROMIS-29:

Patient-Reported Outcomes Measurement Information System-29

RT:

Reverse transcription

SD:

Standard deviation

SIC:

Symptoms of Infection with Coronavirus-19

VAS:

Visual analog scale

References

  1. Centers for Disease Control and Prevention (CDC) (2021) COVID data tracker. COVID-19 vaccinations in the United States. https://covid.cdc.gov/covid-data-tracker/#vaccinations_vacc-total-admin-rate-total Accessed September 16, 2022

  2. World Health Organization (WHO) (2021) WHO coronavirus (COVID-19) dashboard 2021. https://covid19.who.int/. Accessed September 16, 2022

  3. World Health Organization (WHO) (2022) Tracking SARS-CoV-2 variants. https://www.who.int/en/activities/tracking-SARS-CoV-2-variants/. Accessed February 12, 2023

  4. Centers for Disease Control and Prevention (CDC) (2022) What you need to know about variants. https://www.cdc.gov/coronavirus/2019-ncov/variants/about-variants.html. Accessed August 3, 2022

  5. Centers for Disease Control and Prevention (CDC) (2022) SARS-CoV-2 variant classifications and definitions. https://www.cdc.gov/coronavirus/2019-ncov/variants/variant-info.html. Accessed August 4, 2022

  6. Centers for Disease Control and Prevention (CDC) (2021) Symptoms of COVID-19. https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html. Accessed September 24, 2021

  7. World Health Organization (WHO) (2021) Coronavirus disease (COVID-19) advice for the public. https://www.who.int/emergencies/diseases/novel-coronavirus-2019/advice-for-public. Accessed September 24, 2021

  8. Soltan AAS, Kouchaki S, Zhu T, Kiyasseh D, Taylor T, Hussain ZB et al (2021) Rapid triage for COVID-19 using routine clinical data for patients attending hospital: development and prospective validation of an artificial intelligence screening test. Lancet Digit Health 3(2):e78–e87. https://doi.org/10.1016/S2589-7500(20)30274-0

    Article  CAS  PubMed  Google Scholar 

  9. Struyf T, Deeks JJ, Dinnes J, Takwoingi Y, Davenport C, Leeflang MM et al (2020) Signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID-19 disease. Cochrane Database Syst Rev 7:CD013665. https://doi.org/10.1002/14651858.CD013665

    Article  PubMed  Google Scholar 

  10. Marshall M (2021) COVID’s toll on smell and taste: what scientists do and don’t know. Nature 589(7842):342–343. https://doi.org/10.1038/d41586-021-00055-6

    Article  CAS  PubMed  Google Scholar 

  11. Nasserie T, Hittle M, Goodman SN (2021) Assessment of the frequency and variety of persistent symptoms among patients with COVID-19: a systematic review. JAMA Netw Open 4(5):e2111417. https://doi.org/10.1001/jamanetworkopen.2021.11417

    Article  PubMed  PubMed Central  Google Scholar 

  12. Seessle J, Waterboer T, Hippchen T, Simon J, Kirchner M, Lim A et al (2022) Persistent symptoms in adult patients 1 year after coronavirus disease 2019 (COVID-19): a prospective cohort study. Clin Infect Dis 74(7):1191–1198. https://doi.org/10.1093/cid/ciab611

    Article  CAS  PubMed  Google Scholar 

  13. Aiyegbusi OL, Calvert MJ (2020) Patient-reported outcomes: central to the management of COVID-19. Lancet 396(10250):531. https://doi.org/10.1016/S0140-6736(20)31724-4

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Mercieca-Bebber R, King MT, Calvert MJ, Stockler MR, Friedlander M (2018) The importance of patient-reported outcomes in clinical trials and strategies for future optimization. Patient Relat Outcome Meas 9:353–367. https://doi.org/10.2147/PROM.S156279

    Article  PubMed  PubMed Central  Google Scholar 

  15. Bastiani L, Fortunato L, Pieroni S, Bianchi F, Adorni F, Prinelli F et al (2021) Rapid COVID-19 screening based on self-reported symptoms: psychometric assessment and validation of the EPICOVID19 short diagnostic scale. J Med Internet Res 23(1):e23897. https://doi.org/10.2196/23897

    Article  PubMed  PubMed Central  Google Scholar 

  16. Canas LS, Sudre CH, Capdevila Pujol J, Polidori L, Murray B, Molteni E et al (2021) Early detection of COVID-19 in the UK using self-reported symptoms: a large-scale, prospective, epidemiological surveillance study. Lancet Digit Health 3(9):e587–e598. https://doi.org/10.1016/S2589-7500(21)00131-X

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Centers for Disease Control and Prevention (CDC) (2021) Coronavirus self-checker. https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/coronavirus-self-checker.html. Accessed September 22, 2021

  18. Occupational Safety and Health Administration (OSHA) (2021) Sample employee COVID-19 health screening questionnaire. https://www.osha.gov/sites/default/files/publications/OSHA4132.pdf Accessed September 22, 2021.

  19. Verma SS, Chung WK, Dudek S, Williamson JL, Verma A, Robinson S et al (2020) Research on COVID-19 through patient-reported data: a survey for observational studies in the COVID-19 pandemic. J Clin Transl Sci 5(1):E17. https://doi.org/10.1017/cts.2020.509

  20. Amdal CD, Taylor K, Kulis D, Falk RS, Bottomley A, Arraras JI et al (2022) Health-related quality of life in patients with COVID-19; international development of a patient-reported outcome measure. J Patient Rep Outcomes 6(1):26. https://doi.org/10.1186/s41687-022-00434-1

    Article  PubMed  PubMed Central  Google Scholar 

  21. Sivan M, Preston N, Parkin A, Makower S, Gee J, Ross D et al (2022) The modified COVID-19 Yorkshire Rehabilitation Scale (C19-YRSm) patient-reported outcome measure for long covid or Post-COVID-19 syndrome. J Med Virol 94(9):4253–4264. https://doi.org/10.1002/jmv.27878

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Food and Drug Administration (FDA) (2020) Assessing COVID-19-related symptoms in outpatient adult and adolescent subjects in clinical trials of drugs and biological products for COVID-19 prevention or treatment guidance for industry. https://www.fda.gov/media/142143/download. Accessed December 11, 2020

  23. Food and Drug Administration (FDA) (2009) Patient-reported outcome measures: use in medical product development to support labeling claims: guidance for industry. https://www.fda.gov/media/77832/download. Accessed December 11, 2020

  24. Food and Drug Administration (FDA) (2017) Plan for issuance of patient-focused drug development guidance. https://www.fda.gov/media/105979/download. Accessed December 11, 2020

  25. Food and Drug Administration (FDA) (2020) FDA patient-focused drug development guidance series for enhancing the incorporation of the patient’s voice in medical product development and regulatory decision making. https://www.fda.gov/drugs/development-approval-process-drugs/fda-patient-focused-drug-development-guidance-series-enhancing-incorporation-patients-voice-medical. Accessed December 11, 2020

  26. European Medicines Agency (EMA) (2005) Reflection paper on the regulatory guidance for the use of health-related quality of life measures in the evaluation of medicinal products. https://www.ema.europa.eu/en/documents/scientific-guideline/reflection-paper-regulatory-guidance-use-healthrelated-quality-life-hrql-measures-evaluation_en.pdf. Accessed December 11, 2020

  27. Romano C, Fehnel S, Stoddard J, Sadoff J, Lewis S, McNulty P et al (2022) Development of a novel patient reported outcome measure to assess signs and symptoms of COVID 19. J Patient Rep Outcomes 6(1):85. https://doi.org/10.1186/s41687-022-00471-w

    Article  PubMed  PubMed Central  Google Scholar 

  28. Hardt K, Vandebosch A, Sadoff J, Le Gars M, Truyers C, Lowson D et al (2022) Efficacy, safety, and immunogenicity of a booster regimen of Ad26.COV2.S vaccine against COVID-19 (ENSEMBLE2): results of a randomised, double-blind, placebo-controlled, phase 3 trial. Lancet Infect Dis 22(12):1703–1715. https://doi.org/10.1016/S1473-3099(22)00506-0

    Article  PubMed  PubMed Central  Google Scholar 

  29. Food and Drug Administration (2018) Patient-focused drug development guidance: methods to identify what is important to patients & select, develop or modify fit-for-purpose clinical outcomes assessments. https://www.fda.gov/media/116281/download. Accessed January 19, 2023

  30. Food and Drug Administration (2020) Guidance for industry. Assessing COVID-19-related symptoms on outpatient adult and adolescent subjects in clinical trials of drugs and biological products for COVID-19 prevention or treatment. https://www.fda.gov/media/142143/download. Accessed January 19, 2023

  31. HealthMeasures (2020) PROMIS adult profile scoring manual. https://www.healthmeasures.net/images/PROMIS/manuals/PROMIS_Adult_Profile_Scoring_Manual.pdf. Accessed January 19, 2023

  32. EuroQol Group (1990) EuroQol–a new facility for the measurement of health-related quality of life. Health Policy 16(3):199–208. https://doi.org/10.1016/0168-8510(90)90421-9

    Article  Google Scholar 

  33. Hays RD, Anderson R, Revicki D (1993) Psychometric considerations in evaluating health-related quality of life measures. Qual Life Res 2(6):441–449. https://doi.org/10.1007/BF00422218

    Article  CAS  PubMed  Google Scholar 

  34. Safdar N, Abbo LM, Knobloch MJ, Seo SK (2016) Research methods in healthcare epidemiology: survey and qualitative research. Infect Control Hosp Epidemiol 37(11):1272–1277. https://doi.org/10.1017/ice.2016.171

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Medical writing support was provided by Laura Ninger and Sadie van Dyne, PhD, Kurt Kunz, MD, MPH, and Catherine DeBrosse, PhD, of Lumanity Communications Inc., and was funded by Janssen Global Services, LLC.

Funding

This work was supported by Janssen Vaccines & Prevention. Medical writing support was funded by Janssen Vaccines & Prevention. The corresponding author had full access to all the data in the study and final responsibility for the decision to submit for publication.

Author information

Authors and Affiliations

  1. Janssen Global Services, LLC, Raritan, NJ, USA

    Eric K.H. Chan, Jeffrey Stoddard, Yan Liu, Eva G. Katz & Pauline McNulty

  2. RTI Health Solutions, Research Triangle Park, NC, USA

    Valerie Williams, Carla Romano, Sheri Fehnel, Margaret Mayorga, Sandy Lewis, Stuart Yarr & Jia Ma

  3. Aspen Consulting, LLC, Steamboat Springs, CO, USA

    Ashley F. Slagle

  4. Janssen Research & Development, LLC, Raritan, NJ, USA

    Jerald Sadoff

  5. Janssen Research & Development, Beerse, Belgium

    Ilse van Dromme

  6. Janssen Global Services, LLC, Horsham, PA, USA

    Kelly McQuarrie

Authors
  1. Eric K.H. Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Valerie Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carla Romano
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sheri Fehnel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ashley F. Slagle
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jeffrey Stoddard
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jerald Sadoff
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Margaret Mayorga
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Sandy Lewis
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Stuart Yarr
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jia Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Yan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Eva G. Katz
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Pauline McNulty
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Ilse van Dromme
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Kelly McQuarrie
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

EKHC: Conceptualization, methodology, interpretation of results, and writing (original draft and review).

VW: Methodology, data analysis, interpretation of results, and writing (original draft and review).

CR: Conceptualization, methodology, investigation, interpretation of results, and writing (original draft and review).

SF: Methodology, investigation, data analysis, interpretation of results, and writing (original draft and review).

AFS: Conceptualization, interpretation of results, and writing (original draft and review).

JSt: Conceptualization, supervision, and writing (original draft and review).

JSa: Conceptualization, supervision, and writing (original draft and review).

MM: Investigation, data analysis, and writing (original draft and review).

SL: Methodology, resources, and writing (original draft and review).

SY: Data analysis, interpretation of results, and writing (review).

JM: Data analysis, interpretation of results, and writing (review).

YL: Data analysis and writing (original draft and review).

EGK: Conceptualization, methodology, and writing (original draft and review).

PM: Conceptualization and writing (original draft and review).

IvD: Data analysis and writing (original draft and review).

KM: Conceptualization, study design, visualization, project administration, and writing (original draft and review).

All authors read and approved the final manuscript.

Corresponding author

Correspondence to Eric K.H. Chan.

Ethics declarations

Ethics approval and consent to participate

This study was reviewed on ethical grounds by the RTI International institutional review board. For ENSEMBLE2, the protocol and amendments were approved by ethics committees and institutional review boards per local regulations. All participants provided written informed consent.

Consent for publication

Not applicable.

Competing interests

VW, CR, SF, MM, SL, and SY are employees of RTI Health Solutions, which received research funding from Janssen during the conduct of this study. JM was an employee of RTI Health Solutions at the time of the study. EKHC, EGK, and PM are employees of Janssen Global Services, LLC, and hold stock in Johnson & Johnson. JSt, YL, and KM were employees of Janssen at the time of the study. JSa is an employee of Janssen Research & Development, LLC, and holds stock in Johnson & Johnson. IvD is an employee of Johnson & Johnson. AFS is a consultant to Janssen Global Services, LLC.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Appendix: Supplementary Tables and Figure

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chan, E.K., Williams, V., Romano, C. et al. Psychometric evaluation of the Symptoms of Infection with Coronavirus-19 (SIC): results from a cross-sectional study and a phase 3 clinical trial. J Patient Rep Outcomes 7, 45 (2023). https://doi.org/10.1186/s41687-023-00581-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s41687-023-00581-z

Keywords