Do patient-reported outcome measures measure up? A qualitative study to examine perceptions and experiences with heart failure proms among diverse, low-income patients
Journal of Patient-Reported Outcomes volume 6, Article number: 6 (2022)
The Kansas City Cardiomyopathy Questionnaire (KCCQ) is a Patient-Reported Outcome Measure (PROM) used to evaluate the health status of patients with heart failure (HF) but has predominantly been tested in settings serving predominately white, male, and economically well-resourced populations. We sought to examine the acceptability of the shorter version of the KCCQ (KCCQ-12) among racially and ethnically diverse patients receiving care in an urban, safety-net setting.
We conducted cognitive interviews with a diverse population of patients with heart failure in a safety net system to assess their perceptions of the KCCQ-12. We conducted a thematic analysis of the qualitative data then mapped themes to the Capability, Opportunity, Motivation Model of Behavior framework.
We interviewed 18 patients with heart failure and found that patients broadly endorsed the concepts of the KCCQ-12 with minor suggestions to improve the instrument’s content and appearance. Although patients accepted the KCCQ-12, we found that the instrument did not adequately measure aspects of health care and quality of life that patients identified as being important components of managing their heart failure. Patient-important factors of heart failure management coalesced into three main themes: social support, health care environment, and mental health.
Patients from this diverse, low-income, majority non-white population experience unique challenges and circumstances that impact their ability to manage disease. In this study, patients were receptive to the KCCQ-12 as a tool but perceived that it did not adequately capture key health components such as mental health and social relationships that deeply impact their ability to manage HF. Further study on the incorporation of social determinants of health into PROMs could make them more useful tools in evaluating and managing HF in diverse, underserved populations.
Heart failure (HF) is a common, progressive, fatal disease with a prevalence expected to increase by 40% over the next 10–15 years . In the United States, HF incidence, morbidity, and mortality are worse for African American, Hispanic, and other minority populations as compared to Caucasians [2, 3]. Patient-reported outcome measures (PROMs) are important tools to evaluate HF treatments and are associated with clinical HF outcomes . Routine use of PROMs can significantly benefit the provider’s ability to provide personalized, tailored patient care . The Kansas City Cardiomyopathy Questionnaire (KCCQ) is a 23-item questionnaire qualified as a medical device  and drug development tool by the FDA . The KCCQ includes items about daily functioning, symptoms, and quality of life (QoL) in patients with HF. Clinical trials often employ the validated KCCQ as a clinical endpoint, due to its ability to predict hospitalization and death . The shorter version of the KCCQ—the KCCQ-12—retains the psychometric properties of the full KCCQ, but is more feasible to implement as it minimizes the response burden patients experience . The formative studies used to develop and validate the KCCQ and KCCQ-12 drew from predominantly White and male populations, similar to many clinical trials and observational studies in HF [8, 10]. We lack a complete understanding of how patients of different racial/ethnic backgrounds respond to items (questions) within the KCCQ . In addition, providers, payers, and policymakers increasingly recognize social determinants of health (SDOH) as direct contributors to HF morbidity and mortality [11, 12]. Despite this, PROMs like the KCCQ and KCCQ-12 do not include information about SDOH and its impact on HF symptoms or self-management.
The goal of this project is to address this health equity gap through the examination of how a racially/ethnically diverse, low-income sample of patients with HF perceive the KCCQ-12. This qualitative inquiry explores how this PROM resonates with diverse patients, including how lived experiences directly impact HF self-management and influence their symptom reporting.
Study setting and patients
This study enrolled patients receiving outpatient primary or cardiology care at an academic, urban, safety-net system that serves one in eight San Franciscans regardless of immigration or insurance status. The San Francisco Health Network’s (SFHN) Cardiology Clinic at Zuckerberg San Francisco General Hospital (ZSFG) delivers over 4,000 annual visits. Patients have primarily Medicaid or Medicare or are uninsured. Patients were eligible if they met all of the following criteria: (1) spoke English; (2) were between the ages 24–85; (3) received care at primary care or Cardiology clinics at the ZSFG location; (4) diagnosed with HF; and (5) did not have a significant cognitive or visual impairment. We invited eligible individuals to participate in person at ZSFG or by mail, email, or phone. All study visits beginning April 2020 were virtual given the COVID-19 pandemic. Patients provided consent and self-reported demographics. Those who completed an interview with our study team received $25 in cash or gift card for their time. The UCSF Institutional Review Board (IRB) (#18-26769) and the FDA IRB (2019-CDRH-019) approved this study.
To characterize and understand how a diverse sample of HF patients interpreted items and responses within the KCCQ, we presented the KCCQ-12 , an abbreviated version of the KCCQ questionnaire. Compared to the KCCQ, the KCCQ-12 omits the stability, symptom burden, and symptom efficacy domains plus items from the physical limitation, quality of life, and social limitations domains. The KCCQ-12 remains a valid and reliable tool to measure a patient’s HF-related health status and how it impacts QoL, physical, and social activities . The KCCQ-12 was thus used to be better integrated into clinic workflow. All 18 patients received the KCCQ-12 on paper, and of those patients, six (33%) also received the KCCQ-12 tablet-adapted version. For exploratory purposes, six (33%) also received an adapted version with a lower reading level on paper (see Table 4 for Flesh Kincaid (FK) score). The KCCQ-12 and reading level-adapted KCCQ-12 were printed on 8.5 × 11-inch standard white paper and used 10-pt Arial black font. The study team asked patients for feedback on both content and layout of the KCCQ-12 (e.g., how the KCCQ-12 was presented, readability and font size).
Trained interviewers (KO, MS, SL) conducted cognitive interviews using a semi-structured interview guide (Additional file 1). Cognitive interviewing is a qualitative research method that is used to understand whether questionnaires and survey questions work as intended . There are two approaches that can be used to conduct a cognitive interview—think-aloud and verbal probing. Think-aloud is a technique in which participants are instructed to say anything that comes to mind as they go through the survey. Verbal probing is another technique in which the interviewer follows up with another question to elicit a more detailed response from the participant [13, 14]. Our team used both the think-aloud and verbal probing methods concurrently . The interviews took between 60 and 90 min. Study staff audio-recorded interviews with participant permission. The interviews were professionally transcribed for analysis.
According to best practices in qualitative research, we conducted data analysis in parallel with data collection, using the method of constant comparisons. We reached thematic saturation—the point at which no new themes emerged from the interviews—after analyzing 10 interviews and completed data analysis for the remaining two interviews. For the reading level-adapted KCCQ-12, we reached thematic saturation after four of the six interviews.
We utilized Dedoose qualitative analysis software (Los Angeles, CA) to code transcripts . Using an inductive and deductive approach, we coded qualitative interviews to reduce code bias. Two study coders (KO, MS) collaboratively analyzed transcripts to lower variability between interviewers and to support consistent data cleaning.
For the deductive coding, we applied the behavioral Capability, Opportunity, Motivation model of Behavior (COM-B) model  to explore the open-ended participant comments about the behaviors of heart failure symptom management/self-management. Michie et al. describes the capability to entail both the patient’s psychological and physical capacity, plus having the necessary knowledge and skills to engage. Opportunity involves those external factors that impact behavior. Finally, motivation includes those factors that prompt and guide behavior, including conscious and unconscious processes. We developed a list of thematic codes based on the interview guide domains and topics then transcripts were re-reviewed along with notes and memos to determine broader thematic codes. Through this process, we developed sub-themes to capture participant experiences.
In total, we conducted 18 qualitative interviews with heart failure patients from April 2019 to September 2020. The mean age was 52 years, 33% were women, and 78% were non-White. Complete baseline demographics are in Table 1.
Study patients shared a range of experiences managing their heart failure condition as well as a variety of feedback on the KCCQ-12 [original (n = 12) and tablet (n = 6) and reading level-adapted (n = 6) versions]. Table 2 summarizes feedback on the questionnaire including suggested changes.
Applying the COM-B framework  to the interviews, we identified several key issues for the self-management of heart failure (Table 3). In the capability domain, both physical issues related to medication adherence and psychological issues related to both general HF disease management and their experience with health care providers affected self-management. Patients were concerned about their ability to practice HF self-management behaviors such as taking medications, exercising, and maintaining a balanced diet. They reported unclear knowledge of key behaviors and actions related to HF self-management, which correspond with their ability to understand health information. In the motivation domain, themes related to beliefs about their own identity and role as a patient with HF, emotions, and attitudes toward heart disease, and their goals for HF management emerged. Items in the opportunity domain intersected with the motivation domain, namely their social support and physical environment such as living environment. Both strongly impacted their ability and desire to manage their HF. Environment—home, neighborhood, work, and even clinic—repeatedly emerged as a major SDOH that touched every aspect of their HF management.
We calculated the FK reading level of the KCCQ-12 and it was higher than the recommended 8th-grade reading level for patient-reported outcome measures. Therefore, we adapted it to recommended reading levels and explored patient perceptions of the literacy-appropriate version (Table 4). While some patients found them to be “the same” (ptid29, ptid395), others preferred the simplified version. None of the interviewees expressed a preference for the original KCCQ-12 version on paper.
This prospective qualitative thematic analysis of patients’ perceptions of the KCCQ-12 in an integrated, urban safety-net health system demonstrated that while items on the KCCQ-12 were relevant to their symptoms, they did not adequately capture all the factors that affect daily life with HF, such as diet, sexual activity, or key SDOH such as mental health and social relationships (Additional file 2). However, for patients to successfully manage their heart disease, all three components of the COM-B system (capability, opportunity, motivation) are important. Importantly, HF patients participating in these interviews did not draw categorical distinctions between symptoms, such as the extent of their shortness of breath, and their self-management, such as their dietary restrictions. Opportunity domains, such as one’s built environment and social factors like the quality of patient-provider communication, may influence a patient’s opportunity to engage in self-management behaviors. Mental health, resiliency, and social support are patient-important factors that were not captured on the KCCQ-12 but may influence a patient’s motivation to manage their heart disease. This is particularly relevant in this diverse, publicly insured population who face increased medical and social complexity in everyday life . Inclusion of a broader range of factors into HF PROMs may increase their relevance for diverse populations.
In our exploratory analysis, patients preferred an adapted version of the KCCQ-12, which reflected a lower literacy level, consistent with prior studies comparing higher vs. lower literacy patient-facing materials . Modifying the KCCQ-12 literacy level for diverse and low-income populations may improve its performance in predicting HF outcomes. Before the literacy-adapted version of the KCCQ-12 can be used, further testing in diverse populations with HF will need to be performed.
The KCCQ-12, a novel and well-validated health-related quality of life assessment tool for patients with HF, is comprised of distinct domains: physical limitations, symptoms (frequency, severity, and change over time), self-efficacy and knowledge, social interference, and quality of life [8, 18]. The subjective perception of dyspnea varies greatly and impacts the quality of life , and the KCCQ-12 captures this. The KCCQ-12 does not, however, capture patients’ SDOH, which has also been shown to impact risk . It has been shown that correlating physical and psychological symptom profiles are predictive of survival . The multiple factors that impact these, from mental health to housing to food insecurity, are acknowledged but seldom incorporated into patient assessments and shared-decision making . The current study further supports the imperative to prioritize and directly assess the broader SDOHs that impact patient experience, disease management, and ultimately survival.
Social determinants of health deeply impact patient experience and outcomes and are disproportionately present in patients with HF as compared to the general population. For example, depression is more common in patients with HF and portends a worse prognosis [22, 23]. Similar trends are seen with other SDOHs such as socioeconomic position , food insecurity , and unemployment . Non-White patients tend to experience these SDOHs more commonly than white patients, are underrepresented in clinical trials, and have poorer outcomes [2, 3, 10, 27]. This study of almost exclusively non-White patients brings to the surface these multiple factors that impact the burden of living with heart failure, from polypharmacy to dietary recommendations to social support to sex. Assessing and understanding how an individual patient navigates HF management with the competing demands of life, health, and work is paramount to providing thoughtful, personalized, and appropriate care.
Using the COM-B framework , we identified several key issues for the self-management of heart failure congruent with prior research [28, 29]. The KCCQ-12 most heavily captures capability, particularly physical issues, and patients’ feedback in this domain primarily focused on its psychological components. Most notably, patients in this study articulated the underrepresentation of the opportunity and motivation domains. They do not separate their HF symptoms from either external factors or conscious and unconscious processes that prompt and guide behavior. Patient perspectives on the overlap between the opportunity and motivation domains highlight the critical influence that SDOH has on HF self-management. Attributes in the opportunity and motivation domains weigh heavily on those facing medical and social complexity and have not been fully explored to date. This finding highlights a challenge in the use of PROMs across settings. Well-validated tools such as the PHQ-9 unfortunately do not fully capture those additional social determinants of health beyond mental health that impact their heart failure experience. Thus, inclusion of such an assessment in conjunction with the KCCQ-12 still leaves gaps in SDOH assessment. For medical product performance, PROMs need only include the target symptoms for the device or drug. However, for clinical care, patients experience their illness in the context of their social vulnerabilities. PROMs developed for regulatory purposes may be insufficient to inform clinical decision-making without consideration of SDOH.
Several limitations exist in the current study. We conducted interviews in a single safety-net health system hospital limiting external generalizability. However, this study touches on key demographic groups not typically represented in clinical studies. Only English-speaking patients were included, which unfortunately excluded non-English speaking patients in the SFHN, most commonly Spanish and Cantonese. This is an area of ongoing study. We employed the KCCQ-12 rather than the original 23-item KCCQ, which does have an item on depression and an item on sexual function. Based on our patients’ feedback, though, inclusion of these items would not have sufficiently addressed those SDOH deficiencies described above. Finally, the COVID-19 pandemic necessitated that the final 6 months of the study be conducted remotely via telephone or video conferencing. Given the socioeconomic makeup of our patient population, this likely impacted who was able to participate. Shelter-in-place requirements increased patient isolation and impacted patients’ feedback of the KCCQ-12. COVID-19 has exaggerated health disparities [30,31,32], and the impact of PROM and SDOH can be further explored.
Patients from a diverse, low-income, majority non-white population in a safety net health care system experience unique challenges and circumstances that impact their ability to manage disease. It is important that future studies continue to enroll such patients, as SDOH disproportionately affects them negatively and impacts public health outcomes. In this study, patients were receptive to the KCCQ-12 as a tool but perceived that it did not adequately capture their illness experience because it did not assess key SDOH such as mental health and social relationships that deeply impact their ability to manage HF. Further study on the intersection of PROMs and SDOH reporting could investigate how to best use standardized patient reporting in evaluating and managing HF in diverse, underserved populations.
Availability of data and materials
All data generated or analyzed during this study are included in this published article.
Patient-reported outcome measures
Kansas City Cardiomyopathy Questionnaire
Social determinants of health
The San Francisco Health Network
Zuckerberg San Francisco General Hospital
Capability, Opportunity, Motivation Model of Behavior
Virani SS, Alonso A, Benjamin EJ et al (2020) Heart disease and stroke statistics-2020 update: a report from the American Heart Association. Circulation 141:e139–e596. https://doi.org/10.1161/CIR.0000000000000757
Van Nuys KE, Xie Z, Tysinger B et al (2018) Innovation in heart failure treatment: life expectancy, disability, and health disparities. JACC Heart Fail 6:401–409. https://doi.org/10.1016/j.jchf.2017.12.006
Glynn P, Lloyd-Jones DM, Feinstein MJ et al (2019) Disparities in cardiovascular mortality related to heart failure in the United States. J Am Coll Cardiol 73:2354–2355. https://doi.org/10.1016/j.jacc.2019.02.042
Yee D, Novak E, Platts A et al (2019) Comparison of the Kansas City cardiomyopathy questionnaire and Minnesota living with heart failure questionnaire in predicting heart failure outcomes. Am J Cardiol 123:807. https://doi.org/10.1016/j.amjcard.2018.11.037
Spertus J (2014) Barriers to the use of patient-reported outcomes in clinical care. Circ Cardiovasc Qual Outcomes 7:2–4. https://doi.org/10.1161/CIRCOUTCOMES.113.000829
Commissioner O of the (2017) Statement from FDA Commissioner Scott Gottlieb, M.D., on new steps to advance medical device innovation and help patients gain faster access to beneficial technologies. In: FDA. https://www.fda.gov/news-events/press-announcements/statement-fda-commissioner-scott-gottlieb-md-new-steps-advance-medical-device-innovation-and-help. Accessed 19 May 2021
US Food and Drug Administration (2020) Clinical outcome assessments (COA) Qualification Submissions Office Of Cardiology H, endocrinology, and nephrology (OCEHM) Division of Cardiovascular and Nephrology (DCN). DDT COA# 000084: Kansas City Cardiomyopathy Questionnaire (KCCQ). https://www.fda.gov/drugs/ddt-coa-000084-kansas-city-cardiomyopathy-questionnaire-kccq
Green CP, Porter CB, Bresnahan DR, Spertus JA (2000) Development and evaluation of the Kansas City Cardiomyopathy Questionnaire: a new health status measure for heart failure. J Am Coll Cardiol 35:1245–1255. https://doi.org/10.1016/s0735-1097(00)00531-3
Spertus JA, Jones PG (2015) Development and validation of a short version of the Kansas City cardiomyopathy questionnaire. Circ Cardiovasc Qual Outcomes 8:469–476. https://doi.org/10.1161/CIRCOUTCOMES.115.001958
Occa A, Morgan SE, Potter JE (2018) Underrepresentation of Hispanics and other minorities in clinical trials: recruiters’ perspectives. J Racial Ethn Health Disparities 5:322–332. https://doi.org/10.1007/s40615-017-0373-x
White-Williams C, Rossi LP, Bittner VA et al (2020) Addressing social determinants of health in the care of patients with heart failure: a scientific statement from the American Heart Association. Circulation 141:e841–e863. https://doi.org/10.1161/CIR.0000000000000767
Tawakol A, Osborne MT, Wang Y et al (2019) Stress-associated neurobiological pathway linking socioeconomic disparities to cardiovascular disease. J Am Coll Cardiol 73:3243–3255. https://doi.org/10.1016/j.jacc.2019.04.042
Boeije H, Willis G (2013) The cognitive interviewing reporting framework (CIRF). Methodology 9:87–95. https://doi.org/10.1027/1614-2241/a000075
Willis GB, Artino AR (2013) What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. J Grad Med Educ 5:353–356. https://doi.org/10.4300/JGME-D-13-00154.1
Dedoose (2020) Dedoose Version 8.3.45 web application for managing, analyzing, and presenting qualitative and mixed method research data (2020). SocioCultural Research Consultants, LLC (www.dedoose.com), Los Angeles, CA
Michie S, van Stralen MM, West R (2011) The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci 6:42. https://doi.org/10.1186/1748-5908-6-42
Sudore RL, Seth Landefeld C, Barnes DE et al (2007) An advance directive redesigned to meet the literacy level of most adults: a randomized trial. Patient Educ Couns 69:165–195. https://doi.org/10.1016/j.pec.2007.08.015
Krumholz HM, Chaudhry SI, Spertus JA et al (2016) Do non-clinical factors improve prediction of readmission risk? Results from the tele-hf study. JACC Heart Fail 4:12–20. https://doi.org/10.1016/j.jchf.2015.07.017
Faulkner KM, Jurgens CY, Denfeld QE et al (2020) Identifying unique profiles of perceived dyspnea burden in heart failure. Heart Lung J Crit Care 49:488–494. https://doi.org/10.1016/j.hrtlng.2020.03.026
Retrum JH, Boggs J, Hersh A et al (2013) Patient-identified factors related to heart failure readmissions. Circ Cardiovasc Qual Outcomes 6:171–177. https://doi.org/10.1161/CIRCOUTCOMES.112.967356
Lee CS, Gelow JM, Denfeld QE et al (2014) Physical and psychological symptom profiling and event-free survival in adults with moderate to advanced heart failure. J Cardiovasc Nurs 29:315–323. https://doi.org/10.1097/JCN.0b013e318285968a
Celano CM, Villegas AC, Albanese AM et al (2018) Depression and anxiety in heart failure: a review. Harv Rev Psychiatry 26:175–184. https://doi.org/10.1097/HRP.0000000000000162
Jiang W, Alexander J, Christopher E et al (2001) Relationship of depression to increased risk of mortality and rehospitalization in patients with congestive heart failure. Arch Intern Med 161:1849–1856. https://doi.org/10.1001/archinte.161.15.1849
Witte KK, Patel PA, Walker AMN et al (2018) Socioeconomic deprivation and mode-specific outcomes in patients with chronic heart failure. Heart 104:993–998. https://doi.org/10.1136/heartjnl-2017-312539
Sun Y, Liu B, Rong S et al (2020) Food insecurity is associated with cardiovascular and all-cause mortality among adults in the United States. J Am Heart Assoc 9:e014629. https://doi.org/10.1161/JAHA.119.014629
Dupre ME, Nelson A, Lynch SM et al (2017) Socioeconomic, psychosocial and behavioral characteristics of patients hospitalized with cardiovascular disease. Am J Med Sci 354:565–572. https://doi.org/10.1016/j.amjms.2017.07.011
Patel H, Williams KA (2018) Narrowing the disparities in heart failure: treat the event or try to prevent? JACC Heart Fail 6:410–412. https://doi.org/10.1016/j.jchf.2018.01.019
Zou H, Chen Y, Fang W et al (2017) Identification of factors associated with self-care behaviors using the COM-B model in patients with chronic heart failure. Eur J Cardiovasc Nurs J Work Group Cardiovasc Nurs Eur Soc Cardiol 16:530–538. https://doi.org/10.1177/1474515117695722
Whittal A, Störk S, Riegel B, Herber OR (2020) Applying the COM-B behaviour model to overcome barriers to heart failure self-care: a practical application of a conceptual framework for the development of complex interventions (ACHIEVE study). Eur J Cardiovasc Nurs J Work Group Cardiovasc Nurs Eur Soc Cardiol. https://doi.org/10.1177/1474515120957292
Garg S (2020) Hospitalization rates and characteristics of patients hospitalized with laboratory-confirmed coronavirus disease 2019—covid-net, 14 states, March 1–30, 2020. MMWR Morb Mortal Wkly Rep. https://doi.org/10.15585/mmwr.mm6915e3
Chowkwanyun M, Reed AL (2020) Racial health disparities and covid-19—caution and context. N Engl J Med 383:201–203. https://doi.org/10.1056/NEJMp2012910
Yancy CW (2020) COVID-19 and African Americans. JAMA 323:1891–1892. https://doi.org/10.1001/jama.2020.6548
We would like to thank our FDA collaborators for their partnership, support, and guidance. We also acknowledge Kristina Ryan for assistance with screening and recruiting patients, and Mekhala Hoskote, Roy Cherian, Natalie Rivadeneira, and Kevin Chang for their contribution in conducting patient interviews.
This publication is supported by the Food and Drug Administration (FDA) of the U.S. Department of Health and Human Services (HHS) as part of a financial assistance award [Center of Excellence in Regulatory Science and Innovation grant to the University of California, San Francisco (UCSF) and Stanford University, U01FD004979/U01FD005978 totaling $154,114 with 100 percentage funded by FDA/HHS]. The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement, by FDA/HHS, or the U.S. Government.
Ethics approval and consent to participate
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. The Institutional Review Board of the University of California, San Francisco (#18-26769) approved this study.
Consent for publication
US holds grants from the National Institute of Health’s National Cancer Institute, the California Healthcare Foundation, the Center for Care Innovation, the US Food and Drug Administration, the National Library of Medicine, and the Commonwealth Fund. The other authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Davis, J., Olazo, K., Sierra, M. et al. Do patient-reported outcome measures measure up? A qualitative study to examine perceptions and experiences with heart failure proms among diverse, low-income patients. J Patient Rep Outcomes 6, 6 (2022). https://doi.org/10.1186/s41687-022-00410-9
- Heart failure
- Qualitative research
- Outcomes research