Skip to main content
Journal of Patient-Reported Outcomes
Download PDF
Download ePub

Prioritizing Patient Reported Outcome Measures (PROMs) to use in the clinical care of youth living with mental health concerns: a nominal group technique study

Journal of Patient-Reported Outcomes volume 8, Article number: 20 (2024) Cite this article

Abstract

Background

In the past few decades, particularly in the mental health setting, there has been growing interest in using Patient Reported Outcome Measures (PROMs) to assess the efficacy of the treatments in healthcare systems. Despite recent initiatives for global harmonization, there remains a lack of consensus on which PROMs are best practice and appropriate. Engagement of the service users, such as patients and family members/caregivers, is vital at this stage to ensure the selected PROMs are feasible, relevant, and acceptable to them. This study aimed to prioritize PROMs by youth and family/caregiver based on feasibility, relevance, and overall importance to be used in the clinical care of youth living with anxiety and/or depression.

Methods

Ten validated and widely used PROMs were presented to the patients and family/caregivers. Nominal group techniques were employed to prioritize the PROMs based on feasibility, relevance, and overall importance.

Results

For patients and families/caregivers, the PROMs, Revised Child Anxiety and Depression Scale (RCAD 25), and The Young Person’s Core (YP-CORE) were the highest priorities. Both felt that RCAD 25 was comprehensive, short, easy, and quick to complete, whereas regarding YP-CORE, patients and family/caregivers thought it was also short and relevant. Due to some specific concerns, the Strength and Difficulties Questionnaire and Child Health Questionnaire were the lowest prioritized by patients and family/caregivers.

Conclusion

It is of utmost importance that patient’s and family/caregivers’ voices or opinions are considered while selecting and implementing PROMs in mental health settings. Our study provides practical recommendations around measures best suited to achieve this.

Background

In Canada, 1.2 million children and youth are impacted by mental illness, with about 80% of mental health problems starting before age 26. However, less than 20% will receive the necessary care [1, 2]. Mental illness can have long-term implications, including increased unemployment rates, criminal activity, high school dropout, and unstable income [3]. Based on the Canadian Health Survey in 2019, 7.4% of youth aged 12–17 encountered anxiety disorders, yet 3.8% stated mood disorders [4]. In Canada, 75% of those with mental illness never seek specialized treatment services, implying a lack of adequate treatment and a gap in youth mental health services [5]. The urgent need for adequate mental health support and services for youth has been further exacerbated by the COVID-19 pandemic [6]. With the information assessed, the economic hardship of mental health illnesses in Canada conservatively costs $51 billion yearly [7]. Even though there has been an increase in the availability of mental health services in recent eras, the service delivery systems are failing to lessen the prevalence of anxiety and depression in youth [8].

In the past few decades, particularly in the mental health setting, there has been growing interest in using Patient Reported Outcome Measures (PROMs) to assess the efficacy of the treatments in healthcare systems [9]. PROMs are standardized, validated self-reported questionnaires completed by patients to determine whether the impact of the healthcare interventions and practices enhances the patient’s health and quality of life [10]. Routine use of PROMs in clinical care may facilitate better identification of the patient’s unmet needs, enhance patient/family communication, quality management, evaluation of treatment outcomes, and improve patient outcomes [11].

Despite these benefits, there are barriers to integrating PROMs into clinical practice. These include logistical, cost, technological, workflow, and privacy considerations [12, 13]. In addition, McNeill et al. showed that most of the PROMs that have been developed and integrated into practice have been done so without direct input from the service users, resulting in questions about their relevance and appropriateness, particularly from the perspective of patients [14]. It is possible that the PROMs that researchers or clinicians choose may not measure the outcomes that are vital to the target group being studied [15].

A recent review reported inconsistencies in using PROMs in clinical care could potentially be deviations in the intended effects and introducing measures related to weakness [16, 17]. The variations and gaps in the data limit the ability to identify the best practices, guide quality improvement initiatives, and ability to compare various clinical care models [18]. Despite recent initiatives for global harmonization, there remains a lack of consensus as to which PROMs are best practice and appropriate [19]. Engagement of the service users, such as patients and family members/caregivers, is vital at this stage to ensure the selected PROMs are feasible, relevant, and acceptable to them.

This study aimed to prioritize PROMs by youth and family/caregiver based on feasibility, relevance, and overall importance to be used in the clinical care of youth living with anxiety and/or depression. Findings from this study can be widely useful for helping to streamline measurement-based care approaches in youth clinical mental health service delivery and will inform the Measurement-Based Care (MBC) program at a new community mental health center, “The Summit Center,” in Calgary, Canada [20].

Methods

See Fig. 1.

Fig 1.
figure 1

Prioritizing PROMs for youth living with anxiety/depression by patients and family/caregivers to be implemented at the Summit Centre [20]

Full size image

Study design

Modified Nominal Group Technique (NGT) [21,22,23] was used to prioritize PROMS. The NGT was conducted with patients living with anxiety and depression and their families/caregivers. Delbecq and Van de Ven introduced NGT as a process to identify and develop the appropriate solutions to solve strategic problems [24]. It is a stepwise consensus-building process resulting in a list of collective priorities. This approach permits empowering individuals who might otherwise be excluded from decision-making and provides equal opportunity for each group member to participate in the discussion [21]. For instance, NGT has been successfully used to include people with learning disabilities and also with people living with mental illness [22, 25].

Additional benefits of using the NGT include: (1) it is time efficient and cost-effective; (2) priorities can be elicited in a short period; (3) it requires limited budget and preparation; (4) it inspires equal input from various viewpoints [23]. During the NGT, qualitative and quantitative data were collected to present our findings more thoroughly.

Setting

The Summit Centre is a mental health center in Calgary, Alberta that opened in March 2023. It is dedicated to assisting young people aged between 0–18 years old who are struggling with mental health problems. The services offered are (1) child and youth Mental Health walk-in services, (2) a day program for helping youth transitioning from hospital to home, (3) Intensive Community treatment services aimed to avoid the demand for hospitalization, offer short-term, intensive therapy, including individual, group and family therapy to the youth with escalating mental health concerns [20].

Participant recruitment and selection

We recruited youth diagnosed with anxiety and/or depression and family/caregivers living in the Calgary area between January and April 2023. Interested participants were identified via the Mental Health Research for Kids (MHR 4 Kids) Research registry [20]. The MHR 4 Kids Research registry is maintained by the Summit Center to connect children, adolescents, and families who are interested in participating in mental health research with researchers to decrease recruitment barriers.

We targeted approximately five to twelve participants, in line with the standards of the NGT practice [26], where sample sizes range from 2 to 14 [23]. Inclusion criteria included youth 12–17 years of age who have been through anxiety and/or depression and family members/caregivers of patients who have been receiving mental health services. The single exclusion criterion was the inability to provide full informed consent. Separate nominal groups were conducted for family/caregivers and patients, as the youth patient participants may be hesitant to share their perspectives with adult participants.

Participants received a package that included an information sheet describing the aim and objective of the study; a consent form; a socio-demographic questionnaire that was used to describe participant characteristics; a pre-identified top 10 PROMs (see Sect. “Pre-identification of PROMS”) with ranking criteria (e.g., feasibility, relevancy, and overall importance); and a description of the NGT process. This package was sent a week prior to the NGT session.

Participants signed a consent form and completed the socio-demographic information form prior to participating in the NGT session. We conducted two separate virtual NGT sessions with patients and family members/caregivers, lasting approximately two hours each. Both were co-facilitated by two research members (KT and FNS) and one youth patient-partner (JM). Ethics approval was acquired from the University Health Research Ethics Board [REB22-0880] at the University of Calgary.

Pre-identification of PROMS

The NGT is a phase of a more extensive multi-phased mixed-method study that includes the following steps that served to identify the PROMS to be discussed within the NGT.

Systematic literature review to identify PROMs

We conducted a systematic literature search to identify the PROMs that have been used in youth living with mental health concerns. MEDLINE, PubMed, and PsycINFO databases were searched for publication before 2000. Searches were limited from 2000 because the integration of PROMs in routine clinical care was initiated after 2000. The search terms include, but are not limited to “Patient-reported outcome measures”, “routine outcome assessment,” “self-reported outcome,” “patient outcome assessment,” “health-related quality of life,” Mental health,” “mental illness,” “mental disorder,” “mood disorder,” “Schizophrenia,” “eating disorder,” “psychological disorder,” “Depression” or “bipolar,” “anxiety,” and “self-harm.” Articles were selected for review if they: (1) used one or more PROMs (we included PROMs as measurement tools that are validated for use in different settings); (2) were conducted in a population <18 years of age with at least one MHC with a formal diagnosis according to the DSM 5 [24]; (3) was peer-reviewed; and (4) was published in English and the full text was available. Measures could be completed by children, parents, or both. Four team members (KTB, MMA, KW, and FN) independently screened titles and abstracts of all studies against our predetermined inclusion and exclusion criteria. The studies that did not meet the inclusion requirements were eliminated. Discrepancies were discussed and resolved by senior authors (MS and JZ) where necessary.

We identified 28 PROMs that have been used in youth living with mental health concerns. This work has been published [16].

Mapping evidence to the existing resources available and consultation with a family advisory group

The finding from the systematic review was mapped to the PROMs considered by the clinical team. The clinicians considered PROMs recommended by the International Consortium for Health Outcomes Measurements Set of Patient-Centered Outcome Measures for Children and Young People with Anxiety and Depression (ICHOM) [18]; PROMs identified in an Alberta pediatric measurement scan [27, 28] and a Measurement-based care (MBC) environmental scan Mental Health (measures identified in use by major Canadian child and youth mental health clinics and pediatric mental health programs across Canada), as well as a scoping review identifying PROMs in youth living with neurodevelopmental and mental health conditions [29]. After the mapping, the PROMs were reviewed by the researcher and youth partner and, through consultation, reduced to the top 10. These top 10 evidence-based and clinician-informed PROMs were then presented to the youth and family advisory council at the Summit Center [20] for consultation. During the consultation, the advisory group was asked about the appropriateness and general thought on the use of these PROMs in their clinical care (i.e., acceptable, feasible, relevant to the patient’s need). These findings are in a manuscript under submission.

This manuscript describes the project’s final phase, prioritizing the top 10 PROMs by patients and family/caregivers to inform the MBC program at the Summit Centre.

Modified NGT

The objective of the modified NGT was to prioritize the pre-identified PROMs based on relevance, feasibility, and overall importance. We started the NGT session by introducing the team members and participants. The facilitator then described the purpose of the study, the NGT process, and a list of the top 10 PROMs with a description of each PROMs for prioritization. The sessions were audio-recorded and transcribed verbatim. We conducted a content analysis of the transcript.

Typically, NGT comprises four stages: (1) silent generation of ideas, (2) Round robin, (3) Clarification and discussion, and (4) voting/ranking [20, 23]. The NGT process is a flexible consensus method that can be altered to fit the participant’s voice, the study’s purpose and goal, and the aim of generalizability [23, 26, 30]. In this study, the silent generation round was replaced with the explanation stage, where participants voted on the pre-identified PROMs based on relevance to the patient (domain coverage, e.g., pain, mental health or wellbeing, physical, social functioning, etc.), feasibility, (e.g., number of questions, timely completion, mode of administrations) and overall importance (Fig. 2). The list of PROMs for prioritization was displayed in Microsoft Forms for all participants to view in real-time. Participants were then asked to complete the survey, prioritizing PROMs from 1 to 10, with one being the least important and 10 being the most important. Prioritization was done anonymously using Microsoft Forms. Each participant received a unique sheet with the top 10 PROMs in one column and prioritizing values (1–10) listed in the adjacent column. Participants were then asked to rank each PROM based on the criteria provided, assigning only one number per PROM without sharing this with others. This step was crucial because it allowed participants to rank the PROMs independently before considering other perspectives that were revealed in the group discussion. Once the individual ranking was completed, the summary sheet that included the list of PROMs was reorganized based on the order of priority at the group level for further discussion.

Fig. 2
figure 2

Study design: modified nominal group technique process

Full size image

The next step after the voting round is discussing and sharing ideas. Here, there was a group review of the aggregate score of the initial rankings. These aggregate scores were displayed on a summary sheet and presented to participants. The participants were asked to explain their priorities and any comments related to the summary sheet. After this first round of discussion, participants had an opportunity to reconsider their initial ranking of the PROMs based on the discussion. Participants were not pressured to change their ranking or reach a consensus. The facilitators made sure that each participant had a chance to contribute to the discussion.

Throughout the NGT session, the youth researcher, with previous experience engaging in mental health research, was actively involved in making sure the research activities and outputs matched the youth patient’s perceptions and feedback and in creating a supported and comfortable environment for the youth patient to share their views and experiences as youth researcher share the similar experiences and language [31]. This role was crucial to ensure patients’ voices were central to the discussions and reduce the power imbalances during the NGT process.

Data analysis

To boost our interpretation of the findings, we used a mixed methods approach to analyze the result [23]. Participants’ descriptive characteristics were summarised using numbers and percentages. To determine the priority of each PROM, we calculated the median ranking and interquartile range for each prioritized PROM, as well as the minimum and maximum range ranking. Analysis was conducted using Microsoft Excel. We also created a cluster bar chart to compare the ranking of PROMs across patients and family/caregivers. A verbatim transcript was produced from the Zoom meeting recording and analyzed thematically.

Results

There were thirteen participants (five patients and eight family/caregivers). Five patients living with anxiety/and or depression participated in the NGT session, as seen in Table 1. Most of the youth participants were female (60%), and 40% of them were non-binary; 80% were older than 15 years, and only one participant was aged between 12 and 14 (20%). Eight families/caregivers aged between 40 and 49 participated in the NGT session; seven (87.5%) were female.

Table 1 Characteristics of participants
Full size table

Nominal group ranking

The PROMs prioritized by patients and family/caregivers before the discussion are displayed in Tables 2 and 3, respectively. We calculated the median and Interquartile range (IQR) for each PROM. Based on the median score and IQR, we ranked PROMs from top priority (highest median) to lowest priority (lowest median).

Table 2 Summary of NGT prioritization, parents/caregivers (N) = 8
Full size table
Table 3 Summary of NGT prioritization, patients (N = 5)
Full size table

The highest prioritized PROMs among family/caregivers are the Revised Child Anxiety and Depression Scale (RCAD 25), with the highest median score (9.5) for being comprehensive, short, and brief, followed by Screen for Child Anxiety Related Disorders (SCARED). The Child Health Questionnaire (CHQ) is moderate on the choice among Family/caregivers, with a median score (7.25). Similarly, the KIDSCREEN become the least choice among family/caregivers because of generic measures.

While the RCAD 25 was aligned with the top prioritized PROs by family/caregivers, as the top prioritized PROM among patients with the highest median rank score (9) as being relevant, comprehensive, and brief, the Beck Depression Inventory (BDI) was at the second choice with the median rank score (9) rather than SCARED. Family/caregivers poorly ranked the BDI.

Both groups of participants ranked The Young Person’s Core (YP-CORE) in third place. In contrast, while the Child Health Questionnaire was ranked fourth by family/caregivers, it was ranked lowest by youth patients. The Spence Children’s Anxiety Scale, the longest and, therefore, most time-consuming to fill out, ranked poorly by both patients and family/caregivers (7).

Re-ranking

After discussion, participants were asked if they wanted to revise their original ranking.

Family/caregiver priorities

After the discussion, the participants (family/caregivers) settled on the RCAD 25 as the best choice because of its relevancy, short form, and comprehensibility. The ranking order for the other two PROMs, SCARED and YP-CORE, remained the same.

A notable change in ranking was for Kidscreen 10 and CHQ, in which Kidscreen 10 increased in priority, and CHQ fell to the lowest rank after the discussion round as participants felt Kidscreen was short with ten questions and had good domain coverage. Some participants revealed that they liked the additional details asked in the SCAS. So, the participants decided to move SCAS higher in the ranking. Participants also discussed the importance of the aggression and rule-breaking behavior-related questions addressed in CBCL. This resulted in a change in the order of these PROMs. Participants also revealed a high preference for the PROMs that are rated by both parents and youths. Participants expressed that youth, especially teens, are ashamed of their feelings and some stigma around honesty in completing the questionnaire. Others stated that outcome measures should be filled out entirely by the patient independently, while some participants supported the idea of parents and youth filling out PROMs.

Further discussions revealed that the “negative” items in the questionnaires, relating to questions about the challenges and difficulties with mental health concerns, could be distressing. Two participants praised the Strength and Difficulties questionnaire for its content about positive mental health characteristics. The details regarding PROMs prioritization during NGT are summarized in the Table 4, and supporting quotations have been included in the Appendix.

Table 4 Summary of PROMs prioritized by family/caregivers and patients
Full size table

Patients’ priorities

There were a few changes in the ranking of PROMs order after discussion. The rank order of top 1, 2, and 3 among the youth participants remained the same. After the discussion, participants reached an agreement with CBCL (rank 4, pre-discussion) and SCAS (rank 6, pre-discussion). They noted that CBCL was too long and could lose accuracy at the end; participants felt that SCAS needed to be ranked higher because it was more relevant and shorter. Most of the participants agreed to lower their prioritization of PedsQl as they assumed the items were non-relevant to them as it was viewed to be targeted to a younger population. The PROMS ranked 5, 8, 9, and 10 were consistent.

Views among the youth participants regarding the number of questions in the tools were highly variable. Most youth reported getting bored doing those questionnaires if there were 100 questions. However, some participants prefer longer questionnaires with easier answers that provide more in-depth information while in crisis. One youth participant stated that filling out the questionnaire is not just tedious but does not accurately reflect patient perspectives, as patients tend to answer negatively just because they feel annoyed by the questionnaire. Participants indicated they like to have a chat with clinicians/counselors instead.

As shown in Fig. 3, the final rankings by patients and family/caregivers were the same for RCAD 25 (ranked as #1), YP-CORE (ranked #3), CBCL (ranked as #6), PedsQoL (ranked as #7), SDQ (ranked as #8), CHQ (ranked as #10). Both patients and family/caregivers highly prioritized RCAD 25 regarding feasibility, reliability, and overall importance. After discussion, NGT participants from both patients and family members felt that RCAD 25 was the best PROM because of its comprehensiveness (comprised both anxiety and depression, as well as obsessive-compulsive disorder) and because it was short and quick to complete.

Fig. 3
figure 3

PROMs prioritized by family and patients. Note 1 = Highest priority, 10 = lowest priority

Full size image

Discussion

We used a modified NGT to identify the relative priorities of pre-identified PROMs from the perspective of youth living with anxiety and/or depression and family/caregivers [20]. This study offered a novel and essential insight into integrating PROMs into clinical care to support youth mental health services. In general, the prioritized PROMs were broadly similar, with some minor differences in prioritizing priorities between patients and family/caregivers.

For patients and families/caregivers, the PROMs RCAD 25 and YP-CORE were the highest priorities. These priorities for PROMs reflected concerns about their relevance and feasibility. Both felt that RCAD 25 was comprehensive, short, easy, and quick to complete, whereas regarding YP-CORE, patients and family/caregivers thought it was also short and relevant. On the other hand, SDQ and CHQ were the lowest prioritized by patients and family/caregivers. The prioritization was underpinned by some reasons. Both thought the PROMs were too general, and most of the items needed to be more relevant to them.

There were some differences in the prioritization of PROMs by patients and family/caregivers. Based upon the relevance, family/caregivers gave high priority to SCARED, whereas it was lower prioritized among patients because patients thought it had too many items. Comparably, BDI was highly prioritized among the patients because they felt it was to the point and by length; it was neither too short nor too long. Surprisingly, parents/caregivers prioritized Kidscreen 10 higher because it is short and includes the items for assessing the quality of life. However, patients thought it was too general and irrelevant to them.

To our knowledge, this is one of the first studies to explore patients’ and family/caregiver perspectives on the relative priority of PROMs to be implemented in clinical care. Concerning the priorities, the ranking of PROMs did not alter markedly between the rounds in NGT among both groups. However, the discussion round led to greater consensus because the prioritized PROMs were deemed top priorities beyond the participants instead of having specific PROMs that were more important to the particular participants. Hence, it notifies that to comprehend the different perspectives and consensus building, the discussion was essential [32].

The study participants were quite clear that patient-reported measures are crucial to evaluate the outcome of treatments and interventions. They highlight the importance of parents/proxy-reported and self-reported questionnaires due to the stigma around youth not expressing their feelings so that parents would have more authentic feedback on behalf of their kids. Participants were concerned about the behavior and acting out related questions that do not necessarily indicate the patients’ struggle. This finding is consistent with the finding from Dowrick and colleagues, who found that some patients living with depression/anxiety may not answer the PROMs accurately to lessen stigma and/or influence treatment [33].

Our systematic approach led to the selection of disease-specific PROMs over generic PROMs. When we presented generic and disease-specific measures to NGT participants for prioritization, the benefit of disease-specific PROMs outweighed the generic PROMs; for instance, RCAD 25 was a priority, whereas CHQ was the least priority among the participants. This contradicts the previous study, which showed that NGT consensus for generic PROMs is opposed to arthritis-specific PROMs [34].

We found that participants were concerned with the overwhelming presence of negative items, which they felt would lead to inaccurate and invalid responses. This finding, accord to Marsh, presumed that younger children respond in polar extremes to negatively worded items because they require more verbal reasoning than positively worded items [35]. One solution to this might be the use of balanced scales in PROMs. That is, ensuring that scales have an equal number of positive and negative items [36].

In our study, participants noted concerns about the lengths of various questionnaires. Others have also indicated similar circumstances surrounding the acceptability of questionnaires based on length [25, 37]. A study by Kost et al. showed that shorter questionnaires are reliable and have higher responses and completion rates than lengthier questionnaires [38]. As the number of questions increases, participants tend to rush or skip over some of the items to get over it, adversely impacting the accuracy, reliability, and response rate [39].

The evidence suggested that the generic measures may not be suitable for those living with mental health concerns; there is also less evidence to support their appropriateness in this population, such as anxiety, depression, and so on [40, 41]. While the outcome measure such as PedsQl has been precisely developed to assess the health-related quality of life, the youth participants from our study conveyed the view that this measure was inadequately detailed to portray the quality of life of youth living with mental illness. Conversely, the family/caregivers articulated that they prefer Kidscreen 10, which is short and includes the items for assessing the quality of life. Remarkably, some participants also raised concerns about gender-based questionnaires, as identical questionnaires could not be effective for males, females, and non-binary because they are all present in very different ways.

This study is not without limitations. Both patients and family/caregivers were recruited via one research registry, which might affect the generalizability of the results to other clinical settings. In our NGT session, only one male participant in the family/caregiver group, and none of the participants in the youth patient group were male. Due to the small sample size, our findings may not represent the views of all youth patients living with anxiety and/or depression and family/caregivers [20]. However, the sample size is inherent to the NGT, a small group consensus-building method. We adhered to NGT standards, including five to eight participants [20, 42]. Having more than 10 participants per group may inhibit the open discussions needed in NGT, owing to the size of the group [43]. Lastly, although we included the measures that have been widely validated and used in youth mental health, focusing on anxiety and depression, others may have been missed in our pre-identified list of PROMS.

Study strengths include the use of qualitative and quantitative approaches to elicit patients’ and family members’ priorities and understand the reason for their prioritization. Qualitative data, audio transcribed verbatim and thematically analyzed, provided richness around PROMs’ preferences, relevance, acceptability, and overall feasibility. Furthermore, we employed a pragmatic approach to sort out the PROMs, where we could attain diverse representation in mapping and selecting the PROMs, representing clinicians, administrators, family advisory groups, and youth partners from Alberta. Finally, we conducted separate NGTs for patients and family/caregivers to ensure both perspectives were captured and taken into account.

Conclusion

The importance of incorporating patients’ and family/caregivers’ perspectives has been recognized in healthcare settings and research. This study provides practical recommendations around measures best suited to achieve this. The healthcare professional can incorporate these PROMs in their routine patient assessment and management. Additionally, these PROMs provide valuable information that might be utilized to address the concerns about the treatment efficacy, help to evaluate disease progression and regression, as well as the value of care provided [44], and help to identify the patients who may be at risk and allow the healthcare professional to evaluate the quality of their care continuously [45]. Healthcare professionals and policymakers can be guided by the patient-family/caregivers’ voices captured through consensus methodologies such as NGT to implement the PROMs relevant to patients and family members in mental health settings.

Data availability

All data generated and analyzed during this review are included in this published article.

Abbreviations

MHR 4 Kids:

Mental Health Research for Kids Research registry

MBC:

Measurement-based care

RCAD-25:

Revised Child Anxiety and Depression Scale

SCARED:

Screen for Child Anxiety Related Disorders

YP-CORE:

The Young Person’s Core

CHQ:

Child Health Questionnaire

PedsQl:

Pediatric Quality of Life Inventory

SDQ:

Strength and Difficulties Questionnaire

CBCL:

Child Behaviour Checklist

SCAS:

Spence Children’s Anxiety Scale

BDI:

Beck Depression Inventory

References

  1. Youth Mental Health Canada Youth mental health reality: the difference we can make. Available at https://ymhc.ngo/resources/ymh-stats/

  2. Power E, Hughes S, Cotter D, Cannon M (2020) Youth mental health in the time of COVID-19. Ir J Psychol Med 37(4):301–305. https://doi.org/10.1017/ipm.2020.84

    Article  CAS  PubMed  Google Scholar 

  3. Canada, Public Health Agency of (2015) Report from the Canadian chronic disease surveillance system: mental illness in Canada, 2015. https://www.canada.ca/en/public-health/services/publications/diseases-conditions/report-canadian-chronic-disease-surveillance-system-mental-illness-canada-2015.html

  4. Statistics Canada (2019) Table 13-10-0763-01 health characteristics of children and youth aged 1 to 17 years. In: Canadian health survey on CHILDREN and youth 2019. Statistics Canada

    Google Scholar 

  5. Center for Addiction and Mental Health. Mental illness and addiction: facts and statistics. Available at https://www.camh.ca/en/driving-change/the-crisis-is-real/mental-health-statistics

  6. Northcott A (2023) Canadian teens still struggling with mental health even as pandemic wanes. CBC NEWS

  7. Act for Mental Health The mental health strategy for Canada Available at https://www.mentalhealthcommission.ca/wpcontent/uploads/drupal/MHStrategy_Strategy_ENG.pdf

  8. Jorm AF, Patten SB, Brugha TS, Mojtabai R (2017) Has increased provision of treatment reduced the prevalence of common mental disorders? Review of the evidence from four countries. World Psychiatry 16(1):90–99

    Article  PubMed  PubMed Central  Google Scholar 

  9. Roe D, Mazor Y, Gelkopf M (2022) Patient-reported outcome measurements (PROMs) and provider assessment in mental health: a systematic review of the context of implementation. Int J Qual Health Care 34(Supplement_1):ii28–39

    Article  Google Scholar 

  10. Kristensen S, Mainz J, Baandrup L, Bonde M, Videbech P, Holmskov J, Bech P (2018) Conceptualizing patient-reported outcome measures for use within two Danish psychiatric clinical registries: description of an iterative co-creation process between patients and healthcare professionals. Nord J Psychiatry 72(6):409–419

    Article  CAS  PubMed  Google Scholar 

  11. Bickman L, Kelley SD, Breda C, de Andrade AR, Reimer M (2011) Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatr Serv 62(12):1423–1429. https://doi.org/10.1176/appi.ps.002052011

    Article  PubMed  Google Scholar 

  12. Basch E, Barbera L, Kerrigan CL, Velikova G (2018) Implementation of patient-reported outcomes in routine medical care. Am Soc Clin Oncol Educ Book 38:122–134. https://doi.org/10.1200/EDBK_200383

    Article  PubMed  Google Scholar 

  13. Pope RB, Hu I, D’Agostino M, Yoder W, Fagan E (2020) Push for PROMs patient-reported outcomes are necessary for evidence-based & client-centered community mental health treatment. https://familyandcommunityimpact.org/wp-content/uploads/2020/06/Push-for-PROMs.pdf

  14. McNeill M, Noyek S, Engeda E, Fayed N (2021) Assessing the engagement of children and families in selecting patient-reported outcomes (PROs) and developing their measures: a systematic review. Qual Life Res 30:983–995

    Article  PubMed  Google Scholar 

  15. Morris C, Janssens A, Shilling V, Allard A, Fellowes A, Tomlinson R, et al. (2015) Meaningful health outcomes for paediatric neurodisability: stakeholder prioritisation and appropriateness of patient reported outcome measures. Health Qual Life Outcomes 13:87. https://doi.org/10.1186/s12955-015-0284-7

    Article  PubMed  PubMed Central  Google Scholar 

  16. Thapa Bajgain K, Amarbayan M, Wittevrongel K, et al. (2023) Patient-reported outcome measures used to improve youth mental health services: a systematic review. J Patient Rep Outcomes 7(14). https://doi.org/10.1186/s41687-023-00556-0

  17. Thestrup Hansen S, Kjerholt M, Friis Christensen S, et al. (2020) User experiences on implementation of patient reported outcome measures (PROMs) in a haematological outpatient clinic. J Patient Rep Outcomes 4(87). https://doi.org/10.1186/s41687-020-00256-z

  18. Krause KR, Chung S, Adewuya AO, Albano AM, Babins-Wagner R, Birkinshaw L, Brann P, Creswell C, Delaney K, Falissard B, Forrest CB, Hudson JL, Ishikawa SI, Khatwani M, Kieling C, Krause J, Malik K, Martínez V, Mughal F, Ollendick TH, Ong SH, Patton GC, Ravens-Sieberer U, Szatmari P, Thomas E, Walters L, Young B, Zhao Y, Wolpert M (2021) International consensus on a standard set of outcome measures for child and youth anxiety, depression, obsessive-compulsive disorder, and post-traumatic stress disorder. Lancet Psychiatry 8(1):76–86. https://doi.org/10.1016/S2215-0366(20)30356-4

    Article  PubMed  Google Scholar 

  19. McKenzie E, Matkin L, Sousa Fialho L, Emelurumonye IN, Gintner T, Ilesanmi C, Jagger B, Quinney S, Anderson E, Baandrup L, Bakhshy AK, Brabban A, Coombs T, Correll CU, Cupitt C, Keetharuth AD, Lima DN, McCrone P, Moller M, Mulder CL, Roe D, Sara G, Shokraneh F, Sin J, Woodberry KA, Addington D, Psychotic Disorders Working Group of the International Consortium for Health Outcomes Measurement (2022) Developing an international standard set of patient-reported outcome measures for psychotic disorders. Psychiatr Serv 73(3):249–258. https://doi.org/10.1176/appi.ps.202000888

    Article  Google Scholar 

  20. https://www.buildthemup.ca/

  21. Carney O, McIntosh J, Worth A (1996) The use of the nominal group technique in research with community nurses. J Adv Nurs 23:1024–1029. https://doi.org/10.1046/j.1365-2648.1996.09623.x

    Article  CAS  PubMed  Google Scholar 

  22. Tuffrey-Wijne I, Bernal J, Butler G, Hollins S, Curfs L (2007) Using nominal group technique to investigate the views of people with intellectual disabilities on end-of-life care provision. J Adv Nurs 58(1):80–89. https://doi.org/10.1111/j.1365-2648.2007.04227.x

    Article  PubMed  Google Scholar 

  23. McMillan SS, Kelly F, Sav A, et al. (2014) Using the nominal group technique: how to analyse across multiple groups. Health Serv Outcomes Res Methodol 14:92–108

    Article  Google Scholar 

  24. Van de Ven AH, Delbecq AL (1972) The nominal group as a research instrument for exploratory health studies. Am J Public Health 62:337–342

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Crawford MJ, Robotham D, Thana L, Patterson S, Weaver T, Barber R, Wykes T, Rose D (2011) Selecting outcome measures in mental health: the views of service users. J Ment Health 20(4):336–346. https://doi.org/10.3109/09638237.2011.577114

    Article  PubMed  Google Scholar 

  26. Mullen R, Kydd A, Fleming A, McMillan L (2021) A practical guide to the systematic application of nominal group technique. Nurse Res 29:14–20

    Article  PubMed  Google Scholar 

  27. McCabe E, Rabi S, Bele S, et al. (2023) Factors affecting implementation of patient-reported outcome and experience measures in a pediatric health system. J Patient Rep Outcomes 7(24). https://doi.org/10.1186/s41687-023-00563-1

  28. Bele S, Rabi S, Zhang M, Oddone Paolucci E, Johnson DW, Quan H, Santana MJ Uptake of pediatric patient patient-reported outcome and experience measures and challenges associated with their implementation in Alberta. A mixed-method study (under review)

  29. Amarbayan M, Wittevrongel K, McCabe E, Thapa Bajgain K, et al. Measuring patient-reported outcome in youth with neurodevelopmental disorders and/or mental health conditions: a scoping review (Under review)

  30. Tully MP, Cantrill JA (1997) The use of the nominal group technique in pharmacy practice research: processes and practicalities. J Soc Adm Pharm 14:93–104

    Google Scholar 

  31. Spuerck I, Stankovic M, Fatima SZ, et al. (2023) International youth mental health case study of peer researchers’ experiences. Res Involv Engagem 9(33). https://doi.org/10.1186/s40900-023-00443-4

  32. Manalili K, Scott CM, Hemmelgarn B, O’Beirne M, Bailey AL, Haener MK, Banerjee C, Peters SP, Chiodo M, Aghajafari F, Santana MJ (2022) Co-designing person-centred quality indicator implementation for primary care in Alberta: a consensus study. Res Involv Engagem 8(1):59. https://doi.org/10.1186/s40900-022-00397-z

    Article  PubMed  PubMed Central  Google Scholar 

  33. Dowrick C, Leydon GM, McBride A, Howe A, Burgess H, Clarke P, et al. (2009) Patients’ and doctors’ views on depression severity questionnaires incentivised in UK quality and outcomes framework: qualitative study. BMJ 338:b663. https://doi.org/10.1136/bmj.b663

    Article  PubMed  Google Scholar 

  34. Schifferdecker KE, Butcher RL, Knight E, Creek E, Schrandt MS, Marrow L, Jaffe M, Vinci A, Eakin G (2020) Stakeholder development of an online program to track arthritis-related patient-reported outcomes longitudinally: live yes! INSIGHTS. ACR Open Rheumatol 2(12):750–759. https://doi.org/10.1002/acr2.11203

    Article  PubMed  PubMed Central  Google Scholar 

  35. Marsh H (1986) Negative item bias in ratings scales for preadolescent children: a cognitive-developmental phenomenon. Dev Psychol 22:37–49

    Article  Google Scholar 

  36. Coleman CM (2013) Effects of negative keying and wording in attitude measures: a mixed-methods study. Doctoral dissertation, James Madison University

  37. Blount C, Evans C, Birch S, Warren F, Norton K (2002) The properties of self-report research measures: beyond psychometrics. Psychol Psychother 75:151–164

    Article  PubMed  Google Scholar 

  38. Kost RG, de Rosa JC (2018) Impact of survey length and compensation on validity, reliability, and sample characteristics for ultrashort-, short-, and long-research participant perception surveys. J Clin Transl Sci 2:31–37

    Article  PubMed  PubMed Central  Google Scholar 

  39. Sharma H (2022) How short or long should be a questionnaire for any research? Researchers dilemma in deciding the appropriate questionnaire length. Saudi J Anaesth 16(1):65–68. https://doi.org/10.4103/sja.sja_163_21

    Article  PubMed  PubMed Central  Google Scholar 

  40. Papaioannou D, Brazier J, Parry G (2013) How to measure quality of life for cost-effectiveness analyses of personality disorders: a systematic review. J Pers Disord 27(3):383–401

    Article  PubMed  Google Scholar 

  41. Brazier J, Connell J, Papaioannou D, Mukuria C, Mulhern B, Peasgood T, Jones ML, Paisley S, O’Cathain A, Barkham M, Knapp M (2014) A systematic review, psychometric analysis and qualitative assessment of generic preference-based measures of health in mental health populations and the estimation of mapping functions from widely used specific measures. Health Technol Assess 18(34):vii

    Article  PubMed  PubMed Central  Google Scholar 

  42. Stewart D, Shamdasani P, Rook D (2007) Recruiting focus group participants and designing the interview guide. In: Focus Groups, 2nd ed. SAGE Publications, Newcastle upon Tyne, UK, pp 51–69

  43. Chiwire P, Evers SM, Mahomed H, Hiligsmann M (2022) Identification and prioritization of attributes for a discrete choice experiment using the nominal group technique: patients’ choice of public health facilities in Cape Town, South Africa. Value Health Reg Issues 27:90–98

    Article  PubMed  Google Scholar 

  44. Foley T, Vale L (2023) A framework for understanding, designing, developing and evaluating learning health systems. Learn Health Syst 7(1):e10315

    Article  PubMed  Google Scholar 

  45. Valderas JM, Alonso J, Guyatt GH (2008) Measuring patient-reported outcomes: moving from clinical trials into clinical practice. Med J Aust 189(2):93–94

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors would like to greatly acknowledge the study participants for their great contribution to the study. We also gratefully acknowledge the contributions from the Alberta Children’s Hospital Foundation funding. Jennifer D. Zwicker is supported by a Tier II Canada Research Chair in Disability Policy for Children and Youth. We would also like to express our appreciation to the Mental Health 4 Kids (MH4 Kids) Research Registry for their help in the participant recruitment. The authors have all stated that they had no interests that might be perceived as posing a conflict or bias.

Funding

This work was supported by the Alberta Children’s Hospital Foundation.

Author information

Authors and Affiliations

  1. Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada

    Kalpana Thapa Bajgain, Farwa Naqvi & Maria-Jose Santana

  2. General Psychology, Capella University, Minneapolis, MN, USA

    Justino Mendoza

  3. Department of Medicine and Community Health Science, University of Calgary, Calgary, Alberta, Canada

    Fariba Aghajafari & Karen Tang

  4. School of Public Policy, University of Calgary, Calgary, Alberta, Canada

    Jennifer Zwicker

  5. Department of Pediatrics, University of Calgary, Calgary, Alberta, Canada

    Maria-Jose Santana

  6. Faculty of Kinesiology, University of Calgary, Calgary, Alberta, Canada

    Jennifer Zwicker

  7. Alberta Children’s Hospital, 2888 Shaganappi Trail NW, Calgary, Canada

    Maria-Jose Santana

Authors
  1. Kalpana Thapa Bajgain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Justino Mendoza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Farwa Naqvi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fariba Aghajafari
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Karen Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jennifer Zwicker
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Maria-Jose Santana
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MS and JZ conceptualized and designed the study, coordinated supervised data collection, and reviewed and revised the manuscript. KTB, JM, and FN conducted the NGT session and critically revised the manuscript for important intellectual content. KTB conceptualized and designed the study, designed the data collection instruments, collected data and initial analysis, drafted the initial manuscript, and revised the manuscript. KT and FA provided expert opinions on the conduct of the study. All authors contributed to the drafting of the manuscript and approved the final version.

Corresponding author

Correspondence to Maria-Jose Santana.

Ethics declarations

Ethical approval

This study was performed in line with the principles of the Declaration of Helsinki. The ethical approval for this study was obtained from the University of Calgary’s Research Ethics Board (REB22-0880).

Informed consent

The ethical approval obtained from the University of Calgary’s Research Ethics Board (REB22-0880) included obtaining verbal informed consent from all study participants prior to participating in the Nominal Group Technique session.

Consent for publication

Not applicable.

Competing interests

The authors declare that the research was conducted in the absence of any commercial or financial relationship that could be constructed as a potential conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 5 Summary of PROMs prioritized by family/caregivers and patients
Full size table

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bajgain, K., Mendoza, J., Naqvi, F. et al. Prioritizing Patient Reported Outcome Measures (PROMs) to use in the clinical care of youth living with mental health concerns: a nominal group technique study. J Patient Rep Outcomes 8, 20 (2024). https://doi.org/10.1186/s41687-024-00694-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s41687-024-00694-z

Keywords