- Short report
- Open access
- Published:
First stage psychometric testing of a new instrument for adolescents with visual impairment: the Participation and Activity Inventory for Children and Youth (PAI-CY) 13–17 years
Journal of Patient-Reported Outcomes volume 4, Article number: 62 (2020)
Abstract
Background
To assess participation of children with visual impairment, the Participation and Activity Inventory for Children and Youth (PAI-CY) was recently developed. This study assessed some initial psychometric properties of the PAI-CY 13–17 years version, and investigated its feasibility.
Methods
Adolescents with visual impairment and their parents (n = 72 dyads) completed the self-report and proxy-report version of the 58-item PAI-CY, an evaluation form and several questionnaires measuring related constructs. Item deletion was informed by item responses, inter-item correlations, test-retest reliability, adolescent-parent agreement and participants’ feedback. Known-group validity and concurrent validity with related questionnaires were investigated for the final item-set.
Results
Twelve items had > 20% missing values, whereas 39 items showed floor effects. Eight item pairs showed high inter-item correlations. Test-retest reliability was acceptable for most items (kappa ≥0.4). Evaluation forms showed that over 90% of respondents was neutral to very positive regarding several feasibility aspects such as administration time and comprehensiveness. Adolescent-parent agreement was mostly low. These results informed the deletion of three items. Known-group validity seemed adequate since PAI-CY scores were significantly worse for participants with comorbidity compared to those without. A trend towards worse scores for participants with more severe visual impairment was also observed. Correlations between the PAI-CY and related questionnaires confirmed concurrent validity.
Conclusions
Initial psychometric properties of the PAI-CY 13–17 were acceptable, although more work is needed to assess other psychometric properties, such as the underlying construct. Following implementation in low vision care to assess participation needs, enabling larger samples, acceptability of the PAI-CY 13–17 to end-users should be carefully monitored, especially if alterations are made based on the current study.
Background
Patient-reported outcome measures (PROMs) evaluate outcomes of health, illness or treatment that are important to patients and directly reported by patients themselves. They include for example symptom severity, health perception, and quality of life [1]. PROMs are increasingly used for evaluating health services, establishing treatment effectiveness, and informing clinical decision-making. PROMs can contribute to clinician-patient communication, facilitate the process of shared decision-making and improve patient satisfaction with care [2,3,4].
Numerous PROMs currently exist, and a fair share are developed specifically for children. These include generic instruments (e.g. [5,6,7]) facilitating comparison of different patient populations, as well as disease-specific instruments specifically targeted to those with the condition of interest. Until recently there was a paucity in the development of PROMs to capture the perspectives of children and adolescents with visual impairment (VI), possibly relating to the challenges to develop PROMs for this population. For instance, psychometric validation of PROMs depends on large and representative samples, but the population of children with VI is numerically small, complex, heterogeneous and difficult to reach. Despite the challenges, various instruments for children and adolescents with VI have been developed in recent years, mainly focusing on vision-related quality of life and functional vision (e.g. [8,9,10,11,12]).
We recently developed the Participation and Activity Inventory for Children and Youth (PAI-CY), which focuses on participation of children with VI. Four age-appropriate versions of the PAI-CY were developed reflecting the developmental age-bands of the World Health Organization (WHO): 0–2 years (proxy-report), 3–6 years (proxy-report), 7–12 years (proxy and self-report) and 13–17 years (proxy and self-report). Additionally, the PAI Young Adults (PAI-YA) was developed aimed at persons aged 18–25 years. The PAI-CY was developed to structure the process of identifying needs from the perspectives of children with VI and their parents at Dutch low vision services. Its content was shaped involving end-users as stakeholders in a concept-mapping study [13], ensuring face and content validity, and a small-scale pilot-study demonstrated its feasibility and acceptability [14].
This study reports the first stage psychometric evaluation of the PAI-CY 13–17. This includes the assessment of some important psychometric properties, such as test-retest reliability, known-group validity and concurrent validity with other instruments. These analyses were intended to capture the worst performing items, resulting in initial item reduction. Moreover, some feasibility aspects were assessed.
Methods
Subjects and procedures
Adolescents aged 13–17 years registered at two Dutch low vision services and their parents/caretakers (parents for brevity) were invited to participate. Participants had to have adequate knowledge and understanding of the Dutch language. Children with major cognitive impairment were excluded from invitation by the low vision rehabilitation centers. Prior to participation in the study, written informed consent was obtained from adolescents and their parents. Adolescents completed the questionnaires through a face-to-face interview in their own homes, whereas parents completed the questionnaires through a web-based survey. For adolescents, the questionnaires consisted of the PAI-CY 13–17, a self-constructed evaluation form to further assess some feasibility aspects, and the Dutch versions of the Child and Adolescent Scale of Participation (CASP) [7], Kidscreen-27 [5], and the Functional Vision Questionnaire for Children and Young People (FVQ_CYP_NL) [8, 15]. Parents completed the proxy versions of the PAI-CY 13–17 and evaluation form, CASP and Kidscreen. Parents also completed questions regarding sociodemographic and clinical characteristics of their child. Approximately 2 weeks after initial completion [16], participants were asked to complete a retest. Adolescents were interviewed by the same interviewer. Ophthalmic information of adolescents was retrieved from the patient files at the low vision services organizations; missing values were complemented with self-reported data from parents (n = 8). Visual acuity was classified using five levels based on the better seeing eye, according the criteria of the WHO [17].
The PAI-CY 13–17
The preliminary version of the PAI-CY 13–17 contained 58 items grouped into eight domains (for descriptive purposes only, in order to provide contextual meaning) that were informed by concept-mapping workshops with end-users (see Table 2) [13]. Items were scored on a 4-point Likert scale with response options not difficult (1), slightly difficult (2), very difficult (3), and impossible (4). The response option not applicable was handled as a missing value. The items in the proxy version only differed in first or third person tense.
Statistical analyses
As a first step in the psychometric evaluation of the PAI-CY 13–17, item reduction was performed conservatively, using lenient criteria. Items were considered for elimination if the psychometric performance was suboptimal on several criteria and if item content was already adequately represented in other items: a) Items with missing scores > 20% [8, 16] in both the adolescent and parent version and/or items with > 70% [8, 18, 19] of both adolescents and parents endorsing the first or last response category (i.e. floor or ceiling effects). b) Items showing inter-item correlations > 0.8, indicating potential redundancy. c) Suboptimal values of test-retest reliability [16] using weighted kappa and percentage agreement [20, 21]. The intraclass correlation coefficient (ICC) for sum scores of the test and retest data were calculated based on absolute agreement, with a two-way mixed-effects model.
Subsequently, adolescent-parent agreement at item level was investigated using weighted kappa and percentage agreement. The ICC for the sum scores of the adolescent and parent data were calculated based on absolute agreement, with a two-way random-effects model. Comments and suggestions evolving from the evaluation forms were used to inform decisions on item elimination or instrument adaptation.
Known-group validity on the final item-set [16] was investigated using independent samples t-tests (median split for age) and ANOVAs with a post-hoc Tukey test. Lastly, concurrent validity [16] with subscales of the Kidscreen, CASP and FVQ_CYP_NL was investigated with Spearman’s correlations. Negative correlations were expected between the PAI-CY and subscales of the Kidscreen and CASP, whereas positive correlations were expected with the FVQ_CYP_NL. Correlations were expected to be smallest between the PAI-CY and the Kidscreen and largest for the FVQ_CYP_NL.
Results
Of an estimated 700 invited adolescents and their parents, 77 provided written informed consent and completed the first questionnaire (either the adolescent or parent version). Main reasons for non-participation were no time, not interested or adolescents not willing to participate. Sociodemographic and clinical characteristics are presented in Table 1. Complete data was obtained from 72 adolescent-parent dyads. The retest was completed by 64 parents after 28.3 ± 24.3 (range 9–121, median 18) days and 74 adolescents after 15.9 ± 7.0 (range 7–44, median 14) days.
Twelve items had missing responses > 20% in either the adolescent or parent version, of which eight items had > 20% missing scores in both versions (Table 2). Floor effects were found in 39 items in either the adolescent or parent version, of which 13 items displayed floor effects in both versions. Inter-item correlations of > 0.8 were found in one item pair for adolescents and seven item pairs for parents. Most items showed satisfactory test-retest reliability. For three items agreement was < 60%, whereas for eight items weighted kappa was < 0.4. ICC for the sum score on test and retest for adolescents was 0.866 (95%-confidence interval: 0.791–0.915). For parents, the ICC was 0.882 (95%-confidence interval: 0.815–0.927). With respect to adolescent-parent agreement, 34% of the items showed agreement < 60%, while 79% of the items showed weighted kappa’s ≤ 0.4. ICC between adolescent and parent scores was 0.438 (95%-confidence interval: 0.000–0.694).
The evaluation forms showed that parents’ self-reported administration time (including questions about sociodemographic and clinical characteristics) was 22 ± 14 (range 8–60, median 17) minutes. Over 90% of both adolescents and parents were neutral to very positive regarding various feasibility aspects of the PAI-CY (Fig. 1). However, comments and suggestions for improvements were made by 42 adolescents and/or parents (Table 3).
From these results, it was decided to eliminate items LT8, MO2, and CO4, resulting in a PAI-CY 13–17 containing 55 items. Infrequent endorsement of the answer category ‘impossible’ motivated collapsing this category with the category ‘very difficult’. Table 4 shows expected patterns of correlations between the PAI-CY and concurrent questionnaires.
Adolescents with comorbidity scored significantly worse on the PAI-CY 13–17 than adolescents without comorbidity, both in the adolescent and parent version (effect sizes were moderate to large). In the adolescent version, significantly worse scores were found for adolescents with blindness compared to those with moderate VI and no VI (large effect sizes). All other effect sizes were small (Fig. 2).
Discussion
In this study the psychometric properties of the PAI-CY 13–17 were evaluated, a questionnaire to assess the needs of Dutch adolescents with VI and their parents. We used less stringent criteria for item elimination because of the importance of face and content validity and the small sample size. Therefore, we only eliminated three items, of which content was thought to be adequately represented in the remaining items (LT8: making music can be captured with LT9: performing a hobby, MO2: cycling to something independently with MO1: cycling and CO4: participating in a conversation actively with CO1: Being able to express in words properly). Eliminating more items while having a small sample size could be counterproductive in the long term, causing potentially informative items to be discarded before larger samples are available. Once the PAI-CY 13–17 is used in future, confirmation of the findings and the assessment of other psychometric properties should inform further item reduction, although the current number of items is in line with other age-versions of the PAI-CY and PAI-YA, containing 27–60 items [22,23,24,25]. Moreover, with larger samples item response theory modeling might be employed to make the instrument more precise and user-friendly.
Test-retest reliability was satisfactory for most items. Exclusion of participants who completed the retest > 45 days after the first questionnaire (n = 10) in general had a positive influence on kappa, agreement and the ICC (data not shown). Remarkably, the items with suboptimal kappa values showed adequate agreement. This phenomenon has previously been reported as the ‘paradox’ of the kappa statistic, in which low values of kappa can be found for high values of agreement [26, 27], caused by symmetrically imbalanced contingency tables [28]. For example, for item LT7: participating at a club, many adolescents opted “not difficult” on the test and retest, resulting in a high proportion of adolescents (0.98) in row 1 in the contingency table. As the proportion of respondents in row 1 increases, kappa decreases rapidly and may even fall below zero, as has happened for this item, despite excellent agreement. Therefore, agreement and weighted kappa should be re-examined with larger samples. The ICC between the sum scores of the test and retest data were high for both adolescents and parents, indicating good reliability. Adolescent-parent agreement and weighted kappa was low for many items, as was the ICC between the sum scores, indicating that adolescents and their parents have different perceptions about the difficulties adolescents experience. In general, adolescents perceive items as less difficult than parents. Notably, agreement was highest for those items taking place at home, i.e. items in the self-reliance domain. This might indicate that for activities taking place elsewhere, parents might be less aware about the difficulties experienced by their child, and therefore overestimate the difficulties experienced by adolescents. However, due to their age, adolescents also might overestimate their abilities and perceive less risks. Therefore, retrieving information from both sources is important to comprehensively assess adolescents’ needs.
It was possible to differentiate between adolescents with or without comorbidity, both with the adolescent and parent version. Furthermore, it was possible to differentiate between different degrees of VI with the adolescent version, while there was a trend with the parent version. The correlations with other instruments demonstrated concurrent validity of the PAI-CY 13–17. The relatively low correlation with subscales of the Kidscreen demonstrates that the PAI-CY 13–17 is not necessarily measuring quality of life, whereas the strong correlations with (subscales) of the CASP and the FVQ_CYP_NL confirms that the PAI-CY 13–17 rather measures a construct related to participation or functional vision. Although one can argue that the PAI-CY and FVQ_CYP_NL might be similar, the relatively strong but not perfect correlation shows that both instruments measure a similar construct, but each in its unique way.
One of the major limitations of the current study is the small sample size and the low response rate, although anticipated from previous studies involving similar populations. Main reasons for non-participation according to parents were no time, not interested or their child stating they did not want to participate. Adolescents might be reluctant to participate in scientific research, especially if they have to make an appointment and are visited by a researcher in their own homes.
The small sample size prevented factor analysis and the use of item response theory to optimize the PAI-CY 13–17. However, previous studies examining other age-versions of the PAI-CY and the PAI-YA showed the items comprised a unidimensional scale [22,23,24]. As such, the PAI-CY 13–17 is expected to be unidimensional as well, and we calculated sum scores for the complete scale.
The current study used a national sample, implying that the PAI-CY 13–17 should be applicable across the Dutch population. Although the PAI-CY 13–17 can be used as a template for use in other countries, cross-cultural validation is recommended for use outside the Netherlands. Recent interest to use various age-versions of the PAI-CY and the PAI-YA in Nepal and Australia, has resulted in official forward-backward translations of the instruments and manuals (available upon request from the authors) to English and Nepali, in which cultural applicability of items was also taken into account. Further research should indicate whether the PAI-CY 13–17 is indeed applicable and valid in these countries.
Conclusions
Based on initial psychometric tests, the PAI-CY 13–17 appears to have acceptable measurement properties, although more work with larger samples is needed, for example to assess whether the items comprise a unidimensional scale. It is possible to implement the current version of the PAI-CY 13–17 in Dutch low vision services, enabling more data collection and more extensive understanding of its psychometric properties. Acceptability of the PAI-CY 13–17 to end-users should be carefully monitored, especially if the changes suggested in this study are going to be incorporated.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Abbreviations
- AC:
-
Acceptance/self-consciousness
- CASP:
-
Child and Adolescent Scale of Participation
- CO:
-
Communication
- FI:
-
Finances
- FVQ_CYP_NL:
-
Dutch version of the Functional Vision Questionnaire for Children and Young People
- ICC:
-
Intraclass correlation coefficient
- LT:
-
Leisure time
- MO:
-
Mobility
- PAI-CY:
-
Participation and Activity Inventory for Children and Youth
- PROMs:
-
Patient-reported outcome measures
- SC:
-
Social contacts
- SL:
-
School
- SR:
-
Self-reliance
- VI:
-
Visual impairment
- WHO:
-
World Health Organization
References
Valderas, J. M., & Alonso, J. (2008). Patient reported outcome measures: a model-based classification system for research and clinical practice. Quality of Life Research, 17(9), 1125–1135. https://doi.org/10.1007/s11136-008-9396-4.
Santana, M. J., & Feeny, D. (2014). Framework to assess the effects of using patient-reported outcome measures in chronic care management. Quality of Life Research, 23(5), 1505–1513. https://doi.org/10.1007/s11136-013-0596-1.
Valderas, J. M., Kotzeva, A., Espallargues, M., Guyatt, G., Ferrans, C. E., Halyard, M. Y., et al. (2008). The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Quality of Life Research, 17(2), 179–193. https://doi.org/10.1007/s11136-007-9295-0.
Chen, J., Ou, L. X., & Hollis, S. J. (2013). A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Services Research, 13, Artn 311. https://doi.org/10.1186/1472-6963-13-211.
Ravens-Sieberer, U., Auquier, P., Erhart, M., Gosch, A., Rajmil, L., Bruil, J., et al. (2007). The KIDSCREEN-27 quality of life measure for children and adolescents: psychometric results from a cross-cultural survey in 13 European countries. Quality of Life Research, 16(8), 1347–1356. https://doi.org/10.1007/s11136-007-9240-2.
Varni, J. W., Seid, M., & Kurtin, P. S. (2001). PedsQL™ 4.0: reliability and validity of the pediatric quality of life inventory™ version 4.0 generic core scales in healthy and patient populations. Medical Care, 39(8), 800–812.
Bedell, G. (2009). Further validation of the Child and Adolescent Scale of Participation (CASP). Developmental Neurorehabilitation, 12(5), 342–351.
Tadic, V., Cooper, A., Cumberland, P., Lewando-Hundt, G., & Rahi, J. S. (2013). Vision-related quality of life G. Development of the functional vision questionnaire for children and young people with visual impairment: The FVQ_CYP. Ophthalmology., 120(12), 2725–2732. https://doi.org/10.1016/j.ophtha.2013.07.055.
Khadka, J., Ryan, B., Margrain, T. H., Court, H., & Woodhouse, J. M. (2010). Development of the 25-item Cardiff Visual Ability Questionnaire for Children (CVAQC). The British Journal of Ophthalmology, 94(6), 730–735. https://doi.org/10.1136/bjo.2009.171181.
Birch, E. E., Cheng, C. S., & Felius, J. (2007). Validity and reliability of the Children’s Visual Function Questionnaire (CVFQ). Journal of American Association for Pediatric Ophthalmology and Strabismus, 11(5), 473–479.
Cochrane, G. M., Marella, M., Keeffe, J. E., & Lamoureux, E. L. (2011). The Impact of Vision Impairment for Children (IVI_C): validation of a vision-specific pediatric quality-of-life questionnaire using Rasch analysis. Investigative Ophthalmology & Visual Science, 52(3), 1632–1640.
Hatt, S. R., Leske, D. A., Castaneda, Y. S., Wernimont, S. M., Liebermann, L., Cheng-Patel, C. S., et al. (2019). Development of pediatric eye questionnaires for children with eye conditions. American Journal of Ophthalmology, 200, 201–217. https://doi.org/10.1016/j.ajo.2019.01.001.
Rainey, L., Elsman, E. B. M., van Nispen, R. M. A., van Leeuwen, L. M., & van Rens, G. (2016). Comprehending the impact of low vision on the lives of children and adolescents: A qualitative approach. Quality of Life Research, 25(10), 2633–2643. https://doi.org/10.1007/s11136-016-1292-8.
Elsman, E. B. M., van Nispen, R. M. A., & van Rens, G. (2017). Feasibility of the Participation and Activity Inventory for Children and Youth (PAI-CY) and Young Adults (PAI-YA) with a visual impairment: a pilot study. Health and Quality of Life Outcomes, 15(1), 98. https://doi.org/10.1186/s12955-017-0677-x.
Elsman, E. B., Tadić, V., Peeters, C. F., van Rens, G. H., Rahi, J. S., & van Nispen, R. M. (2019). Cross-cultural validation of the Functional Vision Questionnaire for Children and Young People (FVQ_CYP) with visual impairment in the Dutch population: challenges and opportunities. BMC Medical Research Methodology, 19(1), 221.
De Vet, H. C. W., Terwee, C. B., Mokkink, L. B., & Knol, D. L. (2011). Measurement in medicine: a practical guide. New York: Cambridge University Press.
WHO (1994). ICD-10: International statistical classification of diseases and related health problems, 10th revision. Geneva: WHO.
Tadic, V., Cooper, A., Cumberland, P., Lewando-Hundt, G., & Rahi, J. S. (2016). Vision-related quality of life g. Measuring the quality of life of visually impaired children: first stage psychometric evaluation of the novel VQoL_CYP instrument. PLoS One, 11(2), e0146225. https://doi.org/10.1371/journal.pone.0146225.
Pesudovs, K., Burr, J. M., Harley, C., & Elliott, D. B. (2007). The development, assessment, and selection of questionnaires. Optometry and Vision Science, 84(8), 663–674.
Altman, D. G. (1991). Practical statistics for medical research. London: Chapman and Hall.
Singh, A. S., Vik, F. N., Chinapaw, M. J. M., Uijtdewilligen, L., Verloigne, M., Fernandez-Alvira, J. M., et al. (2011). Test-retest reliability and construct validity of the ENERGY-child questionnaire on energy balance-related behaviours and their potential determinants: the ENERGY-project. International Journal of Behavioral Nutrition and Physical Activity, 8, Artn 136. https://doi.org/10.1186/1479-5868-8-136.
Elsman, E. B., van Nispen, R. M., & van Rens, G. H. (2020). Psychometric evaluation of the Participation and Activity Inventory for Children and Youth (PAI-CY) 0–2 years with visual impairment. Quality of Life Research, 29(3), 775–781.
Elsman, E. B. M., van Nispen, R. M. A., & van Rens, G. H. (2019). Psychometric evaluation of a new proxy-instrument to assess participation in children aged 3–6 years with visual impairment: PAI-CY 3-6. Ophthalmic and Physiological Optics, 39(5), 378–391. https://doi.org/10.1111/opo.12642.
Elsman, E. B. M., van Rens, G. H. M. B., & van Nispen, R. M. A. (2018). Psychometric properties of a new intake questionnaire for visually impaired young adults: The Participation and Activity Inventory for Young Adults (PAI-YA). PLos One, 13(8), e0201701.
Elsman, E. B., Peeters, C. F., van Nispen, R. M., & van Rens, G. H. (2020). Network analysis of the Participation and Activity Inventory for Children and Youth (PAI-CY) 7–12 years with visual impairment. Translational Vision Science & Technology, 9(6), 19.
Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but Low Kappa .1. The problems of 2 paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10.1016/0895-4356(90)90158-l.
Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but Low Kappa .2. Resolving the Paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558. https://doi.org/10.1016/0895-4356(90)90159-M.
Flight, L., & Julious, S. A. (2019). The disagreeable behaviour of the kappa statistic. Sheffield https://www.sheffield.ac.uk/polopoly_fs/1.404095!/file/RSS_Poster_Laura_Flight_Final.pdf: The University of Sheffield.
Acknowledgements
We greatly thank all participating parents and adolescents for their contributions.
Funding
This study was funded by Royal Dutch Visio/Novum.
Author information
Authors and Affiliations
Contributions
GvR and RvN contributed to the inception and design of the study. EE recruited participants and collected the data. EE and RvN contributed to the statistical analyses. EE drafted the manuscript. GvR and RvN provided substantial feedback on the manuscript. All authors approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
The study protocol was approved by the Medical Ethical Committee of the VU University Medical Center, Amsterdam, the Netherlands. All procedures performed in the study involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Elsman, E.B.M., van Nispen, R.M.A. & van Rens, G.H.M.B. First stage psychometric testing of a new instrument for adolescents with visual impairment: the Participation and Activity Inventory for Children and Youth (PAI-CY) 13–17 years. J Patient Rep Outcomes 4, 62 (2020). https://doi.org/10.1186/s41687-020-00228-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s41687-020-00228-3