First stage psychometric testing of a new instrument for adolescents with visual impairment: the Participation and Activity Inventory for Children and Youth (PAI-CY) 13–17 years

Abstract

Background

To assess participation of children with visual impairment, the Participation and Activity Inventory for Children and Youth (PAI-CY) was recently developed. This study assessed some initial psychometric properties of the PAI-CY 13–17 years version, and investigated its feasibility.

Methods

Adolescents with visual impairment and their parents (n = 72 dyads) completed the self-report and proxy-report version of the 58-item PAI-CY, an evaluation form and several questionnaires measuring related constructs. Item deletion was informed by item responses, inter-item correlations, test-retest reliability, adolescent-parent agreement and participants’ feedback. Known-group validity and concurrent validity with related questionnaires were investigated for the final item-set.

Results

Twelve items had > 20% missing values, whereas 39 items showed floor effects. Eight item pairs showed high inter-item correlations. Test-retest reliability was acceptable for most items (kappa ≥0.4). Evaluation forms showed that over 90% of respondents was neutral to very positive regarding several feasibility aspects such as administration time and comprehensiveness. Adolescent-parent agreement was mostly low. These results informed the deletion of three items. Known-group validity seemed adequate since PAI-CY scores were significantly worse for participants with comorbidity compared to those without. A trend towards worse scores for participants with more severe visual impairment was also observed. Correlations between the PAI-CY and related questionnaires confirmed concurrent validity.

Conclusions

Initial psychometric properties of the PAI-CY 13–17 were acceptable, although more work is needed to assess other psychometric properties, such as the underlying construct. Following implementation in low vision care to assess participation needs, enabling larger samples, acceptability of the PAI-CY 13–17 to end-users should be carefully monitored, especially if alterations are made based on the current study.

Background

Patient-reported outcome measures (PROMs) evaluate outcomes of health, illness or treatment that are important to patients and directly reported by patients themselves. They include for example symptom severity, health perception, and quality of life [1]. PROMs are increasingly used for evaluating health services, establishing treatment effectiveness, and informing clinical decision-making. PROMs can contribute to clinician-patient communication, facilitate the process of shared decision-making and improve patient satisfaction with care [2,3,4].

Numerous PROMs currently exist, and a fair share are developed specifically for children. These include generic instruments (e.g. [5,6,7]) facilitating comparison of different patient populations, as well as disease-specific instruments specifically targeted to those with the condition of interest. Until recently there was a paucity in the development of PROMs to capture the perspectives of children and adolescents with visual impairment (VI), possibly relating to the challenges to develop PROMs for this population. For instance, psychometric validation of PROMs depends on large and representative samples, but the population of children with VI is numerically small, complex, heterogeneous and difficult to reach. Despite the challenges, various instruments for children and adolescents with VI have been developed in recent years, mainly focusing on vision-related quality of life and functional vision (e.g. [8,9,10,11,12]).

We recently developed the Participation and Activity Inventory for Children and Youth (PAI-CY), which focuses on participation of children with VI. Four age-appropriate versions of the PAI-CY were developed reflecting the developmental age-bands of the World Health Organization (WHO): 0–2 years (proxy-report), 3–6 years (proxy-report), 7–12 years (proxy and self-report) and 13–17 years (proxy and self-report). Additionally, the PAI Young Adults (PAI-YA) was developed aimed at persons aged 18–25 years. The PAI-CY was developed to structure the process of identifying needs from the perspectives of children with VI and their parents at Dutch low vision services. Its content was shaped involving end-users as stakeholders in a concept-mapping study [13], ensuring face and content validity, and a small-scale pilot-study demonstrated its feasibility and acceptability [14].

This study reports the first stage psychometric evaluation of the PAI-CY 13–17. This includes the assessment of some important psychometric properties, such as test-retest reliability, known-group validity and concurrent validity with other instruments. These analyses were intended to capture the worst performing items, resulting in initial item reduction. Moreover, some feasibility aspects were assessed.

Methods

Subjects and procedures

Adolescents aged 13–17 years registered at two Dutch low vision services and their parents/caretakers (parents for brevity) were invited to participate. Participants had to have adequate knowledge and understanding of the Dutch language. Children with major cognitive impairment were excluded from invitation by the low vision rehabilitation centers. Prior to participation in the study, written informed consent was obtained from adolescents and their parents. Adolescents completed the questionnaires through a face-to-face interview in their own homes, whereas parents completed the questionnaires through a web-based survey. For adolescents, the questionnaires consisted of the PAI-CY 13–17, a self-constructed evaluation form to further assess some feasibility aspects, and the Dutch versions of the Child and Adolescent Scale of Participation (CASP) [7], Kidscreen-27 [5], and the Functional Vision Questionnaire for Children and Young People (FVQ_CYP_NL) [8, 15]. Parents completed the proxy versions of the PAI-CY 13–17 and evaluation form, CASP and Kidscreen. Parents also completed questions regarding sociodemographic and clinical characteristics of their child. Approximately 2 weeks after initial completion [16], participants were asked to complete a retest. Adolescents were interviewed by the same interviewer. Ophthalmic information of adolescents was retrieved from the patient files at the low vision services organizations; missing values were complemented with self-reported data from parents (n = 8). Visual acuity was classified using five levels based on the better seeing eye, according the criteria of the WHO [17].

The PAI-CY 13–17

The preliminary version of the PAI-CY 13–17 contained 58 items grouped into eight domains (for descriptive purposes only, in order to provide contextual meaning) that were informed by concept-mapping workshops with end-users (see Table 2) [13]. Items were scored on a 4-point Likert scale with response options not difficult (1), slightly difficult (2), very difficult (3), and impossible (4). The response option not applicable was handled as a missing value. The items in the proxy version only differed in first or third person tense.

Statistical analyses

As a first step in the psychometric evaluation of the PAI-CY 13–17, item reduction was performed conservatively, using lenient criteria. Items were considered for elimination if the psychometric performance was suboptimal on several criteria and if item content was already adequately represented in other items: a) Items with missing scores > 20% [8, 16] in both the adolescent and parent version and/or items with > 70% [8, 18, 19] of both adolescents and parents endorsing the first or last response category (i.e. floor or ceiling effects). b) Items showing inter-item correlations > 0.8, indicating potential redundancy. c) Suboptimal values of test-retest reliability [16] using weighted kappa and percentage agreement [20, 21]. The intraclass correlation coefficient (ICC) for sum scores of the test and retest data were calculated based on absolute agreement, with a two-way mixed-effects model.

Subsequently, adolescent-parent agreement at item level was investigated using weighted kappa and percentage agreement. The ICC for the sum scores of the adolescent and parent data were calculated based on absolute agreement, with a two-way random-effects model. Comments and suggestions evolving from the evaluation forms were used to inform decisions on item elimination or instrument adaptation.

Known-group validity on the final item-set [16] was investigated using independent samples t-tests (median split for age) and ANOVAs with a post-hoc Tukey test. Lastly, concurrent validity [16] with subscales of the Kidscreen, CASP and FVQ_CYP_NL was investigated with Spearman’s correlations. Negative correlations were expected between the PAI-CY and subscales of the Kidscreen and CASP, whereas positive correlations were expected with the FVQ_CYP_NL. Correlations were expected to be smallest between the PAI-CY and the Kidscreen and largest for the FVQ_CYP_NL.

Results

Of an estimated 700 invited adolescents and their parents, 77 provided written informed consent and completed the first questionnaire (either the adolescent or parent version). Main reasons for non-participation were no time, not interested or adolescents not willing to participate. Sociodemographic and clinical characteristics are presented in Table 1. Complete data was obtained from 72 adolescent-parent dyads. The retest was completed by 64 parents after 28.3 ± 24.3 (range 9–121, median 18) days and 74 adolescents after 15.9 ± 7.0 (range 7–44, median 14) days.

Table 1 Socio-demographic and clinical characteristics of participants

Twelve items had missing responses > 20% in either the adolescent or parent version, of which eight items had > 20% missing scores in both versions (Table 2). Floor effects were found in 39 items in either the adolescent or parent version, of which 13 items displayed floor effects in both versions. Inter-item correlations of > 0.8 were found in one item pair for adolescents and seven item pairs for parents. Most items showed satisfactory test-retest reliability. For three items agreement was < 60%, whereas for eight items weighted kappa was < 0.4. ICC for the sum score on test and retest for adolescents was 0.866 (95%-confidence interval: 0.791–0.915). For parents, the ICC was 0.882 (95%-confidence interval: 0.815–0.927). With respect to adolescent-parent agreement, 34% of the items showed agreement < 60%, while 79% of the items showed weighted kappa’s ≤ 0.4. ICC between adolescent and parent scores was 0.438 (95%-confidence interval: 0.000–0.694).

Table 2 Distribution of responses and parameters of test-retest reliability and adolescent-parent agreement

The evaluation forms showed that parents’ self-reported administration time (including questions about sociodemographic and clinical characteristics) was 22 ± 14 (range 8–60, median 17) minutes. Over 90% of both adolescents and parents were neutral to very positive regarding various feasibility aspects of the PAI-CY (Fig. 1). However, comments and suggestions for improvements were made by 42 adolescents and/or parents (Table 3).

Fig. 1

Evaluation of the PAI-CY 13–17 by adolescents and parents

Table 3 Comments of > 1 respondents relevant to the PAI-CY 13–17 and suggested solutions

From these results, it was decided to eliminate items LT8, MO2, and CO4, resulting in a PAI-CY 13–17 containing 55 items. Infrequent endorsement of the answer category ‘impossible’ motivated collapsing this category with the category ‘very difficult’. Table 4 shows expected patterns of correlations between the PAI-CY and concurrent questionnaires.

Table 4 Correlation coefficients of the PAI-CY 13–17 with the CASP, Kidscreen and FVQ_CYP_NL

Adolescents with comorbidity scored significantly worse on the PAI-CY 13–17 than adolescents without comorbidity, both in the adolescent and parent version (effect sizes were moderate to large). In the adolescent version, significantly worse scores were found for adolescents with blindness compared to those with moderate VI and no VI (large effect sizes). All other effect sizes were small (Fig. 2).

Fig. 2

Boxplots for groups that differ on gender, comorbidity, age and degree of VI. Box width represents group sizes. ES: effect size; VI: visual impairment

Discussion

In this study the psychometric properties of the PAI-CY 13–17 were evaluated, a questionnaire to assess the needs of Dutch adolescents with VI and their parents. We used less stringent criteria for item elimination because of the importance of face and content validity and the small sample size. Therefore, we only eliminated three items, of which content was thought to be adequately represented in the remaining items (LT8: making music can be captured with LT9: performing a hobby, MO2: cycling to something independently with MO1: cycling and CO4: participating in a conversation actively with CO1: Being able to express in words properly). Eliminating more items while having a small sample size could be counterproductive in the long term, causing potentially informative items to be discarded before larger samples are available. Once the PAI-CY 13–17 is used in future, confirmation of the findings and the assessment of other psychometric properties should inform further item reduction, although the current number of items is in line with other age-versions of the PAI-CY and PAI-YA, containing 27–60 items [22,23,24,25]. Moreover, with larger samples item response theory modeling might be employed to make the instrument more precise and user-friendly.

Test-retest reliability was satisfactory for most items. Exclusion of participants who completed the retest > 45 days after the first questionnaire (n = 10) in general had a positive influence on kappa, agreement and the ICC (data not shown). Remarkably, the items with suboptimal kappa values showed adequate agreement. This phenomenon has previously been reported as the ‘paradox’ of the kappa statistic, in which low values of kappa can be found for high values of agreement [26, 27], caused by symmetrically imbalanced contingency tables [28]. For example, for item LT7: participating at a club, many adolescents opted “not difficult” on the test and retest, resulting in a high proportion of adolescents (0.98) in row 1 in the contingency table. As the proportion of respondents in row 1 increases, kappa decreases rapidly and may even fall below zero, as has happened for this item, despite excellent agreement. Therefore, agreement and weighted kappa should be re-examined with larger samples. The ICC between the sum scores of the test and retest data were high for both adolescents and parents, indicating good reliability. Adolescent-parent agreement and weighted kappa was low for many items, as was the ICC between the sum scores, indicating that adolescents and their parents have different perceptions about the difficulties adolescents experience. In general, adolescents perceive items as less difficult than parents. Notably, agreement was highest for those items taking place at home, i.e. items in the self-reliance domain. This might indicate that for activities taking place elsewhere, parents might be less aware about the difficulties experienced by their child, and therefore overestimate the difficulties experienced by adolescents. However, due to their age, adolescents also might overestimate their abilities and perceive less risks. Therefore, retrieving information from both sources is important to comprehensively assess adolescents’ needs.

It was possible to differentiate between adolescents with or without comorbidity, both with the adolescent and parent version. Furthermore, it was possible to differentiate between different degrees of VI with the adolescent version, while there was a trend with the parent version. The correlations with other instruments demonstrated concurrent validity of the PAI-CY 13–17. The relatively low correlation with subscales of the Kidscreen demonstrates that the PAI-CY 13–17 is not necessarily measuring quality of life, whereas the strong correlations with (subscales) of the CASP and the FVQ_CYP_NL confirms that the PAI-CY 13–17 rather measures a construct related to participation or functional vision. Although one can argue that the PAI-CY and FVQ_CYP_NL might be similar, the relatively strong but not perfect correlation shows that both instruments measure a similar construct, but each in its unique way.

One of the major limitations of the current study is the small sample size and the low response rate, although anticipated from previous studies involving similar populations. Main reasons for non-participation according to parents were no time, not interested or their child stating they did not want to participate. Adolescents might be reluctant to participate in scientific research, especially if they have to make an appointment and are visited by a researcher in their own homes.

The small sample size prevented factor analysis and the use of item response theory to optimize the PAI-CY 13–17. However, previous studies examining other age-versions of the PAI-CY and the PAI-YA showed the items comprised a unidimensional scale [22,23,24]. As such, the PAI-CY 13–17 is expected to be unidimensional as well, and we calculated sum scores for the complete scale.

The current study used a national sample, implying that the PAI-CY 13–17 should be applicable across the Dutch population. Although the PAI-CY 13–17 can be used as a template for use in other countries, cross-cultural validation is recommended for use outside the Netherlands. Recent interest to use various age-versions of the PAI-CY and the PAI-YA in Nepal and Australia, has resulted in official forward-backward translations of the instruments and manuals (available upon request from the authors) to English and Nepali, in which cultural applicability of items was also taken into account. Further research should indicate whether the PAI-CY 13–17 is indeed applicable and valid in these countries.

Conclusions

Based on initial psychometric tests, the PAI-CY 13–17 appears to have acceptable measurement properties, although more work with larger samples is needed, for example to assess whether the items comprise a unidimensional scale. It is possible to implement the current version of the PAI-CY 13–17 in Dutch low vision services, enabling more data collection and more extensive understanding of its psychometric properties. Acceptability of the PAI-CY 13–17 to end-users should be carefully monitored, especially if the changes suggested in this study are going to be incorporated.