Skip to main content

PROMs and PREMs in routine perinatal care: mixed methods evaluation of their implementation into integrated obstetric care networks



In the transition towards value-based healthcare, patient-reported outcome and experience measures (PROM and PREM) are recommended by international collaborations and government programs to guide clinical practice and quality improvement. For many conditions, using PROM/PREM over the complete continuum of care requires implementation across care organizations and disciplines. Along PROM/PREM implementation in obstetric care networks (OCN), we aimed to evaluate implementation outcomes and the processes influencing these outcomes in the complex context of care networks across the continuum of perinatal care.


Three OCN in the Netherlands implemented PROM/PREM in routine practice, using an internationally developed outcomes set with care professionals and patient advocates. Their aim was to use PROM/PREM results individually to guide patient-specific care decisions and at group-level to improve quality of care. The implementation process was designed following the principles of action research: iteratively planning implementation, action, data generation and reflection to refine subsequent actions, involving both researchers and care professionals. During the one-year implementation period in each OCN, implementation outcomes and processes were evaluated in this mixed-methods study. Data generation (including observation, surveys and focus groups) and analysis were guided by two theoretical implementation frameworks: the Normalization Process Theory and Proctor’s taxonomy for implementation outcomes. Qualitative findings were supplemented with survey data to solidify findings in a broader group of care professionals.


Care professionals in OCN found the use of PROM/PREM acceptable and appropriate, recognized their benefits and felt facilitated in their patient-centered goals and vision. However, feasibility for daily practice was low, mainly due to IT issues and time constraints. Hence PROM/PREM implementation did not sustain, but strategies for future PROM/PREM implementation were formulated in all OCN. Processes contributing positively to implementation outcomes were internalization (understand the value) and initiation (driven by key-participants), whereas challenges in relational integration (maintain confidence) and reconfiguration (refine activities) affected implementation negatively.


Although implementation did not sustain, network-broad PROM/PREM use in clinic and quality improvement matched professionals’ motivation. This study provides recommendations to implement PROM/PREM meaningfully in practice in ways that support professionals in their drive towards patient-centered care. In order for PROM/PREM to fulfill their potential for value-based healthcare, our work highlights the need for sustainable IT infrastructures, as well as an iterative approach to refine their complex implementation into local contexts.


In the past decade, the discourse of value-based healthcare (VBHC) has had an immense uptake in healthcare [1]. At system level, healthcare systems strive to use patients’ well-being to evaluate care performance for full treatment cycles for a condition [2]. At patient level, professionals aim to organize integrated care around a health condition and make personal values prescriptive to guide treatment decisions [3]. In the development towards VBHC, patient-reported outcomes and experiences measures (PROM and PREM) have been embraced to generate data about what matters to patients and drive patient-centered quality improvement (QI) [4]. Therefore, the capture and use of PROM/PREM has been encouraged in many healthcare settings by international collaborations and government programs [5, 6]. Nevertheless, PROM/PREM implementation remains challenging, especially in network settings like perinatal care [7, 8].

PROM/PREM implementation has been considerably studied with an implementation science approach, identifying common influencing factors such as technology and clinical leadership [9, 10]. Different challenges have been described dependent on the purpose of PROM/PREM implementation. For example, a challenge for individual-level use includes fitting PROM collection to appointment schedules, while at group-level motivating care professionals for (external) QI appears more challenging [8, 9]. Yet most implementation factors have been explored in single organization settings or primary care predominantly [11, 12], whereas the majority of health conditions require interdisciplinary and interorganizational collaboration across healthcare tiers to provide the full continuum of care [13, 14]. As for pregnancy and childbirth, where care professionals collaborate both interdisciplinary (e.g., obstetrics, neonatology) and interorganizational (e.g., hospitals, midwife practices, youth care) to provide acute and long-term care with in-hospital, outpatient and community-based care and support. Thus, to contribute to patient- and family-centered care, PROM/PREM in perinatal care would ideally be implemented across care networks, to cover patients’ whole care trajectory in individual-level use and involve all stakeholders in group-level use for QI. Yet, implementation in network context prompts other challenges, like engaging diverse stakeholders, aligning incentives and resources, and building common infrastructures [4, 15]. Evaluations of individual-level PROM/PREM implementation in network context are scarce, but needed to advance our understanding of practice challenges, contextual factors, and mechanisms through which implementation strategies work across organizations [10, 16].

For perinatal care, until recently, no consensus on PROM/PREM had been formed to evaluate its patient outcomes [17]. Yet, in 2017, a set of standardized patient-centered outcomes measures for pregnancy and childbirth (PCB set) was developed internationally with perinatal care professionals and patient advocates [17, 18]. This set includes PROM/PREM from beginning of pregnancy until six months postpartum. Over the last years, the PCB set has been adopted internationally and implementation efforts have been started worldwide, of which most are in research context [19,20,21]. Potential factors influencing PCB set adoption in practice have been explored in pre-implementation analyses, indicating all stakeholders recognized the relevance and potential benefits of PROM/PREM [8, 22]. At the same time, stakeholders acknowledged important efforts yet to be made, e.g., embedding PROM/PREM into service processes or informing care professionals and patients about their purpose.

Recently, the patient-reported measures of the PCB set were implemented in three obstetric care networks (OCN) in the Netherlands, that aimed to use these PROM/PREM for two levels of VHBC: individual scores to guide patient-specific care decisions and group-level results in to improve quality of care. This implementation process was designed following the principles of action research to enhance practice change and, concurrently, gain knowledge about PROM/PREM implementation in the context of care networks. Guided by theoretical frameworks for implementation, this study aimed to evaluate (1) the outcomes of PROM/PREM implementation in obstetric care networks and (2) the implementation processes that influence these outcomes to increase our understanding of this complex implementation, its practice challenges, and underlying change mechanisms.



This mixed-methods study was conducted between December 2019 and June 2022 as part of an action research project aimed at PROM/PREM implementation in clinical practice and QI processes of OCN. Action research aims to both change practice and develop knowledge about that change via a cyclic design of action, data generation and reflection, while involving all stakeholders in research and practice change [23]. Action research is particularly useful to implement a complex intervention that needs adjustment to the local context, as detailed data are generated on both the implementation activities (what it involved) and change mechanisms (how it worked). This way, the outcomes achieved can be explained for, increasing the transferability of findings [24]. To understand the change mechanisms underlying the complex implementation of PROM/PREM, the use of multilevel implementation frameworks and theories has been recommended by scoping literature [10, 25]. To evaluate PROM/PREM implementation in the context of care networks, this study combinedly used Proctor’s taxonomy for implementation outcomes [26] and the Normalization Process Theory (NPT) [27]. Proctor’s taxonomy describes the outcomes of different stages in implementation, whereas the NPT describes implementation processes in terms of what care professionals (don’t) do to embed a new way of working in routine practice and is distinct in proposing mechanisms for sustained uptake. Proctor and NPT guided the collection and analysis of both qualitative and quantitative data within the mixed-methods design, increasing both the depth and transferability of our findings.

Setting and participants

Dutch perinatal care is provided interdisciplinary from two healthcare tiers: primary care by community midwives and maternity care organizations; and secondary/tertiary care by hospital employed care professionals. Hospitals, regional community midwife practices and maternity care organizations increasingly cooperate in OCN to provide continuity of care across pregnancy, childbirth and puerperium. In 2019, PROM/PREM implementation was initiated from a regional collaborative between ten OCN in the middle of the Netherlands, of which three OCN participated. In each OCN, the hospital and 2–4 midwifery practices implemented individual-level PROM/PREM in clinic. All other professionals working in the OCN (e.g., from other midwifery practices, maternity care organizations, youth care) could join network-broad QI with group-level outcomes. Each OCN had an interdisciplinary team in charge of implementation (including, at least one obstetrician, clinical midwife, and community midwife from each participating midwifery practice), of which one was appointed project leader. In this study, participants were defined as (1) professionals directly involved in implementation: project team members (key participants) or obstetricians/midwives using individual-level PROM/PREM, and (2) indirectly involved professionals: from other OCN-organizations or discipline, such as nurses. Patients were involved in implementation as they completed PROM/PREM for routine care but did not actively participate in this evaluation study. As patients had participated in our pre-implementation analysis and feasibility pilot [8, 28], their needs were incorporated in the initial implementation strategy.

Action research project

The PROM/PREM implemented in this project were those proposed in the PCB set: questionnaires at two moments during pregnancy (T1: first trimester, T2: early third trimester) and three postpartum (T3: maternity week, T4: 6 weeks postpartum, T5: 6 months postpartum). The PCB set was developed internationally and subsequently translated to the Dutch setting, both phases involving all stakeholders, including care professionals and patients [18, 29]. An overview of the PCB set’s patient-reported domains and timeline for completion is provided in Additional file 1: Fig. S1. The set’s PROM/PREM were implemented for two purposes. First, individual-level PROM/PREM were implemented in clinic: reviewing N = 1 scores with patients during a regular care contact after completing a questionnaire. The timeline of collection, workflow, and follow-up services (including scoring and alert values) were organized as described in the national pilot project [30]. Second, the same PROM/PREM outcomes would be used at group-level in network-broad QI sessions. Despite the complexity of combining these purposes, findings in our pre-implementation research amongst care professionals, patients and other stakeholders in perinatal care suggested both goals could also reinforce each other [8]. Direct usability in clinical practice could, for instance, motivate care professionals and patients to comply, thereby generating data for group-level use (and vice-versa). Likewise, other previous findings from our pre-implementation analysis and feasibility pilot [8, 28], were used to design the initial implementation strategy. Important elements for individual-level use included visual alerts to support care professionals in interpreting the answers and offering patients a choice whether their care professional had insight in their individual PREM answers. During the action research project, this initial implementation strategy (Fig. 1) was continuously refined guided by action research principles in iterative cycles of planning and executing implementation activities, data generation, and reflection on these data to refine subsequent activities. These cycles were conducted jointly by researchers and care professionals. The researchers developed the baseline strategy for project organization and education (e.g. identified possible IT-systems, developed an e-learning and kick-off meeting), provided materials and support for its execution (e.g. patient information folder, for working protocol for care professionals), and facilitated data generation for its refinement (e.g. organized focus groups, sent out the survey). The project teams designed and coordinated local implementation (e.g. adapt instruction material to local workflow, chose the IT system that best fitted local needs and resources) and participated in data generation and reflections (e.g. survey results were discussed in project team meetings, participation in focus groups). Three OCN started implementation sequentially to be able to learn from previous experiences, exchanged via the researchers and directly between care professionals from different OCN. After the one-year implementation period, project teams reported their experiences to their OCN and advised future steps in an end-evaluation.

Fig. 1
figure 1

Timeline of implementation and data generation activities. PROM, patient-reported outcome measure. PREM, patient-reported experience measure. QI, quality improvement. OCN, obstetric care network. CP, care professional. VHBC, value-based healthcare

Outcome measures

First, implementation outcomes were assessed using Proctor’s taxonomy of implementation outcomes. Inspired by the translation to PROM/PREM specific implementation outcomes by Stover et al. [10], implementation outcomes and the indicators to assess them were defined for this study’s context (Table 1). These indicators were evaluated with survey items of the Measurement Instrument for Determinants of Innovations (MIDI), via administrative data and embedded in qualitative methods such as observation checklists. The MIDI was developed to identify factors influencing the use of an implemented intervention by measuring determinants in innovation, user, organization and socio-political context [31]. As recommended by its developers, a selection of items was made based on relevance for our context. Second, implementation processes were evaluated along the NPT, which describes four core mechanisms towards normalization. These mechanisms and their subconstructs were measured trough the validated Normalization Measurement Development (NoMAD) instrument [32, 33], and were included in the survey and qualitative methods (Additional file 1: Table S1). The complete survey administered to care professionals consisted of validated NoMAD and MIDI items, completed with three extra questions (about education used, knowledge level, needs in implementation) based on our feasibility pilot and PROM/PREM specific implementation literature [28, 34]. All survey questions and details about scoring are provided in Additional file 1: Table S2.

Table 1 Implementation outcomes and their assessment

Data generation

A timeline of data generation is provided in Fig. 1. For quantitative data, the survey was sent to all OCN care professionals at the start and end of implementation by e-mail. Care professionals indirectly involved in implementation were led to a short version. Demographics were collected on profession and working experience. This way, the survey explored implementation processes in a broad group of care professionals, which was used to solidify qualitative findings and to guide reflection on the implementation process and needs with participants during qualitative methods. Qualitative data were generated through focus group discussions, observations, reflections and naturally occurring data. At each kick-off session, group discussion was organized using photovoice (i.e. a method to empower all participants to share their perspectives [35]), of which notes were taken for the observation report. Along the QI sessions, traditional focus group discussions were led by two researchers (AD, AK) along statements about implementation based on outcome indicators and NPT subconstructs (Additional file 1: Table S3). For each focus group, a selection of these statements was made to address specific gaps in data generation emerging from collective iterative reflections and quantitative results from the survey. After informed consent, focus groups were recorded and transcribed ad verbatim. During the whole implementation, two researchers (AD, ML) conducted participative observations in all meetings and kept a reflection logbook, both structured along the theoretical frameworks. Considered as naturally occurring data [36], all documents emerging during the implementation process were gathered (e.g., meeting reports), containing administrative data too (e.g., IT system data on costs, professionals with account).

Data analysis

Quantitative survey data were analyzed in R version 4.0.2 [37]. Mean scores were calculated for items consisting of multiple statements and multiple items measuring a subconstruct. Frequencies of responses to items were visualized in stacked-bar diagrams to gain insights in the diversity of opinions. All qualitative data (i.e., open-ended survey answers, transcriptions, observation reports, reflection logbook, documents) were thematically analyzed in Microsoft Excel version 16.61 conform QUAGOL guidelines, combining a deductive and inductive approach [38]. The researchers assigned codes from the conceptual frameworks (Proctor and NPT) as well as open codes describing themes within their concepts. At start, three researchers (AD, BP, ML) coded three documents independently, and discussed the resulting codes to develop a mature coding scheme. Data were then analyzed by AD until saturation was reached, after which four researchers (AD, ML, BP, MB) reviewed and discussed the codes to establish final interpretations. Quantitative and qualitative data were then triangulated by exploring (dis)agreements and silences between both datasets. This was conducted by a single researcher (AD) identifying items and subconstructs in the quantitative dataset demonstrating particularly high or low survey scores, to compare these against qualitative themes and discuss that among the research team. In this process, quantitative data were used to solidify quantitative findings in a broader group of professionals and over time.


Overall, 159 surveys were returned, of which 63 (39%) in phase 1 and 97 (61%) in phase 3. Five focus groups were held with, in total, 78 care professionals attending QI sessions. Other data (from observations, reflections, documents) were generated along 39 project team meetings, 3 kick-off sessions, 5 QI sessions, and the logbook. Participants’ characteristics for the survey and focus groups are presented in Table 2. Of survey respondents, 62% (99/159) was directly involved in implementation (i.e., project team member or using individual-level PROM/PREM). Mean survey scores were largely in agreement with qualitative themes, thus strengthening each other, and are together presented per theoretical framework below. Full response frequencies per survey item are provided in Additional file 1: Fig. S2.

Table 2 Baseline characteristics survey and focus group participants

Implementation outcomes

Below, Proctor’s outcomes as defined in Table 1 are provided along our most important findings.

Motivations and objectives

At the start, potential benefits of PROM/PREM were recognized by most care professionals, contributing to acceptability and adoption. Care professionals expected that individual-level PROM/PREM would assist them in recognizing symptoms and identifying topics important to their patient and empower patients to prepare visits and raise issues. Moreover, care professionals expressed enthusiasm for using group-level PROM/PREM for patient-centered quality improvement. Patients’ opinions were care professionals’ main motivation to comply and 54% (46/85) of survey respondents expected their cooperation, whereas 11% (9/85) did not.

Experienced benefits

According to care professionals, system-wide PROM/PREM capture and use facilitated their patient-centered goals and vision, expressing good appropriateness. In consultations, several care professionals felt supported by PROM/PREM results to identify and discuss patients’ issues, sometimes leading to richer conversations and/or appropriate referrals. From the group-level PROM/PREM data in QI sessions, care professionals gained valuable insights and directions for improvement in their patients’ wellbeing and experiences, which contributed to their work pleasure. In practice, the PROM/PREM content was considered appropriate for most of their patients, except for non-Dutch speaking women and those with low health literacy, who care professionals hesitated to invite for that reason. Also, some adaptations to PROM/PREM content were suggested, such as open answer options to enable personalized care even more.

Experienced barriers

Whilst most care professionals strongly favored integration in their electronic health record (EHR), the locally explored IT options either could not function across different EHRs, or their costs to realize that were too high. Hence, in each OCN, an affordable start-up IT system without EHR integration was chosen, that promised automated data capture, visualization for care professionals, network communication and privacy. This IT system enabled PROM/PREM adoption in all participating practices per OCN but became the main barrier for acceptability, feasibility, and further implementation. Care professionals did not consider it to be user-friendly (complicated access, frequent issues and bugs, poor overview, not visible whether responses had been discussed and unable to connect PROM/PREM measurements to visits) and felt increasingly frustrated by the IT supplier’s slow pace, and sometimes inability, to solve issues. Although their patients often appeared willing to complete PROM/PREM, IT was considered a major barrier for patients too, due to poor accessibility and bugs, leading to privacy concerns too. Other patient barriers mentioned were a lack of motivation or time (especially postpartum) and misunderstanding of the purpose.

Additionally, the high time investment for care professionals negatively influenced acceptability (44% of care professionals (60/135) expected it would take too much time), appropriateness (for their high current workload) and feasibility (of workflow integration). Factors contributing to a high time investment in practice included the administrative burden of the non-integrated IT system, instructing patients, reviewing PROM/PREM results, and learning a new skill.


The IT systems’ costs and care professionals’ time investment (i.e., project team efforts and using PROM/PREM in practice) were the main drivers for implementation costs. In two of three OCN, these costs demanded external funding (used for the IT system and project leader allocation); the third OCN could finance them from a joint reimbursement structure.

Fidelity and penetration

The process of creating an account for the external IT system, inviting patients, and discussing individual PROM/PREM responses required continuous support from project teams and action researchers to reach fidelity and penetration in participating practices. At start, project teams decided to begin with a selection of patient groups, measurement moments (all selected T1–T4) and care professionals. Eventually, most targeted care professionals created an IT system account, but only few actively invited patients: others often missed eligible patients due to time constraints and low exposure resulting from the patient group selection. Half of the invited patients created an account and completed PROM/PREM; postpartum response rates were lower. Based on experiences shared in project team meetings, almost all completed PROM/PREM were discussed in the next visit, except in case of IT bugs or care transitions in the maternity week (T3). Regarding group-level PROM/PREM use, five QI sessions were carried out during the pilot periods. Reflecting good fidelity, local care professionals actively participated in preparation, presentation, and elaboration of these sessions, which were attended by an average of 17 (range 11–25) care professionals representing all participating disciplines. Unlike in-clinic PROM/PREM use, QI sessions extended penetration to care professionals without direct involvement in implementation.


Except for one community midwifery practice that sustainably integrated PROM/PROM, routine PROM/PREM administration was stopped in all OCN after the one-year implementation period. However, all OCN intended to continue the QI sessions with data available in the OCN and, after EHR integration, reinitiate PROM/PREM capture and use. After the decision to stop, the second QI session in one OCN was not conducted, because the project team expected it would be of more benefit to a future restart.

Implementation process

The complete NPT framework analysis is listed in Table 3 with supportive qualitative and quantitative data (mean survey scores on a 5-point Likert scale) per subconstruct. Per core mechanism, subconstructs contributing most to (un)successful implementation outcomes are elaborated on below. Overall, main processes contributing to implementation positively were internalization (understand value) and initiation (drive by key-participants), whereas relational integration (maintain confidence) and reconfiguration (attempts to redefine) affected implementation negatively.

Table 3 Framework analysis for the NPT subconstructs with supportive data

Coherence: sense-making

As terminology like PROM/PREM and VBHC often appeared abstract at the start, hearing experiences directly from participants of earlier regions helped to gain understanding of practical aspects. This was arranged both across and within OCN enhancing differentiation and individual specification. Care professionals early recognizing the potential benefits of PROM/PREM contributed to internalization and willingness for implementation. The ability to incorporate patients’ voice in QI appeared their main driver, so they were enthused by the QI sessions. Although some experienced that individual-level PROM/PREM supported time-efficiency and personalized care by discussing important rather than all topics, care professionals felt they needed more exposure to these benefits for sustained internalization.

Cognitive participation: relational work

Formally appointed local project leaders mainly drove initiation, particularly if this was a clinician from a participating practice with OCN management support (both in resources and vision). Project team members representing each participating practice and discipline could engage colleagues, reflect on practical challenges, and establish possible solutions. Initiation by key-participants was facilitated by action researchers’ activities (e.g., share experiences and materials, participate in identifying and solving issues, practical support) and by the one-year implementation period, making them feel able to try PROM/PREM without being ‘stuck’ to them. Whether local key-participants drove initiation or relied on the action researcher, depended on the level of ownership felt by local project teams. In-clinic support from key-participants and action researchers was most important for enrolment of other care professionals, since training reached a minority: 22% (22/99) of survey respondents had used support or training. Enrolment was harder in large practices, as care professionals felt less influence on the decision (or had little knowledge of the reasons) to participate. Care professionals differed in their feeling of PROM/PREM being a legitimate part of their role, which could be supported by positive practice experiences or those of colleagues. Additionally, enrolment and legitimation appeared to improve by the QI sessions, where valuable interprofessional conversations led to concrete improvement actions. However, care professionals’ positive expectations and involvement decreased over time by enduring IT issues and low exposure to benefits. At the end, (key) participants kept support for the potentials of PROM/PREM for VBHC and formulated future strategies for sustainable activation.

Collective action: operational work

Discussed in 92% of project team meetings (36/39), feasibility issues dominated the implementation process and impaired workflow integration (i.e., interactional workability). Key-participants’ and action researchers’ time and efforts mainly went into getting the IT system working and supporting users (care professionals and patients) in operational work. Project teams experienced a vicious circle of poor-usable IT and not building up workflow routine: their attempts to increase routine, like expanding patient groups, were withheld by IT issues and concurrent time investment. The IT system affected participants’ confidence in the innovation (i.e., relational integration), especially the inability to improve or solve issues in time. Also, reliability of PROM/PREM results was questioned, because care professionals experienced varying clinical relevance of alerts, inappropriate timing, unsuitable answer options and, at group-level, numbers were too small. Most care professionals expressed confidence about discussing PROM/PREM, but the challenging part of skill set workability was allocating all tasks appropriately, for example ensuring that individual-level PROM/PREM were discussed across participating practices. To solve this, allocating a principal care provider to discuss PROM/PREM was opted by care professionals, both to keep overview of which responses had been discussed, as to gain most value from that conversation in a trusted relationship.

Reflexive monitoring: appraisal work

Facilitated by action researchers, project teams continuously reflected on (systemization) and tried to refine (reconfiguration) processes to improve implementation, like standard phrases to report PROM/PREM conversations to decrease administration burden. Reconfiguration was easier for smaller practices, such as temporarily collect T3 (maternity week) on paper to increase response rates. However, limited adaptability was experienced for several reasons: IT suppliers’ inability to improve, time constraints and the PCB set’s international origin. Key-participants’ evaluation reports stated reconfigurations needed for future restart and sustained implementation. For individual use, PROM/PREM should be easily accessible for patients and professionals, with EHR-integration across the network. For QI with group-level data, essential aspects were data analysis and visualization (provided by the researchers during the action research project) and linking PROM/PREM to clinical outcomes.


In this mixed-method evaluation of PROM/PREM implementation in the context of care networks, the use of PROM/PREM was found to be acceptable and appropriate but not feasible in daily practice, mainly due to IT issues and time constraints. Hence PROM/PREM implementation did not sustain, but their potentials for VBHC fitted professionals’ motivation and strategies for their future adoption were formulated in all OCN. In line with previous evidence [10, 11], our findings affirm the value of individual-level PROM/PREM for clinical care perceived by professionals and emphasize the need for workflow integration. Based on participants’ and researchers’ reflections on the re-adjusted, co-created implementation strategy, recommendations for PROM/PREM implementation across care networks were formulated in end-evaluations and summarized in Table 4. To embed these recommendations, an iterative approach is key to adjust to local context.

Table 4 Recommendations for PROM/PREM implementation across care networks

Despite tailoring the strategy to our pre-implementation analysis amongst patients and care professionals and further adaption of implementation activities during each action research cycle [8, 28], the feasibility of integrating PROM in practice was lower than expected, largely explainable by poor usability of the IT system chosen at start. Of the numerous PROM/PREM capture systems developed in the past years, most were designed for single center settings or group-level, anonymous use only [11, 39, 40]. Besides healthcare systems with a shared EHR [41], successful system-wide PROM collection with direct visualization for individual-level use in clinic has proven challenging to realize and was only recently described and developed in a Welsh national program [42]. To support PROM/PREM implementation and network collaboration for patient-centered care, there is a need for PROM/PREM integration into EHRs and, moreover, infrastructures for cross-EHR data exchange [43]. Structural financial support for their development and governance should be explored, as most network collaborations are temporarily funded which undermines adoption, feasibility, and sustainability [4, 11, 44].

Previous PROM/PREM implementation strategies, both at the individual and group level, often emphasize the selection of PROMs and the challenge of involving care professionals [5, 11, 45]. Although we acknowledge their importance, most care professionals in our study already demonstrated a positive attitude towards PROM/PREM at start, reflected in good coherence and cognitive participation and consistent with previous findings [22, 41]. They were keen to learn from previous experiences and motivated by the prospective of patient-centered QI with group-level PROM/PREM, which fueled their efforts for individual-level implementation as well. In the current healthcare landscape with professional shortage and high turnover, care professionals’ work pleasure might be one of the most valuable benefits of PROM/PREM [46, 47]. Despite feasibility challenges and IT issues, key participants’ threshold to adopt such complex implementation was lowered by the iterative approach that gave space to ‘try out’ and adapt to local context, which enabled them to get acquainted with PROM/PREM and their potential for VBHC. Concurrently, other care professionals felt demotivated and overruled by management when unaware of the reasons to participate in such implementation and driving their workload even higher. So new initiatives should be carefully selected and coordinated across care networks, where an iterative and participative approach to implementation can provide space for early adopters’ energy, sharing practice experiences to engage others, and fine-tuning to local context.

The integrated care context affected implementation not only by challenges in IT infrastructure, fragmented leadership and allocation of costs, but also in consistency of discussing individual-level PROM/PREM results across care transitions. To ensure that individual-level results were discussed, care professionals opted to allocate a principal care provider, arguing that a conversation about the topics would gain most value in a trusted relationship, similar to a solution to improve continuity of perinatal care in general [48]. However, the issues arising from network-broad implementation are lacking in current PROM/PREM implementation frameworks and strategies [9, 10]. Further research within real-life projects should identify and address barriers and enablers for innovation across organizational boundaries. That way, innovations can improve value of care for individuals and overall care performance from patients’ perspective.

Reflecting on the action researchers’ role, many similarities were seen with the facilitator role described by Roberts in the iPHARIS framework [49]. Similar to their findings, our action researcher was a crucial enabler for implementation, providing an external view with expert knowledge to identify and solve emerging issues in practice, especially in collaboration with the local project leader. Additionally, participating in all regions resulted in overview, expertise and sharing previous lessons in new regions. However, the tension between guidance in problem solving and doing the work to fit local workflow was present in our projects as well: in some regions, the PROM/PREM workflow never became completely independent of the action researcher. Across OCN, the level of implementation ownership of the project teams varied, which could partly be explained by existing collaboration mechanisms and integrated reimbursement in some OCN.

As called for in recent literature [10, 16], this study substantially contributes to the understanding of care professionals’ real-life experiences and challenges for PROM/PREM implementation, specifically addressing the integrated care context in a realistic range of collaborating organizations. In the mixed-methods design, consistency in data from different sources and methods strengthened our findings. Also, our data collection and analyses were supported by widely used implementation science theories and their validated instruments. The iterative, participatory action research approach enabled in-depth understanding of implementation activities, processes and outcomes, which contributes to the transferability of findings. An important limitation of our study was that we did not invite patients to the evaluation of the implementation process and outcomes, except indirectly via care professionals. We did explore patients’ experiences with individual-level use in another study along a national pilot with the PCB set [50], while the current project focused on the (organizational challenges of) implementation. In next action cycles, patients should be certainly involved. Here, special attention should go to women with low health-literacy and language barriers, who are prone to be neglected by PROM/PREM, to prevent existing health inequities becoming even larger [51]. Besides providing digital support and translating questionnaires, solutions to involve these women should be sought outside the idea of questionnaire completion. In thinking of solutions, research methods should be embraced that centralize patients and local opportunities (e.g. linkage to primary care, community-based solutions) [52, 53]. Another limitation of our study is that the IT-system used appeared such a major barrier to implementation, that other factors might have been undervalued. Selection bias of both early adopter OCN and professionals is likely to have enhanced a positive attitude towards PROM/PREM. We attempted to reach professionals broader by inviting the whole OCN for QI sessions and the survey, which had a short version for indirectly involved professionals. Lastly, the COVID-19 outbreak has probably influenced care professionals’ willingness and ability to adopt a new way of working, affected implementation planning (e.g., paused, postponed) and restricted study activities to online contacts with minimal field work.


Although implementation did not sustain, network-broad PROM/PREM use in clinic and for QI matched professionals’ motivation for patient-centered care. This study provides recommendations to implement PROM/PREM meaningfully in practice, in ways that support professionals in their drive towards patient-centered care by efficient, person-centered assessment of patients’ wellbeing. For PROM/PREM to fulfill their potential for VBHC, our work highlights the need for sustainably funded technology infrastructures that communicate across healthcare tiers, as well as an iterative and participative approach to refine their complex implementation to local contexts.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.



International Consortium for Health Outcomes Measurement


Obstetric care networks

PCB set:

Pregnancy and childbirth patient-centered outcomes set


Patient-reported experience measures


Patient-reported outcome measures


Quality improvement


Electronic health record


  1. Groenewoud AS, Westert GP, Kremer JAM (2019) Value based competition in health care’s ethical drawbacks and the need for a values-driven approach. BMC Health Serv Res 19(1):1–6.

    Article  Google Scholar 

  2. Zorginstituut Nederland. Meer patiëntregie door meer uitkomstinformatie in 2022. Published online 2018:Available from: https://www.zorginstituutnederland

  3. Steinmann G, Van De Bovenkamp H, De Bont A, Delnoij D (2020) Redefining value: a discourse analysis on value-based health care. BMC Health Serv Res 20(1):1–13.

    Article  Google Scholar 

  4. Austin E, LeRouge C, Hartzler AL, Segal C, Lavallee DC (2020) Capturing the patient voice: implementing patient-reported outcomes across the health system. Qual Life Res 29(2):347–355.

    Article  PubMed  Google Scholar 

  5. NHS England (2017) National Patient Reported Outcome Measures (PROMS) Programme Guidance.

  6. Alonso J, Bartlett SJ, Rose M et al (2013) The case for an international patient-reported outcomes measurement information system (PROMIS®) initiative. Health Qual Life Outcomes 11(1):1–5.

    Article  Google Scholar 

  7. Greenhalgh J (2021) Understanding the complexities of collecting and using PRO data in a primary care context. BMJ Qual Saf.

    Article  PubMed  Google Scholar 

  8. Depla AL, Crombag NM, Franx A, Bekker MN (2021) Implementation of a standard outcome set in perinatal care: a qualitative analysis of barriers and facilitators from all stakeholder perspectives. BMC Health Serv Res 21(1):1–13.

    Article  Google Scholar 

  9. Foster A, Croot L, Brazier J, Harris J, O’cathain A (2018) The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews. J Patient-Rep Outcomes 2:46.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Stover AM, Haverman L, van Oers HA et al (2021) Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res 30(11):3015–3033.

    Article  PubMed  Google Scholar 

  11. Gelkopf M, Mazor Y, Roe D (2021) A systematic review of patient-reported outcome measurement (PROM) and provider assessment in mental health: goals, implementation, setting, measurement characteristics and barriers. Int J Qual Heal Care.

    Article  Google Scholar 

  12. Biber J, Ose D, Reese J et al (2018) Patient reported outcomes: experiences with implementation in a University Health Care setting. J Patient-Reported Outcomes 2:34.

    Article  Google Scholar 

  13. Van Kemenade E, De Kuiper M, Booij M, Minkman M (2022) How different quality paradigms undermine a shared value base for integrated care: the need for collective reflexivity. Int J Integr Care 22(1):1–6.

    Article  Google Scholar 

  14. Pascucci D, Sassano M, Nurchis MC et al (2021) Impact of interprofessional collaboration on chronic disease management: findings from a systematic review of clinical trial and meta-analysis. Health Policy 125(2):191–202.

    Article  PubMed  Google Scholar 

  15. D’Amour D, Goulet L, Labadie JF, Martín-Rodriguez LS, Pineault R (2008) A model and typology of collaboration between professionals in healthcare organizations. BMC Health Serv Res 8:118.

    Article  Google Scholar 

  16. Wolff AC, Dresselhuis A, Hejazi S et al (2021) Healthcare provider characteristics that influence the implementation of individual-level patient-centered outcome measure (PROM) and patient-reported experience measure (PREM) data across practice settings: a protocol for a mixed methods systematic revie. Syst Rev 10(1):1–12.

    Article  Google Scholar 

  17. Sudhof L, Shah NT (2019) In pursuit of value-based maternity care. Obstet Gynecol 133(3):541–551.

    Article  PubMed  Google Scholar 

  18. Nijagal MA, Wissig S, Stowell C et al (2018) Standardized outcome measures for pregnancy and childbirth, an ICHOM proposal. BMC Health Serv Res 18(1):953.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Al-Shammari I, Roa L, Yorlets RR et al (2019) Implementation of an international standardized set of outcome indicators in pregnancy and childbirth in Kenya: utilizing mobile technology to collect patient-reported outcomes. PLoS ONE 14(10):e0222978.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Chen A (2021) Introducing patient reported measures (PRMs) into finnish public maternity service: a study on professionals ’ views. BMC Health Serv Res 21:645.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Slavin V, Gamble J, Creedy DK, Fenwick J, Pallant J (2019) Measuring physical and mental health during pregnancy and postpartum in an Australian childbearing population: validation of the PROMIS global short form. BMC Pregnancy Childbirth 19(1):1–19.

    Article  Google Scholar 

  22. Chen A, Väyrynen K, Leskelä RL et al (2021) A qualitative study on professionals’ attitudes and views towards the introduction of patient reported measures into public maternity care pathway. BMC Health Serv Res 21(1):1–15.

    Article  CAS  Google Scholar 

  23. Lieshout F van, Jacobs G, Cardiff S (2017) Actieonderzoek: principes Voor Verandering in Zorg En Welzijn [Action Research: Principles for Change in Health and Welfare]. Koninklijke Van Gorcum BV

  24. Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ (2011) Explaining michigan: developing an ex post theory of a quality improvement program. Milbank Q 89(2):167–205.

    Article  PubMed  PubMed Central  Google Scholar 

  25. May CR, Cummings A, Girling M et al (2018) Using normalization process theory in feasibility studies and process evaluations of complex healthcare interventions: a systematic review. Implement Sci 13(1):1–42.

    Article  Google Scholar 

  26. Proctor E, Silmere H, Raghavan R et al (2011) Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Heal Ment Heal Serv Res 38(2):65–76.

    Article  Google Scholar 

  27. May CR, Mair F, Finch T et al (2009) Development of a theory of implementation and integration: normalization process theory. Implement Sci 4(1):1–9.

    Article  Google Scholar 

  28. Depla AL, Ernst-Smelt HE, Poels M, Crombag NM, Franx A, Bekker MN (2020) A feasibility study of implementing a patient-centered outcome set for pregnancy and childbirth. Heal Sci Rep.

    Article  Google Scholar 

  29. Laureij LT, Been JV, Lugtenberg M et al (2020) Exploring the applicability of the pregnancy and childbirth outcome set: a mixed methods study. Patient Educ Couns 103(3):642–651.

    Article  PubMed  Google Scholar 

  30. Depla AL, de Ruiter ML, Laureij LT et al (2022) Patient-reported outcome and experience measures in perinatal care to guide clinical practice: prospective observational study. J Med Internet Res 24(7):e37725.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Fleuren MAH, Paulussen TGWM, Dommelen P, Van BS (2014) Towards a measurement instrument for determinants of innovations. Int J Qual Heal Care 26(5):501–510.

    Article  Google Scholar 

  32. Finch TL, Girling M, May CR, et al. (2015) NoMAD: Implementation measure based on Normalization Process Theory. [Measurement instrument]. Published.

  33. Finch TL, Girling M, May CR et al (2018) Improving the normalization of complex interventions: part 2: validation of the NoMAD instrument for assessing implementation work based on normalization process theory (NPT). BMC Med Res Methodol 18(1):1–13.

    Article  Google Scholar 

  34. Joeris A, Knoll C, Kalampoki V, Blumenthal A, Gaskell G (2018) Patient-reported outcome measurements in clinical routine of trauma, spine and craniomaxillofacial surgeons: between expectations and reality: a survey among 1212 surgeons. BMJ Open 8(6):1–11.

    Article  Google Scholar 

  35. Budig K, Diez J, Conde P, Sastre M, Hernán M, Franco M (2018) Photovoice and empowerment: evaluating the transformative potential of a participatory action research project. BMC Public Health 18(1):1–9.

    Article  Google Scholar 

  36. Kiyimba N, Lester JN, O’Reilly M (2019) Using naturally occurring data in qualitative health research.

  37. R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing. Published online 2020.

  38. Dierckx de Casterle B, Gastmans C, Bryon E, Denier Y (2012) QUAGOL: a guide for qualitative data analysis. Int J Nurs Stud 49(3):360–371.

    Article  PubMed  Google Scholar 

  39. van Oers HA, Teela L, Schepers SA, Grootenhuis MA, Haverman L (2021) A retrospective assessment of the KLIK PROM portal implementation using the consolidated framework for implementation research (CFIR). Qual Life Res 30(11):3049–3061.

    Article  PubMed  Google Scholar 

  40. Amini M, Oemrawsingh A, Verweij LM et al (2021) Facilitators and barriers for implementing patient-reported outcome measures in clinical care: an academic center’s initial experience. Health Policy 125(9):1247–1255.

    Article  PubMed  Google Scholar 

  41. Mou D, Horn DM, Heng M et al (2021) Primary care physician’s (PCP) perceived value of patient-reported outcomes (PROs) in clinical practice: A mixed methods study. BMJ Qual Saf.

    Article  PubMed  Google Scholar 

  42. Withers K, Palmer R, Lewis S, Carolan-Rees G (2021) First steps in PROMs and PREMs collection in Wales as part of the prudent and value-based healthcare agenda. Qual Life Res 30(11):3157–3170.

    Article  PubMed  Google Scholar 

  43. PCORI. Users’ Guide to Integrating Patient-Reported Outcomes in Electronic Health Records.; 2017.

  44. Ahmed S, Zidarov D, Eilayyan O, Visca R (2021) Prospective application of implementation science theories and frameworks to inform use of PROMs in routine clinical care within an integrated pain network. Qual Life Res 30(11):3035–3047.

    Article  PubMed  Google Scholar 

  45. van der Wees PJ, Verkerk EW, Verbiest MEA et al (2019) Development of a framework with tools to support the selection and implementation of patient-reported outcome measures. J Patient-Rep Outcomes.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Feijen-de Jong EI, van der Voort-Pauw N, Nieuwschepen-Ensing EG, Kool L (2022) Intentions to leave and actual turnover of community midwives in the Netherlands: A mixed method study exploring the reasons why. Women Birth.

    Article  PubMed  Google Scholar 

  47. Lefebvre G, Honey L, Hines K et al (2020) Implementing obstetrics quality improvement, driven by medico-legal risk, is associated with improved workplace culture. J Obstet Gynaecol Canada 42(1):38-47.e5.

    Article  Google Scholar 

  48. Dharni N, Essex H, Bryant MJ et al (2021) The key components of a successful model of midwifery-led continuity of carer, without continuity at birth: findings from a qualitative implementation evaluation. BMC Pregnancy Childbirth 21(1):1–11.

    Article  Google Scholar 

  49. Roberts NA, Janda M, Stover AM, Alexander KE, Wyld D, Mudge A (2021) The utility of the implementation science framework “integrated promoting action on research implementation in health services” (i-PARIHS) and the facilitator role for introducing patient-reported outcome measures (PROMs) in a medical oncology outpatient. Qual Life Res 30(11):3063–3071.

    Article  PubMed  Google Scholar 

  50. Laureij LT, Depla AL, Kariman SS, et al (2023) Women’s experiences with using patient-reported outcome and experience measures in routine perinatal care in the Netherlands: a mixed methods study. BMJ Open (In Press)

  51. Calvert MJ, Rivera SC, Retzer A et al (2022) Patient reported outcome assessment must be inclusive and equitable. Nat Med 28(6):1110–1127.

    Article  CAS  Google Scholar 

  52. Nijagal MA, Patel D, Lyles C et al (2021) Using human centered design to identify opportunities for reducing inequities in perinatal care. BMC Health Serv Res 21(1):1–15.

    Article  Google Scholar 

  53. Mccloskey L, Bernstein J, The Bridging the Chasm Collaborative et al (2021) Bridging the chasm between pregnancy and health over the life course: a national agenda for research and action. Women’s Heal Issues 31(3):204–218.

    Article  Google Scholar 

Download references


We acknowledge the local project teams and care professionals for their implementation efforts, and all patients who participated in the clinical pilots.


This work was supported by ZonMW [Grant Number 516012516]. ZonMW was not involved in study design, data collection, analysis, and interpretation of data, writing the report, and decision to submit the article for publication.

Author information

Authors and Affiliations



AD, AF, and MB designed the study. AD, ML and AK performed the data collection. AD analyzed the data under supervision of ML, BP, and MB. All authors interpreted the data. AD wrote the first version of the manuscript under supervision of ML, BP, and MB. All authors revised all versions of the manuscript and approved the final version.

Corresponding author

Correspondence to Mireille N. Bekker.

Ethics declarations

Ethics approval and consent to participate

Ethical approval for this study was granted by the University of Utrecht Ethics Committee (Reference numbers #19-104 and #18-494). For each site, local approval was obtained from the regional ethical board. Informed consent was obtained of all care professionals. No patient data were included in this study.

Consent for publication

Not applicable.

Competing interests

AF and ML were part of the ICHOM PCB outcome Set Working Group. The other authors have nothing to declare.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file1

. Figure S1: Patient-reported domains and timeline of their measurement (PCB set). Table S1: Normalization Process Theory (NPT): mechanisms, subconstructs and assessment. Table S2: Full statements of the implementation survey administered to obstetric healthcare professionals to evaluate the implementation of the PCB set. Table S3: Topic guide statements for focus groups in care professionals attending QI session. Figure S2: Stacked-bar graphs of survey outcomes.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Depla, A.L., Pluut, B., Lamain-de Ruiter, M. et al. PROMs and PREMs in routine perinatal care: mixed methods evaluation of their implementation into integrated obstetric care networks. J Patient Rep Outcomes 7, 26 (2023).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: