- Open Access
Systematic review of guidance for the collection and use of patient-reported outcomes in real-world evidence generation to support regulation, reimbursement and health policy
Journal of Patient-Reported Outcomes volume 6, Article number: 57 (2022)
Real-world evidence (RWE) plays an increasingly important role within global regulatory and reimbursement processes. RWE generation can be enhanced by the collection and use of patient-reported outcomes (PROs), which can provide valuable information on the effectiveness, safety, and tolerability of health interventions from the patient perspective. This systematic review aims to examine and summarise the available PRO-specific recommendations and guidance for RWE generation.
Methods and findings
Medical Literature Analysis and Retrieval System Online, Excerpta Medica Database, and websites of selected organisations were systematically searched to identify relevant publications. 1,249 articles were screened of which 7 papers met the eligibility criteria and were included in the review. The included publications provided PRO-specific recommendations to facilitate the use of PROs for RWE generation and these were extracted and grouped into eight major categories. These included: (1) instrument selection, (2) participation and engagement, (3) burden to health care professionals and patients, (4) stakeholder collaboration, (5) education and training, (6) PRO implementation process, (7) data collection and management, and (8) data analysis and presentation of results. The main limitation of the study was the potential exclusion of relevant publications, due to poor indexing of the databases and websites searched.
PROs may provide valuable and crucial patient input in RWE generation. Whilst valuable insights can be gained from guidance for use of PROs in clinical care, there is a lack of international guidance specific to RWE generation in the context of use for regulatory decision-making, reimbursement, and health policy. Clear and appropriate evidence-based guidance is required to maximise the potential benefits of implementing PROs for RWE generation. Unique aspects between PRO guidance for clinical care and other purposes should be differentiated. The needs of various stakeholder groups (including patients, health care professionals, regulators, payers, and industry) should be considered when developing future guidelines.
Real-World Evidence (RWE) is defined by the U.S. Food and Drug Administration (FDA) as clinical evidence assessing benefits and risks of a medical product derived from analysis of real-world data (RWD) . RWE can be generated prospectively and retrospectively by different study designs . RWD in turn is defined as “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources” . The most common RWD sources are: electronic health records, claims databases, registries, and patient-generated data .
Currently, there is increasing recognition from global regulators, payers, and policy makers that patient-reported outcomes (PROs) — reports of health status directly provided by patients, without interpretation by a clinician or anyone else  — can provide valuable information on effectiveness, safety and tolerability from the patient perspective [3,4,5,6]. The U.S. FDA’s framework for Real-World Evidence Program acknowledged that PROs provide unique and valuable information which may complement the evidence obtained using traditional clinician-focused parameters . The agency recently published its RWD draft guidelines on data sources, data standards, and regulatory considerations [8,9,10,11]. However, these guidelines make limited reference to PROs beyond referencing existing FDA 2009 guidance  and ensuring appropriate monitoring of the study, including where applicable, PROs.
It is also worth noting that PROs constitute a key part of U.S. Centers for Medicare & Medicaid Services Meaningful Measures Framework . In the UK, the Medicines & Healthcare products Regulatory Agency (MHRA) recently issued two guideline documents focusing on the use of RWD to support regulatory decisions [14, 15]. The European Medicines Agency (EMA) currently uses RWE for safety monitoring and recently announced that the use of RWE will be established across its spectrum of regulatory use cases by 2025 .
Moreover, the recognition of the importance of PROs has led to a growing interest and increase in sponsorship by the pharmaceutical industry of real-world long-term safety studies which incorporate the longitudinal collection of PROs. Currently the PRO data for RWE generation are collected mainly in post-authorisation studies to support labelling claims, reimbursement and health policy making. For instance, the post-authorisation efficacy study for mepolizumab in the treatment of severe asthma  and post-authorisation efficacy and safety study for fingolimod in patients with relapsing–remitting multiple sclerosis  showed that the effectiveness of the drugs is consistent with clinical trial results under real-world settings.
In real-world contexts, prospective PRO collection has been limited and fragmented, with PROs collected in only 14% (8 out of 57) of recent post-authorization safety studies, consisting largely of one-off registries for post-marketing assessment sponsored by drug manufacturers in specific populations . However, increasing collection of PROs in routine clinical care to support individual decision making and audit/benchmarking offers emerging opportunities to use the PRO data for multiple purposes including the assessment of real-world efficacy, safety, and tolerability of health interventions for regulatory, reimbursement and health policy purposes.
Several guidelines on the implementation of PROs exist but mainly focus on RCTs or clinical practice [5, 12, 20,21,22,23,24,25] and provide little or no recommendations for the use of PROs in the context of RWE generation, addressing the needs of regulators and policy makers. Therefore, the aim of this systematic review was to examine relevant literature and summarise PRO-specific recommendations for RWE generation to support regulation, reimbursement, and health policy, and highlight areas for future research.
Scope of the review
The review focused on PRO-specific recommendations for RWE generation. PROs were differentiated from other types of patient-reported or generated data, such as PREMs, unstructured patient-generated health data, patient-reported data about medication used, health care utilisation or events.
Studies were included if they provide recommendations for the use of PROs in RWE generation to support regulation, reimbursement, and health policy. No date limits or country restrictions were applied. In order to capture all available recommendations for PRO use in RWE generation, eligibility was not restricted to formally issued guidelines but also included any publications with recommendations or opinions on PROs in RWE generation including research, reports, discussion papers, books, commentary/opinion pieces and editorials.
Publications containing broad recommendations for PRO use only, e.g., general statements supporting PRO data collection in real-world setting or indicating the usefulness of PRO data, or highlighting the need for more patient-centric RWE research [8,9,10,11, 14,15,16, 26,27,28,29] were excluded. However, these were referenced in our discussions where appropriate.
Publications providing recommendations solely on the use of PROs in RCTs or to guide clinical care, and clinical RWE studies were excluded [23, 24, 30].
Search strategy and publication selection
The systematic review was conducted according to a protocol registered in International Prospective Register of Systematic Reviews (PROSPERO), registration number: CRD42021235709. It was reported in compliance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines  (see Additional file 1 for the completed PRISMA checklist). Medical Literature Analysis and Retrieval System Online (MEDLINE) and Excerpta Medica Database (EMBASE) were searched using broad search terms to identify relevant publications. The search was conducted using the controlled vocabulary and free text of the relevant databases. These included words related to “real-world evidence”, “patient-reported outcomes”, “guidelines” and “recommendations”. Moreover, the search terms used were adapted from published database search filters for “quality of life”  and “guidelines” . No language or publication date restrictions were applied. For the full search strategy, see Additional file 2. Database searches were conducted on January 18, 2021. Two reviewers (KM, BT) independently screened the titles and abstracts according to the inclusion and exclusion criteria. Following this, the reviewers independently assessed the full texts of potentially relevant studies. At each stage, disagreements were resolved by discussion between the reviewers. If no consensus were reached, senior project members were consulted (MC, OLA). Records of screened entries, along with the reviewers’ reasons for inclusion and exclusion were held in EndNote X9 referencing software. When relevant conference abstracts were identified, we attempted to identify the full-text publication or conference output.
Other potentially relevant publications were identified from forward and backward citation searching of included studies. In addition, the grey literature was searched using a combination of the search terms from the original database search. Sources were:
Google Scholar (100 first hits);
HTA (Health Technology Assessment) agency websites: Canadian Agency for Drugs & Technologies in Health (CADTH), Haute Autorité de santé (HAS), The National Institute for Health and Care Excellence (NICE) and International HTA database, and NHS Evidence;
Regulator websites: EMA and FDA;
Professional organisations: Society for Health Economics and Outcomes Research (ISPOR), International Society for Quality of Life Research (ISOQOL), Standards in Analysing Patient-Reported Outcomes and Quality of Life Endpoints (SISAQoL) Consortium, Patient-Centered Outcomes Research Institute (PCORI), Agency for Healthcare Research and Quality (AHRQ), and International Society of Pharmacovigilance (ISOP).
Relevant data were extracted into an Excel spreadsheet from the included publications by one reviewer (KM) and checked for accuracy (by BT). Data related to the following areas were extracted wherever possible: guidance issuing body, aim of the guidance, clinical area, patient population and recommended PRO instruments. Moreover, domains, described in the paper by Calvert et al. , were used as an initial framework for data extraction covering: objectives; patient population; instrument selection; frequency of administration; mode of administration; data collection method; data monitoring; presentation of results; ethics; data ownership and consent; audit; privacy; feedback to clinicians, patients, healthcare providers, drug manufacturers, regulatory authorities; and resources needed. Additional categories were added if identified information did not match any of the previously described domains. All extracted PRO-related recommendations were re-arranged into a smaller number of categories around similar issues addressed by the publications. Finally, these domains were grouped into major categories.
The search strategy identified 1,453 potentially eligible entries, of which 1,249 remained after removing duplicates. After screening titles and abstracts, 1,198 entries were excluded, leaving 51 publications for full-text screening. Of these, five met the study inclusion criteria. An additional two entries were identified by reference and website searching, resulting in a total of seven publications included in the review. The PRISMA flow diagram (Fig. 1) provides an overview of the review process and study selection.
Characteristics of included publications
The summary characteristics of all seven publications are presented in Table 1. Four were published in peer-reviewed journals [6, 34,35,36], two were conference posters [37, 38] and one was an online published report . Four of the publications [6, 36, 37, 39] did not focus on a specific patient population or clinical area and provided recommendations applicable to the general patient cohorts. One of the publications focused on patients with dementia  and one gave recommendations focused on elderly patients . One paper discussed PRO data collection among patients undergoing selected surgical interventions .
The included publications provided recommendations for PRO data collection and its use in different RWE settings. Two papers gave general recommendations relevant to real-world research [37, 39]. The remaining publications focused on: drug development , post-authorisation safety evaluation [35, 36, 38] and pragmatic clinical trials .
The recommendations provided were grouped into eight major categories: (1) instrument selection, (2) participation and engagement, (3) burden to health care professionals (HCPs) and patients, (4) stakeholder collaboration, (5) education and training, (6) PRO implementation process, (7) data collection and management, and (8) data analysis and presentation of results.
An overview of the recommendation categories is presented in Table 2. Additionally, detailed data extracted from included studies for the major categories can be found in Additional file 3.
Five of seven included publications provided some level of advice about choosing appropriate PRO measure [6, 34,35,36,37]. PRO measure selection was discussed in the context of: instrument suitability for the target population, availability of relevant psychometric evidence supporting the use of PRO instrument in a given context and adaptation of existing instruments or development of new measures.
Calvert et al.  gave a broad recommendation stating that PROs measures used in the RWE setting need to be valid, consistent with the intended use and relevant to the identified needs of the target population. Banerjee et al.  proposed a core minimum dataset (including PROs) for non-regulated consumer websites listing information which should be collected from patients to allow for post-approval safety monitoring. Hanson et al.  highlighted the need for outcome measures to address patient or caregiver-centred outcome domains and to be acceptable to respondents.
The need for a definitive evidence base for PRO measures selected for use in a clinical setting was emphasized by Kyte et al. . Hanson et al.  suggested that measure attributes such as psychometric properties (e.g. validity, reliability, sensitivity to change, floor/ceiling effect) should be considered when selecting PRO measures to identify instrument fit for purpose.
For situations where no appropriate measures are available, Hanson et al.  suggested the adaptation of existing measures or the development of de novo instruments. Particular attention was given to translation of existing questionnaires. Despite not recommending specific measures, authors often underlined the importance of using well translated PRO measures. Hanson et al. , Rylands et al.  and Calvert et al.  stressed the importance of adaptation and translation of PRO measures to ensure they match the literacy skills and are culturally relevant to diverse patient populations.
Participation and engagement
This category was split in two sub-domains. The first focuses on recommendations aiming to improve patient participation in a study and enhance quality of collected data. The second focuses on the involvement of different stakeholders in study design or conduct.
Authors of four publications [6, 34, 37, 38] gave recommendations to strengthen patient participation in RWE studies. Calvert et al.  recommended to make questionnaires available in different languages to meet language requirements of diverse patient populations. Hanson et al.  stated that outcome measures used, should address patient or caregiver-centred outcome domain and be acceptable to respondents. Rylands et al.  noted that patient engagement and mode of recruitment strongly depend on the level of patient contact with healthcare services. Thus, it would be beneficial to consider the frequency of clinic visits required by patients when designing a study using RWD. Akiyama et al.  postulated that special attention is required at the participating sites for elderly patients. For example, large letters and simple wording may be helpful to be used for explanatory document and questionnaires dedicated for elderly patients. Also, posters and flyers may be used to promote the study.
Study development and conduct
Stakeholder involvement in designing RWE studies was recommended by five studies [6, 34, 35, 38, 39]. Greater HCP and health care providers involvement in planning study and data collection activities is beneficial. Akiyama et al.  noted the importance of involving clinicians with keen interest in PROs as it is key for successful data collection. Greater involvement of external stakeholders (payers, regulators, industry) in RWE studies can be obtained by demonstrating its benefits and importance to these organisations .
Hanson et al.  and Akiyama et al.  suggested to engage stakeholders early, particularly during PRO measure development process. Hanson et al.  focused mainly on collaboration with key stakeholders such as health system leadership. On the other hand, Akiyama et al.  and the ABPI  focused on both collaboration between internal (within industry or RWE study team) and external stakeholders (external experts, payers, regulators). Informing both internal and external stakeholders about justifications for PRO data collection for RWE and communicating to them the value of PRO assessments was also recommended [34, 38, 39].
Three publications stressed the importance of stakeholder involvement in PRO measure selection [6, 35, 38]. Focus groups and pilot tests were proposed as methods for enhancing stakeholder’s participation in measure selection or development.
Burden to HCPs and patients
The importance of not overburdening patients, clinicians and health care providers with frequent and lengthy data collections were described as key to the successful implementation of PRO measures for RWE generation. Hanson et al.  mentioned that paper questionnaires or patient interviews typically impose high respondent burden and are rarely tested in real-world clinical settings for wide-scale application to learn about patients’ experiences. Thus, computer adaptive testing, which may tailor PRO items to individual patient needs, may be considered to reduce patient burden [40, 41]. Two papers [6, 37] discussed the issue of patient burden and both postulated minimisation of patient, clinician, and health care provider burden by limiting frequency and complexity of data collection to a necessary minimum.
Collaboration between relevant stakeholders was often mentioned as a key component for the successful use of PROs for RWE generation. According to Calvert et al.  international collaboration “…across multiple stakeholders including patients, caregivers, clinicians, regulators, ethicists, industry, payers and policy makers” is needed to establish a standardised approach to PRO assessment for RWE research. This multi-stakeholder collaboration is vital when collecting PRO data for multiple purposes to ensure that the data generated will meet their needs in the future.
Education and training
The importance of educating HCPs, patients, researchers, and other stakeholders on the potential benefits of PROs for RWE generation were mentioned by three publications [34, 35, 38]. Training focused on motivation maintenance and study procedures should be offered to HCPs involved . Kyte et al.  recommended that efforts should be made to provide guidance to health care providers and patients on the interpretation and utilisation of benchmarks based on PROs captured in real world setting. Hanson et al.  created a searchable outcome measures library (including PRO measures) to educate other researchers interested in designing pragmatic trials in dementia.
PRO implementation process
Five publications [6, 35, 36, 38, 39] gave recommendations specific to the process of PRO implementation. Akiyama et al.  described the PRO inclusion process to collect data for post-marketing surveillance. They created a map that covers four stages: internal discussion, design and preparation, implementation, dissemination.
Calvert et al.  emphasised that special attention should be given to the resources needed to successfully implement PROs. Additional staff might be required to assist some of the patients with data collection. It is of paramount importance to secure up-front funding to cover costs associated with additional staff time needed, license fees for PRO measures, PRO training, data collection and devices costs. Kyte et al.  postulated that a shift to a “bottom-up” clinic-based PRO data collection approach that could be used for multiple purposes may be beneficial for patients and cost containment. Wider utilisation of data collected including post-marketing surveillance was postulated.
The implications of PRO data collection in real-world studies to address the legal requirement for obtaining Clinical Trial Authorisation and being compliant with the EU Clinical Trials Directive were mentioned by The Association of the British Pharmaceutical Industry (ABPI) guidance . When PROs that are not in routine use are to be utilised to obtain data in RWE studies, legislation applicable to interventional clinical trials might need to be followed as PRO data collection can be seen as intervention administrated on the top of the regular care provision. Additionally, Banerjee et al.  advocated acceptance of non-medically confirmed adverse events reported by patients to account for more patient-centric approach in post-registration safety surveillance.
Data collection and management
Authors of all seven publications [6, 34,35,36,37,38,39] made recommendations for data collection and management. The following issues for RWE generation were specifically addressed: frequency of data collection, integration with other databases, data audit, data ownership, electronic data capture and impact of disease progression on data collection.
Frequency of data collection
As pointed out by Calvert et al.  frequency of data collection depends on stakeholder needs and the study population which should be considered early in study designing process. Additionally, patients with high symptom burden may require more frequent monitoring . Two publications [6, 37] pointed out that the frequency of measurement is influenced by the schedule of patients’ visits and poses a challenge for data interpretation. Thus, appropriate methods of PRO measurement which facilitate data interpretation might be needed. Additionally, PRO data capture between scheduled visits could be considered. Calvert et al.  advocated the use of alert systems for PRO data collected between the visits, which would inform HCPs about issues requiring immediate attention. Additionally, reminders sent from electronic data capture systems may facilitate data collection and increase patient retention [38, 42].
Integration with other databases
Secondary data collection by integration of data capture with other databases, like electronic health records or registries, was suggested by two papers [6, 34]. Hanson et al.  pointed out that EHR systems might be used to facilitate PRO data collection if they had the capability to do so.
The need for ongoing data quality audit was postulated by Calvert et al. . Moreover, Rylands et al.  noted that potentially the amount of missing data, will be influenced by whether PRO data are routinely collected in clinical practice. Moreover, decisions about RWE study design (prospective or retrospective design) may be influenced by whether PROs are routinely collected or not.
Issues related to data ownership, storage and access were mentioned by four publications [6, 35, 36, 39]. It should be clearly stated who owns the rights to any data or potential intellectual property generated within the real-world study. Moreover, periods of data retention, entities responsible for their storage and applicable conditions need to be determined a priori. Patients should be informed about the way their data will be used and they need to consent to that.
Electronic data capture
Five publications [6, 34,35,36, 38] provided recommendations specific to electronic data capture. All of them advocated the utilisation of electronic capture where appropriate. Akiyama et al.  maintained that electronic data collection is suitable for elderly patients and should be used wherever possible, as it streamlines data collection and improves quality of data collected. Electronic data capture can be conducted using the following devices: smartphone or website applications, automated interactive voice response telephone and wearable devices . PRO-enabled website-based platforms were pointed as a preferable data source for collecting information from patients about treatments’ safety due to the higher quality of data captured . However, the target population’s level of IT literacy should be considered when deciding on the mode of questionnaire administration as this can have serious implications for the representativeness of collected data . Remote delivery of electronic PROs may lead to inequitable access if a substantial proportion of the target population have limited or no access to the internet. These issues could potentially decrease the value of PRO data collected as part of the RWE generation for regulatory purposes and may not be representative. Patients should be provided alternative modes of data collection (e.g., paper questionnaires, automated telephone services).
Impact of disease progression on data collection
Three publications [34, 37, 38] commented on changing patient health status over time, and its impact on data collection activities. Hanson et al.  highlighted that people living with dementia early in the disease trajectory can self-report. Nevertheless, once the disease progresses there may be a need for transition to proxy reporting, yet no best practices exist for interpretation of data reported by proxy. Similar concerns in the context of elderly patients were expressed by Akiyama et al.  Rylands et al.  acknowledged that patients’ ability to self-report need to be assessed early at the stage of study design.
Data analysis and presentation of results
Four publications [6, 35, 36, 39] provided guidance about results presentation and interpretation. Calvert et al.  and Kyte al.  provided general recommendations, claiming that data should be analysed and reported appropriately, according to the study objectives and the measure recommendations, following a methodologically robust process. Potential sources of bias and confounding need to be investigated and researchers could offer guidance on how to interpret and utilise findings. Guidance by ABPI  stressed how important it is to use sound methods for data generation, cleaning and analysis. Banerjee et al.  proposed suitable statistical methods for the analysis of datasets containing information about adverse events, such as appropriate descriptive statistics, methods to address disproportionality of results and multivariate analysis.
This review provides the first summary of available guidance for the use of PROs in RWE generation to support regulation, reimbursement, and health policy. Available guidance is fragmented, and it is evident that a better understanding of how to optimally collect and utilise PROs for RWE generation is needed. The main themes generated from the analysis of the included publications addressed issues relating to PRO data collection, analysis, and stakeholder collaboration.
It was recommended that steps should be taken to minimise the burden of PRO data collection on HCPs and patients, [6, 37] reduce data collection errors, allow automatic score calculations, improve data security, and speed up data collection process through the electronic data capture. These would enhance the quality of PROs obtained as part of RWD .
Statistical methods for the analysis of collected PRO data were also recommended . Nevertheless, recommendations related to data analysis strategies to manage bias and confounding were not identified as part of this review. The need to develop such guidance seems evident. While existing PRO datasets collected in a real-world setting can be used to inform regulatory or reimbursement processes, a tailored approach to PRO data analysis is key to eliminating biases and confounding. Data captured in the real-world setting might require some additional statistical manipulation to account for inequitable access to PROs (e.g. due to low IT literacy among some groups of patients).
Our review highlighted the need for stakeholders’ engagement for successful PRO implementation. To improve efficiency of data collection activities for RWE, collaboration between different stakeholders need to be developed. Each stakeholder might have different expectations from the data collected as they can be used for various purposes. Thus, involvement of various stakeholders early at the stage of research planning is vital. To fully harness the potential benefits of collecting PRO data as part of real-world studies, it was recommended that various issues around stakeholder involvement, instrument selection and implementation need to be resolved [6, 34,35,36,37,38].
The potential benefits of collecting PROs may be maximised by using the data for multiple purposes including trials, routine care, audits, benchmarking and RWE generation . For instance, in routine clinical practice, changes in an individual patient’s health status as indicated by their PRO data could facilitate the tailoring of their clinical management, which may, in turn, improve treatment outcomes. The utilisation of PRO alerts informing clinicians about changes in patients’ health status may lead to improvement in patient care by providing the opportunity for timely interventions (e.g. earlier clinic appointments or immediate hospitalisation) . The same data can be aggregated for patients within healthcare systems to provide RWE of the safety and tolerability of health interventions. The use of PROs for multiple purposes would require agreement on the measures to be used to meet both regulatory and clinical needs. Feasibility of using the same PRO measures for multiple purposes could be explored further in the future research but it seems to be possible when focusing on aspects such as proximal symptoms and treatment tolerability.
The collection of PRO data in RWE research can bring numerous benefits by providing evidence of long-term safety, tolerability, and effectiveness from the patient perspective. The usefulness of longer-term additional data collection for the purpose of pharmaceutical licensing was previously described in article series by London School of Economics, which considered the use of RWE in Europe . Additionally, the value of data reported directly by patients was evidenced by a comparison of chemotherapy-related adverse events reported by patients and clinicians, where patients tended to self-report more frequent and higher levels of symptoms than clinicians .
Although every effort was made to find potentially relevant publications (forwards/backwards citation searches, hand reference list searches, grey literature searches, and website searches were conducted) there is a possibility that some relevant publications were not identified due to poor indexing of the databases and websites searched. A limitation of this work was the dearth of guidance for the use of PROs in RWE to support regulation, reimbursement, and health policy. Even when recommendations were made, in some instances there were limited details on the rationale behind them.
The development of further guidance specific to PROs in RWE generation to support regulation, reimbursement and health policy will be an important next step. In doing so, it is of crucial importance to learn more about the various stakeholders’ needs and the current use of PROs in RWE generation to inform the guideline development. Patients, HCPs, regulators, payers, health care providers and industry will bring important perspectives about the specific needs of all those groups. The ISPOR Special Interest Group for Clinical Outcome Assessment is currently working on the standardisation of outcomes for real-world studies. Nevertheless, further research is needed to better inform the development of methodological recommendations for PRO-specific data generation as part of RWE for regulatory, reimbursement, and health policy.
PROs may provide a valuable source of information in RWE generation from the patient perspective. Whilst valuable insights can be gained from guidance for use of PROs in clinical care, there is a lack of international guidance specific to RWE generation in the context of use for regulatory decision-making, reimbursement, and health policy. Clear and appropriate guidance, developed based on evidence, is required to maximise the potential benefits of implementing PROs for RWE generation. Unique aspects between PRO guidance for clinical care and other purposes should be differentiated. This review summarises some recommendations to optimise the use of PROs for RWE generation and highlights the need for further PRO-specific international guidelines to facilitate RWE generation for regulatory, reimbursement, and health policy. The needs of various stakeholder groups (including patients, health care professionals, regulators, payers, and industry) should be considered when developing future guidelines.
Availability of data and materials
All data generated during this study are included in this published article and its supplementary information files. The record of inclusion/exclusion choices is available upon request.
Association of the British Pharmaceutical Industry
Agency for Healthcare Research and Quality
Canadian Agency for Drugs & Technologies in Health
European Medicines Agency
Excerpta Medica Database
U.S. Food and Drug Administration
Haute Autorité de Santé
Health care professionals
Health technology assessment
International Society of Pharmacovigilance
International Society for Quality of Life Research
Society for Health Economics and Outcomes Research
Medical Literature Analysis and Retrieval System Online
Medicines & Healthcare Products Regulatory Agency
National Institute for Health and Care Excellence
Patient-Centered Outcomes Research Institute
Patient-reported experience measures
Preferred Reporting items for systematic reviews and meta-analyses
International prospective register of systematic reviews
Randomised control trial
Standards in Analysing Patient-Reported Outcomes and Quality of Life Endpoints
FDA (2021) FDA Real-world evidence. https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence. Accessed 05 Mar 2021
Kluetz PG, O’Connor DJ, Soltys K (2018) Incorporating the patient experience into regulatory decision making in the USA, Europe, and Canada. Lancet Oncol 19(5):e267–e274. https://doi.org/10.1016/s1470-2045(18)30097-4
FDA (2021) FDA patient-focused drug development guidance series for enhancing the incorporation of the patient’s voice in medical product development and regulatory decision making. https://www.fda.gov/drugs/development-approval-process-drugs/fda-patient-focused-drug-development-guidance-series-enhancing-incorporation-patients-voice-medical. Accessed 15 Feb 2022
MHRA (2021) Innovative licensing and access pathway. https://www.gov.uk/guidance/innovative-licensing-and-access-pathway. Accessed 15 Feb 2022
European Medicines Agency (2016) Appendix 2 to the guideline on the evaluation of anticancer medicinal products in man: the use of patient-reported outcome (PRO) measures in oncology studies. Available at: https://www.ema.europa.eu/en/documents/other/appendix-2-guideline-evaluation-anticancer-medicinal-products-man_en.pdf
Calvert MJ, O’Connor DJ, Basch EM (2019) Harnessing the patient voice in real-world evidence: the essential role of patient-reported outcomes (News). Nat Rev Drug Discov 18(10):731–732
US Food & Drug Administration (2018) Framework for FDA’s real-world evidence program. Available at: https://www.fda.gov/media/120060/download
FDA (2021) Real-world data: assessing electronic health records and medical claims data to support regulatory decision-making for drug and biological products. Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/real-world-data-assessing-electronic-health-records-and-medical-claims-data-support-regulatory
FDA (2021) Real-world data: assessing registries to support regulatory decision-making for drug and biological products guidance for industry. Available at: https://www.fda.gov/media/154449/download
FDA (2021) Data standards for drug and biological product submissions containing Real-world data. Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/data-standards-drug-and-biological-product-submissions-containing-real-world-data
FDA (2021) Considerations for the use of real-world data and real-world evidence to support regulatory decision-making for drug and biological products. Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-real-world-data-and-real-world-evidence-support-regulatory-decision-making-drug
FDA (2009) Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims. Available at: https://www.fda.gov/media/77832/download
U.S. Centers for Medicare & Medicaid (2021) Meaningful measures framework. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityInitiativesGenInfo/CMS-Quality-Strategy. Accessed 05 Jun 2021
MHRA (2021). MHRA guideline on randomised controlled trials using real-world data to support regulatory decisions. https://www.gov.uk/government/publications/mhra-guidance-on-the-use-of-real-world-data-in-clinical-studies-to-support-regulatory-decisions/mhra-guideline-on-randomised-controlled-trials-using-real-world-data-to-support-regulatory-decisions. Accessed 09 Feb 2022
MHRA (2021). MHRA guidance on the use of real-world data in clinical studies to support regulatory decisions. https://www.gov.uk/government/publications/mhra-guidance-on-the-use-of-real-world-data-in-clinical-studies-to-support-regulatory-decisions/mhra-guidance-on-the-use-of-real-world-data-in-clinical-studies-to-support-regulatory-decisions. Accessed 09 Feb 2022
Arlett P, Kjær J, Broich K, Cooke E (2022) Real-world evidence in EU medicines regulation: enabling use and establishing value. Clin Pharmacol Ther 111(1):21–23. https://doi.org/10.1002/cpt.2479
Harrison T, Canonica GW, Chupp G, Lee J, Schleich F, Welte T et al (2020) Real-world mepolizumab in the prospective severe asthma REALITI-A study: initial analysis. Eur Respir J. https://doi.org/10.1183/13993003.00151-2020
Druart C, El Sankari S, van Pesch V (2017) Long-term safety and real-world effectiveness of fingolimod in relapsing multiple sclerosis. Patient Relat Outcome Meas 9:1–10. https://doi.org/10.2147/PROM.S122401
Engel P, Almas MF, De Bruin ML, Starzyk K, Blackburn S, Dreyer NA (2017) Lessons learned on the design and the conduct of post-authorization safety studies: review of 3 years of PRAC oversight. Br J Clin Pharmacol 83(4):884–893. https://doi.org/10.1111/bcp.13165
Franklin P, Chenok K, Lavalee D, Love R, Paxton L, Segal C et al (2017) Framework to guide the collection and use of patient-reported outcome measures in the learning healthcare system. EGEMS 5(1):17. https://doi.org/10.5334/egems.227
Calvert M, Kyte D, Mercieca-Bebber R, Slade A, Chan AW, King MT et al (2018) Guidelines for inclusion of patient-reported outcomes in clinical trial protocols: the SPIRIT-PRO extension. JAMA 319(5):483–494. https://doi.org/10.1001/jama.2017.21903
Calvert M, Blazeby J, Altman DG, Revicki DA, Moher D, Brundage MD et al (2013) Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. JAMA 309(8):814–822. https://doi.org/10.1001/jama.2013.879%JJAMA
Chan EKH, Edwards TC, Haywood K, Mikles SP, Newton L (2019) Implementing patient-reported outcome measures in clinical practice: a companion guide to the ISOQOL user’s guide. Qual Life Res 28(3):621–627
Aaronson NK, Elliott T, Greenhalgh J, Halyard M, Hess R, Miller D et al (2015) User’s guide to implementing patient-reported outcomes assessment in clinical practice. Available at: https://www.isoqol.org/wp-content/uploads/2019/09/2015UsersGuide-Version2.pdf
CERTAIN (2021) ePROs in clinical care. Guidelines & tools for health systems. http://epros.becertain.org/. Accessed 15 Jan 2021
FDA (2019) Submitting documents using real-world data and real-world evidence to FDA for drugs and biologics. Available at: https://www.fda.gov/media/124795/download
Oehrlein EM, Schoch S et al (2021) Patient-centered real-world evidence: methods recommendations from an evidence-based consensus process. May 2021. National Health Council. Available at: https://nationalhealthcouncil.org/patient-centeredrwe
FDA (2017) Use of real-world evidence to support regulatory decision-making for medical devices. Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-real-world-evidence-support-regulatory-decision-making-medical-devices
Wang SV, Pinheiro S, Hua W, Arlett P, Uyama Y, Berlin JA et al (2021) STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. BMJ 372:m4856. https://doi.org/10.1136/bmj.m4856
LeRouge C, Austin E, Lee J, Segal C, Sangameswaran S, Hartzler A et al (2020) ePROs in clinical care: guidelines and tools for health systems. Seattle, WA: CERTAIN. Available at: http://epros.becertain.org/sites/epros.becertain.org/files/tools/ePROs%20in%20clinical%20care%20print%20edition%20%28v1.1%29%20.pdf
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. https://doi.org/10.1136/bmj.n71
Brettle AJ, Long AF, Grant MJ, Greenhalgh J (1998) Searching for information on outcomes: do you need to be comprehensive? J Qual Health Care 7(3):163–167. https://doi.org/10.1136/qshc.7.3.163
CADTH (2021) Strings attached: CADTH’s database search filters. https://www.cadth.ca/resources/finding-evidence/strings-attached-cadths-database-search-filters. Accessed 15 Jan 2021.
Hanson LC, Bennett AV, Jonsson M, Kelley A, Ritchie C, Saliba D et al (2020) Selecting outcomes to ensure pragmatic trials are relevant to people living with dementia. J Am Geriatr Soc 68(S2):S55–S61
Kyte D, Cockwell P, Lencioni M, Skrybant M, Hildebrand MV, Price G et al (2016) Reflections on the national patient-reported outcome measures (PROMs) programme: where do we go from here? J R Soc Med 109(12):441–445. https://doi.org/10.1177/0141076816677856
Banerjee AK, Okun S, Edwards IR, Wicks P, Smith MY, Mayall SJ et al (2013) Patient-reported outcome measures in safety event reporting: PROSPER consortium guidance. Drug Saf 36(12):1129–1149
Rylands AJ, Boxell E, Bottomley CJ (2018) Key considerations for the collection of patient reported outcome (Pro) data in real world (Rw) studies. ISPOR Europe. October. Barcelona. Available at: https://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=emed19&AN=2001401173
Akiyama S, Fujinuma EW, Sakaguch T, Rossi B, Aitoku Y, Adachi K (2015) Issues of patient-reported outcome assessment in post-marketing surveillance–considerations for diseases in the elderly. ISPOR 18th Annual European Congress. Milan
The Association of the British Pharmaceutical Industry (2011) Guidance. Demonstrating value with real world data: a practical guide. Available at: https://www.abpi.org.uk/media/1591/2011-06-13-abpi-guidance-demonstrating-value-with-real-world-data.pdf
Bass M, Morris S, Neapolitan R (2015) Utilizing multidimensional computer adaptive testing to mitigate burden with patient reported outcomes. AMIA Annu Symp Proc 2015:320–328
Aiyegbusi OL, Nair D, Peipert JD, Schick-Makaroff K, Mucsi I (2021) A narrative review of current evidence supporting the implementation of electronic patient-reported outcome measures in the management of chronic diseases. Ther Adv Chronic Dis 12:20406223211015960. https://doi.org/10.1177/20406223211015958
Akiyama S, Watanabe Fujinuma E, Rossi B, Aitoku Y, Adachi K (2015) Qualitative discussion on issues of patient-reported outcome assessment in post-marketing surveillance for diseases in the elderly. Value Health. https://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=emed16&AN=72085022
Calvert M, Kyte D, Price G, Valderas JM, Hjollund NH (2019) Maximising the impact of patient reported outcome assessment for patients and society. BMJ 364:k5267. https://doi.org/10.1136/bmj.k5267
Gill J, Kanavos P, Albanell J, Dank M, Duncombe R, Fink-Wagner A et al (2017) RWE in Europe paper II: The use of real world evidence in the disease context. Available at: http://eprints.lse.ac.uk/id/eprint/77037
Liu L, Suo T, Shen Y, Geng C, Song Z, Liu F et al (2020) Clinicians versus patients subjective adverse events assessment: based on patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). Qual Life Res 29(11):3009–3015. https://doi.org/10.1007/s11136-020-02558-7
This research was conducted as part of a PhD programme funded by GSK. TK is a co-supervisor of KM (the holder of the GSK PhD grant) and is a Director at GSK Ltd. In his role as co-supervisor TK inputted to all stages of this research.
Ethics approval and consent to participate
Consent for publication
MC is Director of the Birmingham Health Partners Centre for Regulatory Science and Innovation, Director of the Centre for Patient-Reported Outcomes Research and is a National Institute for Health Research (NIHR) Senior Investigator. She receives funding from the NIHR Birmingham Biomedical Research Centre, the NIHR Surgical Reconstruction and Microbiology Research Centre and NIHR ARC West Midlands at the at the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Health Data Research UK, Innovate UK (part of UK Research and Innovation), Macmillan Cancer Support, UCB and GSK. MC has received personal fees from Astellas, Takeda, Merck, Daiichi Sankyo, Glaukos, CIS Oncology, Aparito Ltd, GSK, Genentech and the Patient-Centered Outcomes Research Institute (PCORI) outside the submitted work. OLA receives funding from the NIHR Birmingham Biomedical Research Centre (BRC), NIHR Applied Research Collaboration (ARC) West Midlands at the University of Birmingham and University Hospitals Birmingham NHS Foundation, Innovate UK (part of UK Research and Innovation), Gilead Sciences Ltd, and Janssen Pharmaceuticals, Inc. OLA declares personal fees from Gilead Sciences Ltd, GSK and Merck outside the submitted work. TK is an employee and shareholder of GSK Ltd. KM is the holder of the GSK PhD grant. Other authors declare no competing interests. The views expressed in this article are those of the authors and not necessarily those of the NIHR, or the Department of Health and Social Care.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Philip Collis: Patient partner
Additional file 1:
PRISMA 2020 checklist
Additional file 2:
Additional file 3:
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Maruszczyk, K., Aiyegbusi, O.L., Torlinska, B. et al. Systematic review of guidance for the collection and use of patient-reported outcomes in real-world evidence generation to support regulation, reimbursement and health policy. J Patient Rep Outcomes 6, 57 (2022). https://doi.org/10.1186/s41687-022-00466-7
- Patient-reported outcomes
- Real-world evidence