Development and usability testing of an electronic patient-reported outcome (ePRO) solution for patients with inflammatory diseases in an Advanced Therapy Medicinal Product (ATMP) basket trial
Journal of Patient-Reported Outcomes volume 7, Article number: 98 (2023)
Electronic patient-reported outcome (ePRO) systems are increasingly used in clinical trials to provide evidence of efficacy and tolerability of treatment from the patient perspective. The aim of this study is twofold: (1) to describe how we developed an electronic platform for patients to report their symptoms, and (2) to develop and undertake usability testing of an ePRO solution for use in a study of cell therapy seeking to provide early evidence of efficacy and tolerability of treatment and test the feasibility of the system for use in later phase studies.
An ePRO system was designed to be used in a single arm, multi-centre, phase II basket trial investigating the safety and activity of the use of ORBCEL-C™ in the treatment of patients with inflammatory conditions. ORBCEL-C™ is an enriched Mesenchymal Stromal Cells product isolated from human umbilical cord tissue using CD362+ cell selection. Usability testing sessions were conducted using cognitive interviews and the ‘Think Aloud’ method with patient advisory group members and Research Nurses to assess the usability of the system.
Nine patient partners and seven research nurses took part in one usability testing session. Measures of fatigue and health-related quality of life, the PRO-CTCAE™ and FACT-GP5 global tolerability question were included in the ePRO system. Alert notifications to the clinical team were triggered by PRO-CTCAE™ and FACT-GP5 scores. Patient participants liked the simplicity and responsiveness of the patient-facing app. Two patients were unable to complete the testing session, due to technical issues. Research Nurses suggested minor modifications to improve functionality and the layout of the clinician dashboard and the training materials.
By testing the effectiveness, efficiency, and satisfaction of our novel ePRO system (PROmicsR), we learnt that most people with an inflammatory condition found it easy to report their symptoms using an app on their own device. Their experiences using the PROmicsR ePRO system within a trial environment will be further explored in our upcoming feasibility testing. Research nurses were also positive and found the clinical dashboard easy-to-use. Using ePROs in early phase trials is important in order to provide evidence of therapeutic responses and tolerability, increase the evidence based, and inform methodology development.
Trial registration: ISRCTN, ISRCTN80103507. Registered 01 April 2022, https://www.isrctn.com/ISRCTN80103507
Plain English summary
More and more patients tell clinicians how they feel by completing questionnaires electronically. Therefore, it is important to assess how easy it is for patients to do this. In this study, we describe how we developed an electronic platform for patients to report their symptoms and how we tested the usability of this platform with patient partners and research nurses. Once the electronic platform was developed, quality of life and symptoms questionnaires were programmed onto it. Alerts were sent to the clinical team if specific scores were obtained on the symptoms questionnaires. Although two patient partners were not able to finish the testing session because of technical issues, the ones who completed the session liked its simplicity and responsiveness. The research nurses also liked the system and only suggested minor modifications. Following this testing, we refined the electronic platform to test it further in a larger study which investigates the safety and use of a drug. We hope that thanks to this electronic platform, we will obtain useful information on the safety and efficacy of treatment.
Cell or gene therapies show great potential in treating patients with inflammatory conditions that are not responding to current treatments [1, 2]. The use and evaluation of these novel therapies, also known as Advanced Therapy Medicinal Products (ATMPs), in clinical trials, require assessment of their impact upon patients’ symptoms and health-related quality of life (HRQoL). These impacts can be assessed using patient-reported outcomes (PROs). Assessments are required at multiple timepoints, including before and at the point of receiving the therapy and over the duration of the trial. The European Medicines Agency draft guideline on safety and efficacy follow-up and risk management of ATMPs  states that 'Advanced Therapy Medicinal Products… because of their novelty, complexity and technical specificity may cause new risks to patients'.
A PRO is defined as “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation… by a clinical or anyone else” . They are collected using self-reported questionnaires, known as PRO measures. These are standardised instruments designed to capture PRO information . PROs can provide evidence on the safety and efficacy of ATMPs to support marketing authorisation and provide insight into longer-term effects on patients. In addition, the tolerability of a medicinal product (the degree to which symptomatic and non-symptomatic adverse events (AEs) associated with the product’s administration), can affect the ability or desire of the patient to adhere to the dose or intensity of therapy. A complete understanding of tolerability should include direct measurement from the patient on how they are feeling and functioning while receiving treatment. It has been proposed that ‘suitable PRO tools are selected to capture patient derived data concerning the impact of the adverse events of the therapy and the overall treatment burden for the patient’ [6, 7].
PRO data were traditionally collected using paper questionnaires; however, the use of electronic PRO systems (ePRO) is becoming increasingly common [8, 9]. The equivalence of electronic and paper administration of PRO measures has also been demonstrated . The use of an ePRO system can promote collection of complete PRO data from which conclusions on efficacy can be drawn. Research has shown that there are inconsistencies in how paper-based PROMs are administered by research staff during trials, which may reduce the quality of the resulting data and introduce potential bias . In clinical trials where PRO best practices are not upheld, publication of PRO data has been found to be undermined . Use of ePRO systems may address some of these issues through minimised administrative burden, both for patients and for trial staff; increased acceptance rates; limited secondary data entry error; and lower levels of missing data [8, 9, 13]. It has been established that use of an electronic PRO system can promote patient adherence with PRO collection in comparison to paper questionnaires [14,15,16]. Despite this, ePROs can also exacerbate health disparities in populations with little or no internet access.
Although ePRO systems for use in clinical trials are currently available, evidence suggests there is an additional need for such systems to accommodate “PRO alerts” . PRO alerts are signals that are generated by an ePRO system and sent to the trial clinical team when a patient reports psychological or physical symptoms that require an immediate response . These can be monitored and reviewed during the trial to allow timely and appropriate intervention to promote the wellbeing and safety of the patient. A system with such a feature would facilitate the systematic recording and reporting of PRO alert data, minimising co-intervention bias, promoting patient safety, and reducing liability. Such data can also be used to trigger clinician adverse event reporting. It is worth noting that PRO alerts are being increasingly used in clinical care . As a result, clinical staff have become more familiar with it recently and it has been found that in a clinical setting, symptom monitoring based upon patients’ self-reports has been found to have clinical benefits .
Usability testing is a crucial stage of PRO development [21, 22]. It is defined by the International Organisation for Standardization (ISO) as “the extent to which a product can be used by specified user to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.”  Usability testing ensures that the intended goals are achieved, with minimal use of resources, while promoting satisfaction of users, which is particularly important when users have a choice as to whether or not to use the product . In the case of patients’ completion of PRO measures on an ePRO platform, where patient burden has been established to be a reason for non-completion and missing data  despite patients’ interest in these data , usability is particularly important. As such, when assessing the usability of the ePRO system, the context of use and characteristics of those using the ePRO system must be considered [22, 26, 27].
Research context: POLARISE trial
This usability study is part of the PROmicsR study which is itself set into the context of the POLARISE Trial (ISRCTN 80103507). POLARISE (the name refers to cell polarisation) is a single arm, multi-centre, phase II basket trial investigating the safety and activity of the use of ORBCEL-C™ in the treatment of patients with Primary Sclerosing Cholangitis (PSC), Rheumatoid Arthritis (RA), Lupus Nephritis (LN) and Crohn’s Disease (CD). The aim of the POLARISE trial is to find out whether patients with primary sclerosing cholangitis (PSC), rheumatoid arthritis (RA), lupus nephritis (LN) or Crohn's disease (CD) can potentially be treated safely with a cell therapy called ORBCEL-C™ .
ORBCEL-C™ is a Mesenchymal Stromal Cells enriched MSC product isolated from human UCT (umbilical cord tissue) using CD362 + cell selection and was developed by Orbsen Therapeutics Ltd. in conjunction with NHS Blood and Transplant (NHSBT).
The aim of this study is twofold: (1) to describe how we developed an electronic platform for patients to report their symptoms, and (2) to develop and undertake usability testing of an ePRO solution for use in a study of cell therapy which seeks to provide early evidence of efficacy and tolerability of treatment and test the feasibility of the system for use in later phase studies. Usability testing was undertaken to refine the system ready for use within an ATMP clinical trial, in addition to finalising training, instructional, and consent materials.
Usability testing was approved by the University of Birmingham Research Ethics committee (ERN_19-1271).
Development of the PROmicsR ePRO system
The PROmicsR system is a configuration of Atom5™, developed by Aparito Ltd, a UK-based med-tech company. Study participants accessed a secure application (Atom5™) used by the study participants to complete the selected PRO measures on their own smartphone/tablet (iOS and Android). Atom5™ consists of two interfaces:
A clinician dashboard accessed via a web browser onto which research team members (Research Nurses) input patient and their trial information as part of users’ registration; manage users’ information; and undertake data downloads (Figs. 1 and 2);
The development of the PROmicsR ePRO platform was informed by discussion with members of a local PPI group drawn from the original disease conditions. These PPI group members met monthly for 12 months to agree on the design and feature of the ePRO platform and contributed to the system requirement document Atom5™.[29, 30]
Selection of PROs measures
PSC, RA, LN, and CD are all associated with high symptom burden and reduced QoL. The advanced therapy product may reduce inflammatory processes resulting in an improvement or sustainment of symptoms and HRQoL. All PRO analyses were exploratory due to the small sample size of the trial and provided evidence to support later stages of the development programme. Alongside data on the safety and therapeutic effect of any given advanced therapy product, further evidence is required on its impact on patient symptoms, side effects and HRQoL, both immediately after delivery of the intervention and longer-term effects. As such, the rationale for PRO measures selection was based upon trial hypotheses that the use of the product would lead to:
Improvement in fatigue and stabilisation of bowel-related symptoms for patients with PSC
Improvement in joint pain, reduced disability and fatigue for patients with RA
Improvement in fatigue and general well-being for patients with LN and CD
The findings of a systematic review of core outcome sets and recommended outcome measures across inflammatory diseases  were used to inform selection of PRO measures addressing key concepts of interest, in addition to psychometric properties, and patient preference .
Proposals for PROs were formulated and shared with the Trial Management and PPI Groups. These were reviewed in relation to trial objectives and by disease-specific patient and public involvement groups assessing alignment with patient preferences, research interests, and burden. Regulatory input was sought from the Medicines and Healthcare products Regulatory Agency (MHRA).
Outcomes measures were split between efficacy outcomes and tolerability. Efficacy outcomes measured the impact of the treatment on participants’ quality of life at pre-determined timepoints whereas tolerability outcomes measured toxicity more frequently and on an ad-hoc basis to have a better understanding of any tolerability issues in this early phase setting.
Patient-reported adverse events
As part of the POLARISE trial, existing PRO measures, item banks, and available patient-reported adverse event measures were reviewed by clinicians to be included alongside the existing trial adverse event (AE) infrastructure within the local clinical trials unit. When appropriate, recommended process for AEs were implemented within the POLARISE trial context.
Timepoints for assessment
Triallists and clinical leads were involved in selecting the timing of administration of PROs.
Timepoints for the assessment of PROs were selected based on: (1) the anticipated timescales for treatment effect; (2) PRO measure recall periods; and (3) the burden experienced by the participant and the length of time expected for its completion.
Usability testing of the PROmicsR ePRO system
Study participants and recruitment
Patients were recruited for usability testing from patient and public involvement (PPI) panels (the Liver and Gastro-Intestinal PPI Reference Group and the Birmingham Rheumatology Research Patient Partnership). After being sent the information sheets by the PPI groups leads, members of the research team (CM & SEH) contacted these participants to arrange a suitable date and time for the usability testing. Written informed consent was obtained from all the participants. In total, nine people with RA, PSC, LN or CD were recruited.
Seven Research Nurses were recruited from one trust. Similar to the PPI participants, CM and SEH contacted them to arrange a suitable date and time for the testing session after sending them information sheets and obtaining informed consent.
Testing sessions/Cognitive interviews
One-to-one testing sessions were held with research nurses and PPI participants. These sessions consisted of performing a series of specific tasks independently on the patient-facing app (PPI participants) or the clinician dashboard (Research Nurses) as well as answering debriefing questions afterwards. All testing sessions were held online and recorded via Zoom . The tasks performed by the PPI participants and the Research Nurses are detailed in Box 1.
Cognitive interviews and “think aloud” techniques were used by the researchers (CM & SEH) during the testing sessions. These techniques aim to capture participants’ thought processes while performing specific tasks with the aim of gaining an understanding of how participants understand and interact with the ePRO system [33, 34]. After completing use of the system, participants were asked specific debriefing questions about their overall experience, any difficulties they had encountered as well as any other comments they had not shared with the researcher before.
The issues encountered by the PPI participants and the research nurses as well as the researchers’ notes and observations during the testing sessions were recorded (on video and in writing) to measure the effectiveness, efficiency and satisfaction of the patient-facing app and the clinician dashboard. All data collected was anonymised.
Satisfaction questionnaires/Demographic information
In addition to cognitive interviews, the researchers collected demographic information and assessed participants’ overall satisfaction with the ePRO system with satisfaction questionnaires. The bespoke satisfaction questionnaires consisted of four questions designed to rate satisfaction with the system and its usability on a 5-point scale (1—poor or no/5—excellent or yes).
Comments from all the participants were recorded in a table which included general observations from the researcher, whether participants completed the tasks independently or with support, how they found the user experience, whether they understood the instructions and what could be done to improve the ePRO system (app and clinician dashboard). Two coders (CM & SEH) coded the comments and extracted themes. Any discrepancies between the two coders was settled by a third researcher (MC). Participant ratings for the four usability questions were used to calculate a mean score per question.
Selection of PRO measures
The PRO aim was to gather (i) preliminary evidence of the impact of treatment on HRQoL, (ii) tolerability of treatment, and (iii) specific experience of treatment-related AEs. In addition, we aimed to assess feasibility of assessment to inform the design of future trials. The following PROMs were included in the PROmicsR study (Table 1): EQ-5D-5L, Fatigue Severity Scale, FACIT-Fatigue, and PSC-PRO. Two measures of fatigue were included as PPI members believed that each measure was aimed at different types of people and were unable to decide which one would be the most suitable to include.
The Common Terminology Criteria for Adverse Events (CTCAE) is routinely used in oncology as the means of categorising AEs in trials and is a descriptive terminology that evolved alongside other toxicity grading systems to become a comprehensive, multi-modality grading system [35, 36] for reporting the acute and late effects of cancer treatment. It can also be used to assess toxicity and side effects in non-oncology early phase trials. The PRO-CTCAE™ was developed to characterise the frequency, severity and interference of 78 symptomatic treatment toxicities that could be meaningfully reported from the patient perspective. It has been established that symptomatic adverse events are under-detected by clinicians versus patients, who are better at identifying low-grade symptomatic events [37, 38]. As such, the PRO-CTCAE™ could generate information that complements CTCAE data and can provide a more holistic insight into patient experience on treatment, which is integral for clinical decision-making. In oncology, the PRO-CTCAE™ has demonstrated favourable validity, reliability, and responsiveness . However, the use of the PRO-CTCAE™ outside of oncology is novel . The PRO-CTCAE™ was selected for use in the POLARISE trial to assess patient reported symptoms and potential side effects of therapy.
Due to the early-phase nature of the trial and limited availability of safety profile data, focused evidence identification was initially employed to identify a sub-selection of PRO-CTCAE™ items . However, due to the limited AE reporting by previous studies, the use of different adjunct therapies across trials, and the interaction between underlying disease and physiology resulting in a different disease profile and a range of potential AEs, a scarcity of data was available. Therefore, we presented the entire item library to participants, along with a free text item. This approach addressed issues relating to the inconsistent AE evidence upon which to base item selection and potential channelling of patients to pre-selected items when using AE item sub-selection. This would also enable the development of an evidence base relating to the novel use of the PRO-CTCAE™ outside of oncology.
Choice of AE measures was undertaken in collaboration with the TMG, allowing for gaps in proposed AE measurement to be identified and addressed. Additional items (‘Feeling feverish’ and ‘Rectal bleeding’) were developed with the licence holders of the proposed PRO measures, with input from relevant clinical TMG members and PPI representatives in collaboration with the instrument developers.
Having identified patient-reported global tolerability as a necessary inclusion item [41, 42], the Functional Assessment of Cancer Therapy—General (FACT-G) [43, 44] was chosen as part of the AE and tolerability PRO battery. Specifically, the single item, FACT-G GP5 was used due to evidence of its validity for use outside of oncology and demonstration of its successful capture of data indicating treatment discontinuation .
Following discussion with clinical team patient partners, instrument developer and informal advice from regulatory colleagues, it was decided that a patient’s submission of a PRO-CTCAE™ and/or FACT-G GP5 report via the PROmicsR system with an absolute score of ≥ 3 and/or a PRO-CTCAE™ response of “yes” would initiate the auto-generation of single email per patient. The alert email is to be sent to relevant trial site Research Nurses and staff in the Cancer Trials Unit. Upon receipt of notification email, Research Nurses at site log on to the PROmicsR platform to view the data and validate. The data is assessed by the Research Nurse in accordance with disease group specifications and a CTCAE assessment will be undertaken as required. The assessment is recorded and retained in the patient notes and added to the CRCTU database as per protocol.
The final battery of PROMs was reviewed and agreed by the Trial Management Group and patient partners. Initial completion is estimated can be undertaken in < 30 min and subsequent entries in < 20 min. The time taken to complete the PROs electronically is recorded through use of time-stamps for data entered via the PROmicsR study app. The follow-up period is 24 months.
Timepoints for assessment
PRO-CTCAE™ and FACT-G GP5 are collected on a weekly basis, in clinic or at home with ad hoc reporting available for participants if symptoms change. The remaining PROs are collected at pre-specified timepoints over the 24-month follow-up period, depending on the disease cohort. Further details are provided in the study protocol schedule of events.
Nine PPI participants with PSC, RA, LN or CD and seven Research Nurses took part in a testing session of the PROmicsR ePRO system. Table 2 presents the PPI members’ characteristics. All participants but one reported being comfortable using mobile technology. The testing sessions lasted between 12 and 55 min.
Five participants used an iOS device and three used an Android device (one was not recorded).
Effectiveness of patient-facing ePRO app
Seven of nine participants were able to complete the testing session; two participants failed at outset due to an outdated operating system on their device (n = 1), and a compatibility issue which prevented registration (n = 1). Of the seven who were able to register, issues were identified ranging from technical issues (such as some radio buttons not working, or missing information resources) to human factors (difficulties navigating the layout) (Additional file 1: Table S1). Participants believed that the length of the PROCTCAE™ was appropriate in order to understand any tolerability issues. In addition, although some patients felt that the PROCTCAE™ questionnaire was lengthy at first, they also recognised that, depending on their symptoms, they would not have to answer each question. Further feedback will be obtained during the ongoing feasibility study.
Satisfaction with the patient-facing ePRO app
Patient participants’ satisfaction was high amongst those who were able to complete testing. These 7 participants scored it highly on ease of use, content and visual display, with all participants stating that they would be ‘likely’ or ‘very likely’ to recommend it to others (Additional file 1: Table S2). When asked to reflect on their use of the system, these participants highlighted its strengths including its simplicity and responsiveness, whilst also stating what they did not like (e.g. length of questionnaire, the use of the VAS scale) and feedback to improve its usability (e.g. insertion of a progress bar). One participant noted that ‘Seems like a good replacement to paper PROs (Participant 3)’.
The researchers noted that although the participants liked the design, layout, and colour of the training leaflet, they did not always use it during the testing sessions, preferring to consult the researchers. Participants found it challenging to simultaneously use Zoom, refer to the training manual, navigate, and complete the PROMs on the app.
Effectiveness of the clinician dashboard
Overall, the Research Nurses found the process of navigating through the clinician dashboard and completing the tasks straightforward. They encountered few issues during the process. These issues are summarised in Additional file 1: Table S3. They included not being able to log into the dashboard and not being able to create a new participant because of a duplicate participant number. After the researcher advised the research nurses to use a different internet browser and a different participant number, they were able to continue with the testing session.
During the testing sessions, the research nurses commented on other aspects of the clinician dashboard not included in the tasks, such as the presence of coloured boxes. As they were reflecting on their potential meaning, they suggested that these colours could be used to identify those participants who needed attention depending on potential alerts triggered by their answers to the PROMs. Similar to the PPI participants, CM and SEH noted that the Research Nurses did not all refer to the training manual when they were unsure how to complete a task.
Impact on the ePRO system
The usability testing sessions conducted with PPI participants and research nurses highlighted a number of minor issues that were then reported to the ePRO system developers in order to improve it and make it more accessible to users ready for deployment in the clinical trial.
This article presents the results of the development and usability testing of an ePRO system developed for use by participants with inflammatory conditions and Research Nurses in a phase II ATMP basket trial. Our ePRO system was informed by existing literature, regulatory guidance, patients, and clinicians. Four PRO measures, along with PRO-CTCAE™ and a global tolerability question were selected to be programmed onto ePRO system. The results of the cognitive interviews suggest that the PROmicsR ePRO system was a suitable way to collect PROs in the POLARISE trial. Both the PPI participants and the research nurses found the app and clinician dashboard respectively easy to use, despite a couple of participants encountering a technical issue.
Assessment of PROs in clinical trials provides valuable insights into the patient perspective and enables participants’ disclosure of severe or worsening symptoms that may require immediate clinical attention. Use of formal PRO alert systems in safety monitoring enable structured and planned responses to these data that promote patient safety while avoiding potential bias [46, 47]. Linking patient reports with symptom monitoring systems has been found to be associated with clinical benefits such as better health-related quality of life and less use of emergency medical services and hospitalisation .
While ePROs are an important step forward towards patient-centred care, we need to recognise the risks that are involved with moving from PROMs to ePROMs. In this paper, we have carefully documented this process in order to share the learning that may be applicable to others developing ePROs. The risks include privacy issues , technical difficulties , large financial investments , and issues around diversity, equity and digital exclusion [51, 52]. The latter is one of the most prominent risks and it refers to failing to reach people, either because they do not have access to digital technology (mobile phone/tablet/Internet) or because they do not know how to use it [48, 53]. The findings of our study showed that although all our participants had access to their ‘Bring Your Own Device’ (BYOD), one participant had an incompatible version of iOS and was not able to complete the testing. In addition, another one admitted not being technology savvy and asked their partner to help him download the app and navigate through it. The age (> 55) could perhaps explain the low level of digital literacy, however, this was not necessarily reflected in the rest of the other over 55 PPI participants in our study and other studies [22, 54]. Recent research has shown that using smartphones makes digital communication more accessible to people with lower levels of digital literacy . The support given to this participant by their partner during the testing session raises the issue of how much influence this external support has on the participant’s answers. We therefore recommend ensuring that participants’ devices are compatible with the ePRO system and providing an alternative solution to address digital poverty as well as ensuring inclusion, such as the provision of a device for the duration of the trial or paper-based PROMs, even if using paper-based PROMs might affect the overall reporting of PROs. Other options might be to ensure that the ePRO system is designed to be used with Smartphones and tablets or with older iterations of operating systems or to provide participants with a device, thereby not expecting them to use their own devices. Even amongst those who are not excluded, usability issues including length or content and delays in loading may lead to ePRO fatigue, which causes failure to complete and disengagement of some participants over time. Digital exclusion and digital disengagement should be taken seriously. They can be safety issues (for example if they form part of the safety monitoring of the study through notifications/alerts to the clinical team), but also can introduce significant bias, since not all parts of the target population will be represented.
It was not possible to discern alerts from usability testing as patients had not received cell therapy. This will be evaluated as part of the feasibility which is currently being conducted . Patients’ views on using the ePRO system will also be collected using qualitative methods as part of the ongoing feasibility study.
Implications for ePRO developers
Some of the participants mentioned issues with scrolling down the page to click on a ‘Submit’ button in order to submit the questionnaire and move to the next page. It is possible that the fact participants had to scroll down to see and click on the ‘Submit’ button was due to the fact that they were using a smartphone or iPhone. Although there is a definite advantage to using a portable device to report symptoms, programmers need to ensure its use is as simple and intuitive as possible for participants. This is even more true in the case of our PPI participants, some of whom have a diagnosis of RA and consequently experience difficulties with manual dexterity. Although none of them seemed to be affected by the scrolling issue, it is something that is worth bearing in mind for other people with a similar condition who might be affected by it . People affected by this could use a stylus to help them scrolling through the questions.
The fact that the PPI participants and the Research Nurses did not refer to all refer to the training manual while testing the ePRO system poses the question of its usefulness. One of the explanations for not using the manual could be because the two researchers, CM and SEH, were present and it was easier for the participants to ask questions directly to the researchers. Looking for the answer to their question in the manual would have been more time-consuming. Developers could perhaps include prompts on the screen, which might offer support and be more timesaving.
In addition, developers need to leverage accessibility options available as standard within smartphones and tablets, such as change text size or touch controls to ensure inclusivity and maximise participation.
Strengths and limitations
The main strength of our study is that the development of an ePRO system that brings together multi-sector, multi-disciplinary expertise in PROs, app development, medical regulation, and clinical expertise. In addition, the usability testing phase includes the experiences of PPI participants and of research nurses.
The methodology used which includes cognitive debriefing methods, such as “think aloud” techniques provide insights into real-time feedback and emotional responses involved in using the ePRO system which is not always accessible with more traditional qualitative data collection methods .
Finally, our study builds on a previously validated PROM (PRO-CTCAE™) and provides useful information on feasibility of assessment but only preliminary efficacy/tolerability data.
We recognise that our study has some limitations. Whilst we actively sought to sample a diverse range of users, the small sample size has limited the diversity within the user group. This means that our sample might not be representative of the whole population. One of the reasons for this is that participants were selected from Patient and Public (PPI) groups. Although participants could have been recruited from NHS sites, it would have meant obtaining NHS ethical approval and would have increased the overall length of the study, which we could not afford to do. This will be addressed in the next stage of testing and development as part of the POLARISE trial, along with other features of the PROmics ePRO platform, such as administration burden, reporting of PRO alert data, and clinical interventions based on PRO data . Due to COVID-19 restrictions, all assessments had to be done virtually. This meant that all participants had to use new technology (ePRO app and Zoom) and the printed training guide concurrently. Some participants did not refer to the training manual, preferring to ask the interviewer for clarification as they progressed through the ePRO system. Although we recorded the time participants took to complete each task, this data was not included in the analysis. Indeed, as participants stopped to ask questions whilst completing each task, we felt that the recorded time did not accurately reflect the reality and made it difficult to assess the efficiency of the ePRO system. Finally, although this study identified reasons why a motivated person might be unable to use an ePRO system (e.g. device incompatibility), this study did not explicitly look at other aspects of digital exclusion in the context of ePROs such as those who would not wish to engage with a digital process, or who do not have the physical or mental faculties to reliably use it.
Developing an ePRO system with the input from existing literature, regulatory guidance, patients, and clinicians and testing its usability testing are important steps in its design and implementation. By testing the effectiveness, efficiency, and satisfaction of our novel ePRO system (PROmicsR), we learnt that most people with an inflammatory condition found it easy to report their symptoms using an app on their own device. Their experiences using the PROmicsR ePRO system within a trial environment will be further explored in our upcoming feasibility testing. Research nurses were also positive and found the clinical dashboard easy-to-use. Using ePROs in early phase trials is important in order to provide evidence of therapeutic responses and tolerability, increase the evidence based, and inform methodology development.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Himmel ME et al (2012) Regulatory T-cell therapy for inflammatory bowel disease: more questions than answers. Immunology 136(2):115–122
Hossein-Khannazer N et al (2021) Novel cell-based therapies in inflammatory bowel diseases: the established concept, promising results. Hum Cell 34(5):1289–1300
Agency EM (2008) Guideline on safety and efficacy follow-up—risk management of advanced therapy medicinal products. London
FDA, Patient-reported outcome measures: use in medicinal product development to support labelling claims. 2009.
Fairclough D (2002) Design and analysis of quality of life studies in clinical trials. Chapman & Hall/CRC Press, Boca Raton
Agency EM (2016) Appendix 2 to the guideline on the evaluation of anticancer medicinal products in man: The use of patient-reported outcome (PRO) measures in oncology studies
Kluetz PG et al (2018) Informing the tolerability of cancer treatments using patient-reported outcome measures: summary of an FDA and critical path institute workshop. Value Health 21(6):742–747
Coons SJ et al (2009) Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force report. Value Health 12(4):419–429
Lee SJ, Kavanaugh A, Lenert L (2007) Electronic and computer-generated patient questionnaires in standard care. Best Pract Res Clin Rheumatol 21(4):637–647
Muehlhausen W et al (2015) Equivalence of electronic and paper administration of patient-reported outcome measures: a systematic review and meta-analysis of studies conducted between 2007 and 2013. Health Qual Life Outcomes 13:167
Kyte D et al (2016) Current practices in patient-reported outcome (PRO) data collection in clinical trials: a cross-sectional survey of UK trial staff and management. BMJ Open 6(10):e012281
Kyte D et al (2019) Systematic evaluation of patient-reported outcome protocol content and reporting in cancer trials. J Natl Cancer Inst 111(11):1170–1178
Velikova G et al (1999) Automated collection of quality-of-life data: a comparison of paper and computer touch-screen questionnaires. J Clin Oncol 17(3):998–1007
Bojcic JL et al (2014) Comparison of paper and electronic surveys for measuring patient-reported outcomes after anterior cruciate ligament reconstruction. Perm J 18(3):22–26
Stone AA et al (2002) Patient non-compliance with paper diaries. BMJ 324(7347):1193–1194
Stone AA et al (2003) Patient compliance with paper and electronic diaries. Control Clin Trials 24(2):182–199
Avery KNL et al (2019) Developing a real-time electronic symptom monitoring system for patients after discharge following cancer-related surgery. BMC Cancer 19(1):463
Brundage M et al (2003) Cancer patients’ preferences for communicating clinical trial quality of life information: a qualitative study. Qual Life Res 12(4):395–404
Kennedy F et al (2021) Online monitoring of patient self-reported adverse events in early phase clinical trials: views from patients, clinicians, and trial staff. Clin Trials 18(2):168–179
Basch E et al (2016) Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 34(6):557–565
Aiyegbusi OL (2020) Key methodological considerations for usability testing of electronic patient-reported outcome (ePRO) systems. Qual Life Res 29(2):325–333
Aiyegbusi OL et al (2018) Development and usability testing of an electronic patient-reported outcome measure (ePROM) system for patients with advanced chronic kidney disease. Comput Biol Med 101:120–127
20282–2:2013(en), I.T. (2013) Usability of consumer products and products for public use—Part 2: Summative test method
Mercieca-Bebber R et al (2016) Design, implementation and reporting strategies to reduce the instance and impact of missing patient-reported outcome (PRO) data: a systematic review. BMJ Open 6(6):e010938
Feldman-Stewart DB, Michael D, Tong C (2011) Information that affects patients’ treatment choices for early stage prostate cancer: a review. Can J Urol 18(6):5998–6006
Brooke J (1996) SUS: a “quick and dirty” usability scale. In: Jordan P et al (eds) Usability evaluation in industry. Taylor and Francis, London
Brooke J (2013) SUS: a retrospective. J Usability Stud 8:29–40
Hughes SE et al (2022) Feasibility of a new electronic patient-reported outcome (ePRO) system for an advanced therapy clinical trial in immune-mediated inflammatory disease (PROmics): protocol for a qualitative feasibility study. BMJ Open 12(9):e063199
Retzer A et al (2022) The value of patient-reported outcomes in early-phase clinical trials. Nat Med 28:18–20
Zbrozek A et al (2013) Validation of electronic systems to collect patient-reported outcome (PRO) data-recommendations for clinical trial teams: report of the ISPOR ePRO systems validation good research practices task force. Value Health 16(4):480–489
Aiyegbusi OL et al (2022) Outcome selection for tissue-agnostic drug trials for immune-mediated inflammatory diseases: a systematic review of core outcome sets and regulatory guidance. Trials 23(1):42
Inc., Z.V.C. [cited 2016; Available from: https://d24cgw3uvb9a9h.cloudfront.net/static/81625/doc/Zoom-Security-White-Paper.pdf.
van der Haak M, De Jong M, Schellens PJ (2010) Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav Inf Technol 22:339–351
Wolcott MD, L.N. (2021) Using cognitive interviews and think-aloud protocols to understand thought processes. Curr Pharm Teach Learn 13(2):181–188
Trotti A (2002) The evolution and application of toxicity criteria. Semin Radiat Oncol 12(1 Suppl 1):1–3
Trotti A et al (2003) CTCAE v3.0: development of a comprehensive grading system for the adverse effects of cancer treatment. Semin Radiat Oncol 13(3):176–181
Basch E et al (2006) Patient versus clinician symptom reporting using the National Cancer Institute Common Terminology Criteria for Adverse Events: results of a questionnaire-based study. Lancet Oncol 7(11):903–909
Basch E et al (2009) Adverse symptom event reporting by patients vs clinicians: relationships with clinical outcomes. J Natl Cancer Inst 101(23):1624–1632
Dueck AC et al (2015) Validity and reliability of the US National Cancer Institute’s patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). JAMA Oncol 1(8):1051–1059
Trask PC et al (2018) Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events: Methods for item selection in industry-sponsored oncology clinical trials. Clin Trials 15(6):616–623
Pearman TP et al (2018) Validity and usefulness of a single-item measure of patient-reported bother from side effects of cancer therapy. Cancer 124(5):991–997
Basch E, Campbell A, Hudgens S, Jones L et al (2018) Broadening the definition of tolerability in cancer clinical trials to better measure the patient experience. Friends of Cancer Research White Paper
Cella DF, Tulsky D, Gray G et al (1993) The Functional Assessment of Cancer Therapy (FACT) scale: Development and validation of the general measure. J Clin Oncol 11(3):570–579
Webster K, Odom L, Peterman A et al (1999) The Functional Assessment of Chronic Illness Therapy (FACIT) measurement system: validation of version 4 of the core questionnaire. Qual Life Res 8(7):604
Kaat AJ et al (2018) Physical function metric over measure: an illustration with the Patient-Reported Outcomes Measurement Information System (PROMIS) and the Functional Assessment of Cancer Therapy (FACT). Cancer 124(1):153–160
Kyte D et al (2016) Management of patient-reported outcome (PRO) alerts in clinical trials: a cross sectional survey. PLoS ONE 11(1):e0144658
Banerjee AK et al (2013) Patient-reported outcome measures in safety event reporting: PROSPER consortium guidance. Drug Saf 36(12):1129–1149
McCleary NJ et al (2013) Feasibility of computer-based self-administered cancer-specific geriatric assessment in older patients with gastrointestinal malignancy. Oncologist 18(1):64–72
Jonassaint CR et al (2015) Usability and feasibility of an mhealth intervention for monitoring and managing pain symptoms in sickle cell disease: The Sickle Cell Disease Mobile Application to Record Symptoms via Technology (SMART). Hemoglobin 39(3):162–168
Schick-Makaroff K, Molzahn A (2015) Strategies to use tablet computers for collection of electronic patient-reported outcomes. Health Qual Life Outcomes 13(2):1–6
Calvert M, Cruz Rivera S, Retzer A, Hughes S, Campbell L, Molony-Oates B, Aiyegbusi OL, Stover AM, Wilson R, McMullan C, Turner G, Davies EH, Verdi R, Velikova G, Kamudoni P, Syed M, Gheorghe A, O’Connor D, Liu X, Wu AW, Dennitson A (2022) Patient reported outcome assessment must be inclusive and equitable. Nat Med 28:1120–1124
Meirte J et al (2020) Benefits and disadvantages of electronic patient-reported outcome measures: systematic review. JMIR Perioper Med 3(1):e15588
Digital N, What we mean by digital inclusion. Available from: https://digital.nhs.uk/about-nhs-digital/our-work/digital-inclusion/what-digital-inclusion-is.
Gatto SL, Tak S (2008) Computer, internet, and email use among older adults: benefits and barriers. Educ Gerontol 34(9):800–811
Tsetsi E, Rains SA (2017) Smartphone Internet access and use: Extending the digital divide and usage gap. Mobile Media Commun 5(3):239–255
Jimison H et al (2008) Barriers and drivers of health information technology use for the elderly, chronically ill, and underserved. Evid Rep Technol Assess (Full Rep) 175:1–1422
Bergstrom J (2013) Moderating usability tests. [cited 2022 17/01/2022]; Available from: https://www.usability.gov/get-involved/blog/2013/04/moderating-usability-tests.html.
NICE, Position statement on use of the EQ-5D-5L value set for England 2019
NICE (2018) Position statement on use of the EQ-5D-5L valuation set for England (updated November 2018). Available from: https://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/technology-appraisal-guidance/eq-5d-5l
Agency EM (2017) Guideline on clinical investigation of medicinal products for the treatment of rheumatoid arthritis
Isa F et al (2018) Patient-reported outcome measures used in patients with primary sclerosing cholangitis: a systematic review. Health Qual Life Outcomes 16(1):133
ICHOM (2018) Measuring Results That Matter: Inflammatory Arthritis
Cella D (1997) The Functional Assessment of Cancer Therapy-Anemia (FACT-An) Scale: a new tool for the assessment of outcomes in cancer anemia and fatigue. Semin Hematol 34(3 Suppl 2):13–19
Younossi ZM et al (2018) Development and validation of a primary sclerosing cholangitis-specific patient-reported outcomes instrument: The PSC PRO. Hepatology 68(1):155–165
The authors wish to acknowledge the POLARISE Trial Management Group, the POLARISE Scientific Advisory Group, the PROmicsR Project Management Group, and the Liver and Gastro-Intestinal PPI Reference Group and the Birmingham Rheumatology Research Patient Partnership (R2P2) for their input and support of the project.
The views expressed are those of the authors and not necessarily those of Innovate UK.
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institute for Health Research (NIHR) Birmingham Biomedical Research Centre, Midlands-Wales Advanced Therapies Treatment Centre (MW-ATTC) programme grant from Innovate UK to a consortium of partners including Health Technology Wales, the Welsh Blood Service and the University of Birmingham (Grant number: IUK: 104232) and Innovate UK (part of UK Research and Innovation) grant Patient-reported outcomes assessment to support accelerated access to advanced cell and gene therapies: PROmicsR (Grant No: 104777). The funder was not involved in any aspect of the research work.
Ethics approval and consent to participate
Usability testing was approved by the University of Birmingham Research Ethics committee (ERN_19-1271).
Consent for publication
CM receives funding from NIHR Surgical Reconstruction and Microbiology Research Centre (SRMRC), UKRI, and declares personal fees from Aparito Ltd outside the submitted work. SEH is supported by the National Institute for Health Research (NIHR) Applied Research Centre (ARC) West Midlands at the University of Birmingham and UKRI. SEH has received personal fees from Cochlear Ltd and Aparito Ltd outside the submitted work. OLA receives funding from the National Institute for Health Research (NIHR) Birmingham Biomedical Research Centre, NIHR ARC West Midlands at the University of Birmingham, the Brain Tumour Charity, Innovate UK (part of UK Research and Innovation), Gilead Sciences Ltd. and Janssen Pharmaceuticals Inc. OLA declares personal fees from Gilead Sciences Ltd., GlaxoSmithKline (GSK) and Merck outside the submitted work. MC is a National Institute for Health Research (NIHR) Senior Investigator and receives funding from the National Institute for Health Research (NIHR) Birmingham Biomedical Research Centre, the NIHR Surgical Reconstruction and Microbiology Research Centre and NIHR ARC West Midlands at the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Health Data Research UK, UKRI, Macmillan Cancer Support, UCB Pharma, Gilead, Janssen and GSK. MC has received personal fees from Astellas, Aparito Ltd, CIS Oncology, Takeda, Merck, Daiichi Sankyo, Glaukos, GSK and the Patient-Centred Outcomes Research Institute (PCORI) outside the submitted work. DK reports grants from Macmillan Cancer Support, the NIHR, NIHR Birmingham Biomedical Research Centre, and NIHR SRMRC at the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, and personal fees from Merck and GSK outside the submitted work.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
McMullan, C., Retzer, A., Hughes, S.E. et al. Development and usability testing of an electronic patient-reported outcome (ePRO) solution for patients with inflammatory diseases in an Advanced Therapy Medicinal Product (ATMP) basket trial. J Patient Rep Outcomes 7, 98 (2023). https://doi.org/10.1186/s41687-023-00634-3