- Short report
- Open Access
Delphi survey to inform patient-reported symptom monitoring after ovarian cancer treatment
Journal of Patient-Reported Outcomes volume 4, Article number: 71 (2020)
Increasing numbers of ovarian cancer patients are living longer and requiring regular follow-up to detect disease recurrence. New models of follow-up care are needed to meet the growing number and needs of this patient group. The potential for patient-reported outcome measures (PROMs) to capture key symptoms and online technology to facilitate long-term follow-up has been suggested.
Prior to a pilot study exploring the potential for electronic patient-reported symptom monitoring, the content of an online intervention was developed via Delphi methodology.
Design and setting
A Delphi process was conducted aiming to obtain consensus amongst the clinicians and patients from 4 hospitals on the key aspects to monitor during follow-up after ovarian cancer treatment, and how to monitor them in an online intervention. A two round Delphi was conducted. Consensus was defined as at least 70% agreement.
Out of 43 participants, 30 (18 patients, 12 healthcare professionals) completed round 1 and 19 (11 patients, 8 healthcare professionals) completed round 2. Consensus was reached on the key symptoms to monitor, and the importance of monitoring both duration and frequency of symptoms. Opportunity for review of psychological wellbeing and holistic needs were considered important by both groups. The frequency of online questionnaire completion, timeframe for patients to reflect on (e.g. during the past X weeks), and the choice of PROMs items to monitor symptoms did not reach the consensus threshold.
It is crucial that any intervention and the selection of PROMs is fully described to ensure transparency about the development and decisions taken. In this work, a set of key symptoms and areas to monitor were agreed, which has informed the design of an online intervention and a subsequent pilot study is now underway. The proposed model of remote follow-up using electronic PROMs could be adapted and explored in other cancer sites.
Plain English summary
What is the key problem/issue/question this manuscript addresses?
With growing numbers of patients living longer following cancer treatment, new models of follow-up care are needed. Using technology, symptoms can be monitored remotely as part of long term follow-up, but this requires careful planning to ensure it meets patient needs.
Why is this study needed?
It is important to explore the views of patients and healthcare professionals so they can contribute to the design of any new follow-up process.
What is the main point of your study?
Before piloting a web-based system, we conducted a survey amongst patients and healthcare professionals to gather their views about the symptom-related questions we should ask, and how often patients should be answering these in order to manage symptoms/side effects and detect recurrence.
Provide a brief overview of your results and what they mean.
Patients and healthcare professionals agreed on the key symptoms to monitor, and the importance of having an opportunity to report and request support for any holistic needs following treatment. Differences of opinion were highlighted in how often symptoms should be reported. This work has informed a web-based symptom reporting system which is being piloted within a community follow-up pathway.
Ovarian cancer survival is improving, leading to an increase in patients on routine follow-up and growing pressure on clinics as incidence rates are expected to rise further . Ovarian cancer follow-up is not protocol-driven and evidence-based best practice is yet to be established . Current practice involves monitoring symptoms and serum biomarkers (CA125) through routine appointments to detect recurrence, which is likely to occur within 5 years . Evidence is scarce on the value of routine follow-up for survival or quality of life . Nama et al.  suggested that follow-up could utilise patient-reported outcome measures (PROMs) as symptoms are often present at the point of relapse. Research has also indicated that patients delay help-seeking for symptoms until their next appointment , and some gynaecological patients have high anxiety before/during appointments . More sustainable models of follow-up and long-term symptom tracking are clearly needed to ensure care provision is commensurate with patient need, rather than a one-size-fits-all service which may detract resources from those who need them most .
In recent years pilot studies have explored nurse-led telephone methods [9,10,11,12]. Most have been small qualitative studies, but they illustrate positive experiences amongst early endometrial or stage I-IV ovarian cancer patients. Cox  notes the convenience of telephone follow-up in facilitating psychosocial support and the reassurance of continued blood tests.
Recent proliferation of modern technology means the internet offers another follow-up method . Some patients prefer face-to-face appointments, but a recent UK-based randomised controlled trial illustrates the feasibility and patient value of online symptom self-reporting when clinicians are engaged . Furthermore, web-mediated follow-up (supplementary, weekly) vs. routine follow-up has demonstrated increased survival/earlier relapse detection .
To explore the feasibility of an electronic PROMs (ePROMs) follow-up pathway, the ‘electronic Patient self-Reported outcomes to Improve cancer Management and patient Experiences’ (ePRIME) study proposed an online intervention for ovarian patients following treatment. The intervention included an ePROMs symptom questionnaire and a blood test to monitor CA125 (performed at GP, local hospital or cancer centre). The clinical team interpret these results, instead of the patient attending hospital-based appointments. Prior to piloting this pathway, decisions about the intervention content and logistics required consultation, which is an important step of any intervention development . This paper describes the process undertaken to reach agreement from healthcare professionals (HCPs) and patients on the core symptoms for monitoring relapse and life after treatment, the most appropriate symptom measure/items, and the frequency of completion to inform the ePROM intervention.
A Delphi consultation methodology was chosen as it offers an iterative consensus-based approach to reach agreement ; group responses are fed back to participants allowing them to re-assess their views within multiple rounds. A ‘modified’ Delphi survey  was undertaken which allows the initial round to be informed from group discussions/interviews or a literature review, and is less prescriptive on the number of rounds required. The survey was conducted in two iterative rounds over a 10 month period (Round 1 August 2016–February 2017, Round 2 February–April 2017, but one patient completed in June 2017).
All doctors and clinical nurse specialists (CNS) working in the gynae medical oncology teams at the four participating hospitals were invited to complete the online survey (https://www.onlinesurveys.ac.uk/) via an email, including a unique username/password for access. Paper copies were provided on request.
Patients who were between 6 and 36 months post-treatment and attending routine ovarian cancer follow-up were consecutively invited as they could offer opinions on the proposed online intervention having experienced the follow-up process. Purposive sampling aimed to get a mixture of patient ages and time/experience on follow-up. Patients were approached by their clinical team, and if willing were provided with a study information sheet. Following verbal agreement patients were given a paper survey and a username/password if they preferred to complete the online version.
All participants were reminded that the survey was anonymous, and the survey return was taken as formal consent. One email/telephone reminder was undertaken.
The demographic details collected included patient age, time post-treatment, and whether this was their first diagnosis. HCPs age, gender, role, and years worked in speciality were collected.
Procedure and Delphi survey content
Ethical approval was obtained from NHS Leeds West Research Ethics Committee (ref: 16/YH/0239).
Firstly, the key symptoms to monitor during ovarian cancer follow-up were explored through informal discussions with HCPs at each hospital. A literature review was conducted to identify and select six potential PROMs for this clinical group (see Additional file 1: Appendix 1 for details of the review and the measures).
Relevant symptom items from each of the six PROMs (Additional file 1: Appendix 1, Table A1) were selected and grouped, rather than presenting the full measure (see Additional file 1: Appendix 2 for an example). Then participants rated how well the overall set of items captured the key symptoms to monitor on a 7-point Likert scale (1 = not at all, 7 = very much). The survey also explored the frequency of completion in a follow-up setting, the timeframe for patients to reflect upon (e.g. during the past X weeks), whether symptom duration/frequency/change and holistic needs should be captured.
The results from round 1 (aggregated percentages from HCPs, patients, overall) were presented, and further questions asked where opinion was diverse or new aspects had emerged since round 1.
There is no universal agreement in Delphi methodology on the percentage that constitutes ‘consensus’, with recommendations ranging from 51% to 80%, but Humphrey-Murto et al. (2017) suggested 70% is a typical cut-off  and therefore this was defined a priori as the proposed target level. A lack of consensus was defined as percentages below 70% in round 2.
The proportion of each response was also explored across the participant groups (patients, HCPs, overall). For items on a Likert scale (e.g. round 1-PROMs rating 1–7), mean scores were calculated for each measure for each group/overall, and the highest scoring PROMs were presented in round 2.
Descriptive analysis was performed using Microsoft Excel 2013.
Forty-three participants (28 patients/15 HCPs) were approached, and 30 responded (69.8% response rate), including 18/28 patients (64.3%) and 12/15 HCPs (80%). Table 1 illustrates the demographic characteristics in both rounds. Half the patients (n = 9) were 6–12 months post-treatment, whilst others were 1–3 years post-treatment, and for most patients it was their first diagnosis (n = 14). The HCPs included at least one consultant and CNS from each hospital, but more from the largest hospital (site 1).
The key symptoms agreed were: abdominal pain/discomfort, abdominal swelling/bloating, nausea, vomiting, appetite loss, change in bowel habit, urinary symptoms, shortness of breath, fatigue, swollen legs and unexpected weight change. Table 2 summarises the percentage agreement for the questions in each round.
Overall 93.3% felt it was important to monitor the duration and change in symptoms, and 86.7% agreed that symptom frequency should be explored. Eighty percent felt ongoing chemotherapy toxicity and psychological wellbeing were important.
Consensus was not achieved for how frequently patients should complete the online questionnaire, the timeframe/period that patients reflected upon (… past X weeks), and the specific holistic needs question (e.g. travel insurance, emotional). Similarly, the choice of which PROMs to use did not reach consensus, but two options (MDASI-OC; FACT-O) were removed as these were endorsed the least. HCPs also scored the PRO-CTCAE low but it was retained into round 2 due to patient endorsement.
15/18 patients wished to be contacted for round 2, and 11/15 completed (73.3%). All HCPs were re-contacted regardless if they had completed round 1. One nurse was no longer working in the speciality, but 8/14 HCPs completed (57.1%), including 5 consultants and 3 CNS (5/8 were from site 1, with one representative from the other 3 sites, see Table 1). Overall 19/29 responded (65.5%).
Three new areas were consulted (Table 2), namely: the provision of self-management advice (62.5% HCPs yes, but 45.5% patients unsure), asking about medication use (84.2% overall), and weight changes (89.5% overall).
The frequency of completion fell short of reaching agreement with 68.4% endorsing 3-monthly reports. However, the acceptability of including a checklist of holistic needs (e.g. emotional, financial, sexual, work/employment, family issues) was agreed by 100% HCPs and 81.8% patients.
Sufficient consensus was not reached on the timeframe (… past X weeks) or the preferred PROMs to use. Given the time constraints a third round could not be undertaken. Discussions were held within the project steering committee, and as around half (47.4% overall) chose ‘ … past 2 weeks’ this was selected. PROMs opinions varied widely for three measures across the participant groups (PRAE 42.1% - 5 patients/3 HCPs; MOST 26.3% - 1 patient/4 HCPs; PRO-CTCAE 21.1% - 4 patients) (Table 2). Further consultation selected the PRAE as this measure was endorsed most overall, particularly by patients who were prioritised as the intended online system users.
Figure 2 visually presents the symptom monitoring intervention.
This work achieved its aim of exploring the key symptoms to monitor during ovarian cancer follow-up, and gathering HCP/patient views of the frequency/proposed content of an online follow-up intervention. The value of remote online methods to facilitate follow-up has become increasingly relevant during/following the COVID-19 pandemic where the reduction of face-to-face hospital-based appointments was essential. This has led to calls for rapid introduction of electronic PROMs in cancer clinical practice  and the continued importance of providing support for cancer survivors during/following the pandemic .
For most areas, 70% consensus was achieved within two rounds, but some major differences were observed within and between groups. We hope this report provides transparency of the process  to enable others to learn from our experiences.
Most HCPs felt 3-monthly was an appropriate frequency (in line with clinical practice), but patients were divided between monthly and 3-monthly reports. For the ePRIME pilot 3-monthly was chosen but the system is open-access should patients wish to complete earlier. This work highlights the importance and value of consulting both patient and HCPs equally when developing interventions [22, 23].
Furthermore, rather than a purely remote system whereby patients report online without clinical contact, free-text comments and discussions with both groups led to the decision that a telephone-based appointment would allow a CNS to formally review online completions and blood results. Therefore, this format is being evaluated in the ongoing pilot.
Patients and HCPs placed great value on reporting/reviewing holistic needs, adding to the evidence that holistic needs assessment is integral to delivering person-centred care . Within the ePRIME system we specifically ask about non-medical holistic needs, and we would urge others to continue prioritising this important area in research and clinical practice, especially if follow-up care is being delivered remotely .
The overall survey response rates in both rounds were ~ 65%, and non-response/attrition reasons were not collected. Selection bias is possible as non-responders may have contrasting views of the proposed intervention. Furthermore, those who dropped out may be different, for example less older patients (66+ years) completed round 2. However individuals were consecutively approached, there was a variety of demographics and hospital sites represented across both rounds.
Consensus was not reached on the choice of which PROMs to use, with views divided across patients and HCPs. However, as discussed the level of percentage agreement is disputed in the literature, and the stability of responses through rounds may be more valuable . To our knowledge there is no official guidance for how to proceed in such circumstances. In hindsight a third round or a formal consensus meeting inviting all participants may have been beneficial , but with limited time/resources this was resolved through discussion within the project steering committee . It may be that Delphi methodology is not the most appropriate method of choosing between similar validated measures.
It is imperative that intervention development and the selection of PROMs for trials/research is explicitly described to ensure transparency about the decision-making and confirm the appropriateness and content validity of the intervention . The views of patients and HCPs have informed the development of the ePRIME follow-up intervention which is being piloted in two hospitals in the North of England. If successful, the model of remote follow-up pathway and the ePROM system could be adapted to other cancer sites.
Availability of data and materials
The anonymised datasets collected during the current study are stored at the Patient Centred Outcomes Research group. These datasets are not publicly available, but the corresponding author may consider reasonable requests.
Clinical nurse specialist
Electronic Patient self-Reported outcomes to Improve cancer Management and patient Experiences
Electronic patient-reported outcome measures
Patient Reported Adverse Event
Patient-reported outcome measures
National Cancer Institute Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events
Cancer Research UK. Ovarian cancer statistics. 2020. Available from: https://www.cancerresearchuk.org/health-professional/cancer-statistics/statistics-by-cancer-type/ovarian-cancer. Accessed 14 Jan 2020.
Salani, R., et al. (2017). An update on post-treatment surveillance and diagnosis of recurrence in women with gynecologic malignancies: Society of Gynecologic Oncology (SGO) recommendations. Gynecologic Oncology, 146(1), 3–10.
Marcus, C. S., et al. (2014). Current approaches and challenges in managing and monitoring treatment response in ovarian cancer. Journal of Cancer, 5(1), 25–30.
Clarke, T., et al. (2014). Evaluation of follow-up strategies for patients with epithelial ovarian cancer following completion of primary treatment. Cochrane Database of Systematic Reviews, (9), CD006119. https://doi.org/10.1002/14651858.CD006119.pub3.
Nama, V., Nordin, A., & Bryant, A. (2013). Patient-reported outcome measures for follow-up after gynaecological cancer treatment. Cochrane Database of Systematic Reviews, 2013(11), CD010299.
Olaitan, A., et al. (2001). A critical evaluation of current protocols for the follow-up of women treated for gynecological malignancies: A pilot study. International Journal of Gynecologic Cancer, 11(5), 349–353.
Kew, F. M., Galaal, K., & Manderville, H. (2009). Patients’ views of follow-up after treatment for gynaecological cancer. Journal of Obstetrics and Gynaecology, 29(2), 135–142.
MacMillan Cancer Support, One size doesn’t fit all: Why we need to make cancer care in the NHS more personal. 2018.
Beaver, K., et al. (2017). Comparing hospital and telephone follow-up for patients treated for stage-I endometrial cancer (ENDCAT trial): A randomised, multicentre, non-inferiority trial. BJOG: An International Journal of Obstetrics & Gynaecology, 124(1), 150–160.
Cox, A., & Faithfull, S. (2015). Aiding a reassertion of self: A qualitative study of the views and experiences of women with ovarian cancer receiving long-term nurse-led telephone follow-up. Supportive Care in Cancer, 23(8), 2357–2364.
Williamson, S., et al. (2018). Telephone follow-up after treatment for endometrial cancer: A qualitative study of patients’ and clinical nurse specialists’ experiences in the ENDCAT trial. European Journal of Oncology Nursing, 34, 61–67.
Morrison, V., et al. (2018). Trial of Optimal Personalised Care After Treatment-Gynaecological Cancer (TOPCAT-G) a randomized feasibility trial. International Journal of Gynecological Cancer, 28(2), 401–411.
Dickinson, R., et al. (2014). Using technology to deliver cancer follow-up: A systematic review. BMC Cancer, 14, 311.
Warrington, L., et al. (2019). “You know it’s not just a waste of time, somebody’s looking at it.” Patient engagement with online symptom reports during chemotherapy (eRAPID)-The importance of clinician use of data. Psycho-Oncology, 28, 7.
Denis, F., et al. (2017). Randomized trial comparing a web-mediated follow-up with routine surveillance in lung cancer patients. Journal of the National Cancer Institute, 109(9). https://doi.org/10.1093/jnci/djx029.
Hoddinott, P. (2015). A new era for intervention development studies. Pilot and Feasibility Studies, 1, 36.
Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008–1015.
Hasson, F., & Keeney, S. (2011). Enhancing rigour in the Delphi technique research. Technological Forecasting and Social Change, 78(9), 1695–1704.
Humphrey-Murto, S., et al. (2017). Using consensus group methods such as Delphi and nominal group in medical education research. Medical Teacher, 39(1), 14–19.
Marandino, L., et al. (2020). COVID-19 emergency and the need to speed up the adoption of electronic patient-reported outcomes in cancer clinical practice. JCO Oncology Practice, 16(6), 295-298.
Nekhlyudov, L., et al. (2020). Addressing the needs of cancer survivors during the COVID-19 pandemic. J Cancer Surviv Res Pract, 1–6. Advance online publication. https://doi.org/10.1007/s11764-020-00884-w.
MacLennan, S., et al. (2017). A core outcome set for localised prostate cancer effectiveness trials. BJU International, 120(5B), E64–E79.
Penedo, F. J., et al. (2020). The increasing value of eHealth in the delivery of patient-centred cancer care. The Lancet Oncology, 21(5), e240–e251.
Sandsund, C., et al. (2017). Holistic needs assessment and care plans for women with gynaecological cancer: Do they improve cancer-specific health-related quality of life? A randomised controlled trial using mixed methods. BMJ Supportive & Palliative Care, 10(2), e16.
Sigurdardottir, K. R., et al. (2014). The European Association for Palliative Care basic dataset to describe a palliative care cancer population: Results from an international Delphi process. Palliative Medicine, 28(6), 463–473.
We thank all the participants who took part in this Delphi process and the overall project’s steering committee members for their guidance and advice.
This study presents independent research funded by Yorkshire Cancer Research, grant reference L392, “ePRIME Yorkshire - electronic Patient self-Reported outcomes to Improve cancer Management and patient Experiences”. The charity had no influence on the data collection, interpretation or reporting.
Ethics approval and consent to participate
This Delphi survey was part of a larger project exploring electronic patient self-reported outcomes. The overall project protocol (including the Delphi) was reviewed and approved by NHS Leeds West Research Ethics Committee (ref: 16/YH/0239) and individual NHS hospital approval. Verbal consent plus the return of the survey (online or paper) was taken as formal consent.
Consent for publication
The authors declare that they have no competing interests.
About this article
Cite this article
Shearsmith, L., Kennedy, F., Lindner, O.C. et al. Delphi survey to inform patient-reported symptom monitoring after ovarian cancer treatment. J Patient Rep Outcomes 4, 71 (2020). https://doi.org/10.1186/s41687-020-00237-2
- Ovarian cancer
- Follow-up care