Quantifying the effect email reminders have on patient reported outcome measure returns in a large prostate cancer registry
Journal of Patient-Reported Outcomes volume 6, Article number: 19 (2022)
Systematic collection of patient-reported outcome measures (PROMs) enables researchers to understand trends in clinical care, measure their effects on patient outcomes, and potentially devise interventions to optimise those outcomes . There are impediments to widespread population-based use of PROM surveys. PROM completion is subject to various patient-level factors such as perceived irrelevance of the survey instrument or that inordinate amounts of time required to complete the PROM, difficulty with an electronic mode of completion, and a variety of other phenomena (forgetfulness, differing priorities, procrastination) [2,3,4]. Therefore, strategies to maximise patient engagement and the completion of PROMs instruments are necessary for collection of robust and unbiased outcomes data . Previous studies have found that reminder emails are effective in increasing the response rate to web-based survey instruments [6, 7]. In a clinical trial setting, automatic reminders appear likely to positively impact completion and timeliness of PROM submission .
We sought to quantify the improvement in PROM return rate that a reminder email confers in a large population-based registry setting, without the explicit opt-in consent and active monitoring present in a trial. The Victorian Prostate Cancer Outcomes Registry (PCOR-Vic) is a population-wide clinical quality registry enrolling men with newly diagnosed prostate cancer . As of 2021, coverage is greater than 80% of the incident diagnoses in Victoria, Australia. It operates with an opt-off consent model, i.e., patients are notified that they will be entered into the registry unless they request not to be. PCOR-Vic administers the validated PROM instrument EPIC-26  (Expanded Prostate Cancer Index Composite—26) to patients approximately one year after their initial prostate cancer diagnosis/treatment, predominantly via email contact. Further, we aimed to explore the effect on response rate of patient age at PROM, socioeconomic status, and the day of the week the initial contact email was sent.
Data was extracted from the Victorian Prostate Cancer Outcomes Registry (PCOR-Vic), details of which have been previously described . Approximately 12 months after treatment, registry staff make contact via telephone with patients and invite them to answer the EPIC-26 questionnaire. This instrument measures symptoms in the domains of urinary incontinence, irritation, bowel function, sexual function, and vitality. Three methods of completion are offered: by telephone immediately with the staff member, or to be sent the instrument by email or mail for self-completion. The option to complete the EPIC-26 instrument by email was introduced in April 2018, and currently, the majority of patients opt to use this mode. If a response is not received within seven days, an automatic reminder email is then sent the next day. If the patient has still not returned the instrument after a further week, they will be contacted by telephone and reminded to complete the instrument. Responses collected electronically are automatically entered into the registry database.
Patients that elected to receive the EPIC-26 by email from April 2018 to March 2021 inclusive, and have a recorded collection date, formed the analytic sample for this study. The cumulative percentage of electronically returned and answered questionnaires per ordinal day following initial contact was calculated and plotted. Questionnaires returned completely unanswered, or those subsequently completed by telephone or mail, were not counted in the numerator of the cumulative return percentage. A four-parameter logistic curve was fitted to the overall plot to predict the completion rate in the absence of a reminder email. To examine the return rate over time, and the effect of a reminder on this rate, plots reflecting the change in cumulative returned percentage, relative to two days prior, were also generated i.e. [(percentage_dayn minus percentage_dayn-2) / percentage_dayn-2]. These figures were also generated for the subgroups of patient age, socioeconomic quartile of patient residence, and day of the week of initial email contact. Socioeconomic quartile within the state of Victoria was derived from Australian Bureau of Statistics correspondence to the patient's residential postcode . Further exploration of predictors of electronic completion by day 22 was performed with multivariable logistic regression.
Within the three-year study period, 5065 patients elected to receive and were sent the EPIC-26 PROM instrument via email (and had a recorded return date); 4673 (92%) returned a completed instrument by email (176 returned the instrument unanswered, 207 completed by telephone, 9 completed by mail), 4456 (88%) within 22 days of initial email contact (Additional file 1: figure S1).
A clear uptick in cumulative returns was noted following the 8th day reminder email (Fig. 1) with the relative increase from the 8th day to the 10th day being 9.3% (Table 1). Utilising the four-parameter logistic regression curve, overall completions at the 22nd day were predicted to be 78.5% without the reminders, 9.5% lower than the absolute proportion observed. Prominent immediate increases in the rate of return after the 8th day were noted for older men aged 75 and over (10.7%) and those who had an initial email sent on a Monday and if required, a reminder scheduled on Tuesday (12.7%) (Fig. 2). Considerably shallower increases were observed in those with an initial email sent on Friday (reminder email scheduled on a Saturday) (6.7%) and men aged under 55 (6.3%), though these younger men appeared more receptive to a phone call reminder at approximately two weeks. No clear association was noted for socioeconomic status.
Return proportion by the 22nd day was very similar across all subgroups (86.3%—89.3%) with the exception of patients aged under 55 (83.4%) (Additional file 1: Figure S2). In a multivariable model, age category was predictive of response with the 55–64-year-old (OR = 1.58; 95%CI: 1.13 – 2.21) and 65–74-year-old (OR = 1.60; 95%CI: 1.16 – 2.20) categories both more likely to return the questionnaire than the youngest age category (Table 2).
PCOR-Victoria is a large registry that covers more than 80% of prostate cancer diagnoses in Victoria, Australia. The opt-off model ensures high population coverage, but also minimises the contact between patients and registry staff who do not have a role in clinical management. Thus, patients may not be as invested in answering the PROM compared to if it were administered by their treating clinician or in a trial context. Hence strategies to increase response have been built into the registry operations. An email delivered a full seven days after the original email reminding men to complete the EPIC-26 instrument was generally effective in increasing PROM returns across patient age, socioeconomic status, and day-of-the-week the reminder was sent. However, some differences in the magnitude of effect were observed with respect to patient age.
Email reminders are a simple and effective way of enhancing the response rate to patient reported outcome measures instruments [12, 13]. A study examined the effect of an automatic reminder email delivered within a month of the PROMs due date in men with prostate cancer enrolled in clinical trials . This increased the response rate for the prostate cancer instrument from 75.7% to 79.7%. Given the programmed nature of the email reminder, the additional cost to the program administering the PROM is minimal compared to the large increase in returns. Furthermore, the automated prompt for men to self-report symptomatology may mean they answer in a more truthful way . An additional benefit is that this approach decreases the error rate in collection; patients are entering data directly into the database system, rather than a member of the registry staff talking to a patient, interpreting what they have said, and then entering the data from a telephone conversation. Nevertheless, for those remaining non-responders, a telephone reminder at approximately 2 weeks after initial contact is part of the registry procedure. This contributes a small but measurable increase in the relative cumulative return rate, particularly for men aged under 65, and is consistent with prior studies [15, 16]. While it is suggested that phone reminders are not cost effective , the strategy of an initial reminder email reduces the amount of non-responders and consequently the amount of follow-up registry staff hours.
A limitation of this work is that our study population is older, exclusively male, and more likely to reside in socioeconomically advantaged areas. This is reflective of the risk profile for prostate cancer in our state  though it limits the generalisability of this research to other patient groups. Another limitation is that there was no control group, hence we have had to predict the effect of not receiving an automatic reminder. The precise reasons patients did not respond to the initial email were not explored in this study. It may be that the email was erroneously put into the patient’s spam folder, they were busy at the time of receipt, or they simply forgot about the email and the reminder email provided the impetus to complete the PROM. Patient specific factors such as amount of free time to answer such surveys were not measured with ancillary questions and these may partially explain the results observed. Further research to elucidate such factors may help improve the response rate to surveys in general.
In conclusion, an automated email reminder increased the response rate to a PROM instrument by almost 10% in a population-based cancer registry. We would encourage the use of this method as an effective and inexpensive strategy to increase the completion rate of email administered PROM instruments.
Availability of data and materials
The datasets analysed during the current study are not publicly available as per the patient consent form.
Patient reported outcome measures
Expanded prostate cancer index composite
Prostate cancer outcomes registry-Victoria
Basch E, Barbera L, Kerrigan CL, Velikova G (2018) Implementation of patient-reported outcomes in routine medical care. Am Soc Clin Oncol Educ Book 38:122–134. https://doi.org/10.1200/EDBK_200383
Judson TJ, Bennett AV, Rogak LJ, Sit L, Barz A, Kris MG et al (2013) Feasibility of long-term patient self-reporting of toxicities from home via the Internet during routine chemotherapy. J Clin Oncol 31(20):2580–2585. https://doi.org/10.1200/JCO.2012.47.6804
Nguyen H, Butow P, Dhillon H, Sundaresan P (2021) A review of the barriers to using patient-reported outcomes (PROs) and patient-reported outcome measures (PROMs) in routine cancer care. J Med Radiat Sci 68(2):186–195. https://doi.org/10.1002/jmrs.421
Baeksted C, Pappot H, Nissen A, Hjollund NH, Mitchell SA, Basch E et al (2017) Feasibility and acceptability of electronic symptom surveillance with clinician feedback using the patient-reported outcomes version of common terminology criteria for adverse events (PRO-CTCAE) in Danish prostate cancer patients. J Patient Rep Outcomes 1(1):1. https://doi.org/10.1186/s41687-017-0005-6
Nielsen LK, King M, Moller S, Jarden M, Andersen CL, Frederiksen H et al (2020) Strategies to improve patient-reported outcome completion rates in longitudinal studies. Qual Life Res 29(2):335–346. https://doi.org/10.1007/s11136-019-02304-8
Archer T (2007) Characteristics associated with increasing the response rates of web-based surveys. Pract Assess Res Eval 12(1):12. https://doi.org/10.7275/ec5p-zg50
Balter KA, Balter O, Fondell E, Lagerros YT (2005) Web-based and mailed questionnaires: a comparison of response rates and compliance. Epidemiology 16(4):577–579. https://doi.org/10.1097/01.ede.0000164553.16591.4b
Pugh SL, Rodgers JP, Moughan J, Bonanni R, Boparai J, Chen RC et al (2021) Do reminder emails and past due notifications improve patient completion and institutional data submission for patient-reported outcome measures? Qual Life Res 30(1):81–89. https://doi.org/10.1007/s11136-020-02613-3
Evans SM, Millar JL, Wood JM, Davis ID, Bolton D, Giles GG, et al. The Prostate Cancer Registry: monitoring patterns and quality of care for men diagnosed with prostate cancer. BJU Int. 2013;111(4 Pt B):E158–66. doi: https://doi.org/10.1111/j.1464-410X.2012.11530.x.
Szymanski KM, Wei JT, Dunn RL, Sanda MG (2010) Development and validation of an abbreviated version of the expanded prostate cancer index composite instrument for measuring health-related quality of life among prostate cancer survivors. Urology 76(5):1245–1250. https://doi.org/10.1016/j.urology.2010.01.027
Statistics ABo. 1270.0. 55.005—Australian statistical geography standard (ASGS): Volume 5—Remoteness structure, July 2016. Australian Bureau of Statistics Canberra, Australia; 2018.
Palmen LN, Schrier JC, Scholten R, Jansen JH, Koeter S (2016) Is it too early to move to full electronic PROM data collection? A randomized controlled trial comparing PROM’s after hallux valgus captured by e-mail, traditional mail and telephone. Foot Ankle Surg 22(1):46–49. https://doi.org/10.1016/j.fas.2015.05.001
Triplet JJ, Momoh E, Kurowicki J, Villarroel LD, Law TY, Levy JC (2017) E-mail reminders improve completion rates of patient-reported outcome measures. JSES Open Access 1(1):25–28. https://doi.org/10.1016/j.jses.2017.03.002
Chang EM, Gillespie EF, Shaverdian N (2019) Truthfulness in patient-reported outcomes: factors affecting patients’ responses and impact on data quality. Patient Relat Outcome Meas 10:171–186. https://doi.org/10.2147/PROM.S178344
Udby CL, Riis A, Thomsen JL, Rolving N (2021) Does the use of telephone reminders to increase survey response rates affect outcome estimates? An ancillary analysis of a prospective cohort study of patients with low back pain. BMC Musculoskelet Disord 22(1):893. https://doi.org/10.1186/s12891-021-04787-4
Neve OM, van Benthem PPG, Stiggelbout AM, Hensen EF (2021) Response rate of patient reported outcomes: the delivery method matters. BMC Med Res Methodol 21(1):220. https://doi.org/10.1186/s12874-021-01419-2
Breen CL, Shakeshaft AP, Doran CM, Sanson-Fisher RW, Mattick RP (2010) Cost-effectiveness of follow-up contact for a postal survey: a randomised controlled trial. Aust N Z J Public Health 34(5):508–512. https://doi.org/10.1111/j.1753-6405.2010.00598.x
Wang LL, Begashaw K, Evans M, Earnest A, Evans SM, Millar JL et al (2018) Patterns of care and outcomes for men diagnosed with prostate cancer in Victoria: an update. ANZ J Surg 88(10):1037–1042. https://doi.org/10.1111/ans.14722
The authors wish to thank the registry staff and the men enrolled in the Prostate Cancer Outcomes Registry – Victoria.
Prostate Cancer Outcomes Registry – Victoria is supported by Movember. The funder had no role in the design, performance, analysis, or decision to publish this work.
Ethics approval and consent to participate
This project was approved by the Alfred Health Human Research Ethics Committee (Approval Number: HREC/16/Alfred/98).
Consent for publication
No individually identifiable person is included in this study.
None of the authors declare any relevant conflicts of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Papa, N., Bensley, J.G., Hall, K. et al. Quantifying the effect email reminders have on patient reported outcome measure returns in a large prostate cancer registry. J Patient Rep Outcomes 6, 19 (2022). https://doi.org/10.1186/s41687-022-00426-1