Skip to main content

Content validity and ePRO usability of the BPI-sf and “worst pain” item with pleural and peritoneal mesothelioma



The Brief Pain Inventory-short form (BPI-sf) is widely used in self-reported pain assessment, incorporates pain numeric rating scales (NRS) and is commonly utilized in electronic format in clinical trials, however, there is no published information about its usability as an electronic patient-reported outcome (ePRO) measure. The objective of this qualitative study was threefold: 1) to better understand pain experiences among patients with pleural or peritoneal mesothelioma; 2) to assess the interpretability of the instructions, item stem, recall period, and response option of the “worst pain” item of the BPI-sf; and 3) to examine the usability of the TrialMax Touch™ (CRF Health, Inc., Plymouth Meeting, PA) screen-based handheld device and the electronic format of the BPI-sf in a sub-sample of pleural mesothelioma patients.


A cross-sectional qualitative study was conducted among participants with pleural and peritoneal mesothelioma recruited from 4 clinical sites in the US. Semi-structured telephone or in-person interviews were conducted consisting of concept elicitation, cognitive interviewing of the 11-item BPI-sf, and in-person interview evaluation of ePRO assessment usability in pleural mesothelioma patients.


Twenty-one participants recruited from 4 clinical sites in the US were interviewed in-person (n = 9) and by telephone (n = 12); 71% male; mean age 68.7 ± 13.6 years. Pleural and peritoneal patients described pain as ranging from discomfort to intense pain and reported being able to distinguish tumor pain from treatment pain. The BPI-sf “worst pain” item was relevant to, and easily understood by, study participants with pleural and peritoneal mesothelioma. The ePRO version was found to be easy to use, but readability of small font may be an issue. Participants reported minimal differences between their responses on the paper and ePRO version for all of the pain severity and pain interference items.


Results support the relevance and ease of understanding of the “worst pain” item and provide support for its content validity in patients with pleural and peritoneal mesothelioma. Usability of the ePRO format of the BPI-sf was confirmed for use in clinical trials among patients with pleural mesothelioma.


Mesothelioma is a rare malignant tumor originating from the cells lining the surface of the coelomic cavities of the body, specifically the pleura, peritoneum, pericardium, and tunica vaginalis testis. Pleural mesothelioma is the most common anatomical site (67–75%) of presentation, followed by peritoneal (25–33%) [1]. Mesothelioma is mainly caused by occupational exposure to asbestos (e.g., mining, industry), and as a result, is most prevalent in men. Due to the long latency of the onset of disease after exposure to asbestos (20–40 years), cases are typically detected at an advanced age (median 72 years) and advanced disease stage [2, 3]. The incidence of malignant pleural mesothelioma (MPM) is leveling off in the United States, because asbestos use has decreased since the 1970s [4], however, it is increasing in other countries, such as Russia, Western Europe, China, and India [5].

Clinical practice guidelines acknowledge that mesothelioma has a high symptom burden including pain, fatigue, dyspnea, insomnia, cough, and anorexia [6]. Studies indicate that many patients with pleural mesothelioma report pain and/or dyspnea [7]. Patients typically present with shortness of breath due to pleural effusion or chest pain in a more advanced stage [6]. Patient pain symptoms in mesothelioma are frequently complex due to a combination of nociceptive, neuropathic, and inflammatory factors [8].

Few patient-reported instruments have been used in evaluating mesothelioma treatments. Some that have been used include the Lung Cancer Symptom Scale - Mesothelioma (LCSS-Meso) [9, 10], and the Brief Pain Inventory - short form (BPI-sf) [11]. The BPI-sf is a widely used, self-administered instrument developed to assess the severity of pain and the impact of pain on functioning [11]. The BPI-sf was originally derived from the BPI - long form which was developed to assess cancer-related pain; the BPI-sf is widely used and has evidence in support of its validity across a range of therapeutic conditions [12,13,14]. The BPI-sf includes front and back body diagrams, 4 pain severity items, 7 pain interference items, and an item for rating pain relief from analgesics. The 4 items of pain severity domain assess pain at its “worst,” “least,” “average,” and “now” (current pain) on an 11-point numeric rating scale (NRS) from 0 (No Pain) to 10 (Pain as bad as you can imagine). The BPI-sf uses a 24-h recall period and is designed to assess two domains: Pain Severity and Pain Interference. The BPI “worst pain” item is a commonly used endpoint for measuring pain and has been included in a number of drug labels in the US and Europe [15]. The evidence of the content validity of the BPI-sf and the “worst pain” item has not been established in patients with mesothelioma.

While electronic versions of the BPI-sf including interactive voice response (IVR) have been used in numerous studies, no published information is available about its usability in electronic screen-based format [16]. Usability testing, and/or equivalence testing may be required to confirm that the migration from the paper to electronic version has been faithful and the new implementation captures the same data as the original. In cases where there are significant changes in format when an instrument is adapted from paper to electronic format, equivalence testing might be required [17, 18]. However, as in the case of this study, if the change in format between the paper version and the ePRO version is considered to be minor only cognitive interviews and usability testing are typically conducted. This study was conducted to support the content validity of the BPI “worst pain” item for use in clinical trials of mesothelioma that may be seeking a label claim for pain improvement.


Study design and objectives

This cross-sectional qualitative study included concept elicitation as well as cognitive interviewing components to achieve the following three objectives: 1) to better understand pain experiences among patients with pleural or peritoneal mesothelioma; 2) to assess the interpretability of the instructions, item stem, recall period, and response option of the “worst pain” item of the BPI-sf; and 3) to examine the usability of the TrialMax Touch™ (CRF Health, Inc., Plymouth Meeting, PA) electronic format of the BPI-sf in a sub-sample of pleural mesothelioma patients (ePRO screen shots included as Additional file 1). The study was approved by local Institutional Review Boards at each clinical study site.

Recruitment procedures

Participants were recruited from 4 US clinical sites and interviewed in-person or by telephone (for study participants who had peritoneal mesothelioma or were unable to come to the clinic). Participants included those who had: clinician confirmed diagnosis of mesothelioma, previous or current chemotherapy treatment, Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1, and were 18 years of age or older. It was not a requirement that study participants be symptomatic to participate in the study. All participants provided informed consent prior to participating in the interview and were remunerated a nominal amount for their time.

Interview procedures

The interviews were conducted by qualitative interviewers (AS, ZB) with ten years of combined experience in conducting qualitative interviews. The interviews were scheduled to last 90 min. If time was constrained due to patient illness or time limitation due to treatment appointment or patient time constraints, the concept elicitation portion and the cognitive interview review of a separate questionnaire and the BPI-sf “worst” pain, “average” pain items, and the usability of the handheld device were priority questions.

In-person interviews

The in-person interviews with pleural mesothelioma participants included an evaluation of the usability of the electronic version of the BPI-sf. During the concept elicitation portion of the interview, participants were asked open-ended questions designed to elicit spontaneous descriptions of their mesothelioma symptoms. Next, the study interviewers (ZB, AS) reviewed the instructions for completing the paper version of the BPI-sf before instructing the participant to complete the questionnaire.

Participants completed a brief training on the TrialMax™ handheld device. During the training interviewers provided any needed additional explanation as to how to tap the screen to make a selection, and how to save the data. Once participants felt comfortable with the instructions for operating the electronic device, participants were asked to complete the screen-based format of the BPI-sf. Following the completion of the electronic version of the BPI-sf, the interviewer commenced the cognitive interview to evaluate participants’ comprehension and interpretation of the BPI-sf “worst pain” item (item 3) and “average pain” (item 5) and the appropriateness and ease of use of the response scales and recall periods in relation to their experience with mesothelioma. During the cognitive interview portion of the interview, pleural participants were asked to describe differences in pain concepts, such as “worst pain,” “average pain,” and “general pain.”

In the final stage of the cognitive interview, participants were asked questions related to the experience of answering questions on the screen-based handheld device, as well as additional questions focusing on the participants’ feedback on the comparison of the paper and electronic versions of the BPI-sf.

Telephone interviews

The telephone interviews included concept elicitation and cognitive interview of the paper version of the BPI-sf, but did not include an ePRO component. Participants were mailed a packet with study materials relevant for review during the interview. During the concept elicitation portion of the interview, participants were asked open-ended questions designed to elicit spontaneous descriptions of their mesothelioma symptoms.

Following the concept elicitation, the study interviewers reviewed the instructions for completing the paper-version of the BPI-sf before instructing the participant to complete the questionnaire. Following the completion of the BPI-sf, the interviewer commenced the cognitive interview evaluation of the participant’s comprehension and interpretation of the BPI-sf “worst pain” item (item 3) and “average pain” (item 5) and the appropriateness and ease of use of the response scales and recall periods in relation to their experience with mesothelioma. During the cognitive interview portion of the interview, participants were asked to describe differences in pain concepts, such as “worst pain,” “average pain,” and “general pain.”

Data analysis

The project team developed a coding dictionary structured on the interview guide and based on the themes and concepts that emerged during the concept elicitation and cognitive interview discussions. One of the study team members (ZB) was responsible for supervising another coder. The coder was trained on the coding dictionary and then both ZB and the coder coded the same transcript and reviewed and reconciled their differences to improve standardized use of codes. The study project manager (AS) refereed discrepancies in coding to reconcile final coding decisions. A content analysis approach was used to analyze qualitative data from the interviews, using ATLAS.ti version 7.1.8 qualitative data analysis software [19]. Concept elicitation data were analyzed separately for participants with pleural and peritoneal mesothelioma.

Participant feedback from the evaluation of the screen-based handheld device was evaluated to determine its usability and the similarities and differences in responses between the paper and electronic formats of the BPI-sf. Descriptive statistics were used to characterize the study sample with respect to demographic and clinical characteristics using SAS version 9.1 (SAS Institute, Cary, NC).


Demographic and clinical characteristics

A total of 21 participants recruited from four clinical sites, representing both large and small medical centers in Florida (n = 1), New York (n = 1) and Texas (n = 2), participated in this cross-sectional qualitative study, which included interviews conducted in-person (n = 9) and by telephone (n = 12) between September 2013 and June 2014. The sample included 17 participants with pleural mesothelioma, three with peritoneal mesothelioma, and one with both pleural and peritoneal mesothelioma.

Study participants were 68.7 ± 13.6 years of age (range 22.0 to 84.3 years), and the majority were male (71%). Most participants were White (n = 20, 95%), retired (n = 14, 67%), and living with a partner, family or friend (n = 17, 81%). Education level varied across the sample (Table 1). Participants had a mean of 1.2 (SD = 1.5) years since time of their mesothelioma diagnosis, and most were Stage III (25%) or Stage IV (60%) according to clinician diagnosis (Table 2). Eighty-six percent of the participants were on active treatment. All of the pleural participants (n = 18) reported undergoing treatment for their disease: primarily chemotherapy and/or surgery.

Table 1 Participant self-reported sociodemographic and clinical characteristics
Table 2 Clinician-reported participant characteristics

Concept elicitation results

Forty-three percent (n = 9 of 21) of the overall sample described pain resulting from their mesothelioma, with the majority of these spontaneously reporting pain when asked about symptoms resulting from their mesothelioma. The location of pain varied, with participants mentioning the chest, lungs, and side, pain in the bones, or pain in general. Participants reported being able to differentiate the pain resulting from surgery and treatment compared to the pain resulting from mesothelioma. Words participants used to describe their experience of pain included “discomfort,” “twinge,” a “spot of pain,” “sharp,” and “intense.” When asked how often they experienced pain, several participants indicated the symptom was at least a daily occurrence, with others characterizing their pain as “constant.” Four participants reported that they experienced pain seldom or “every now and again,” with degree and duration ranging from an “instant twinge” to a constant ache. Of the participants who were asked about the impact of pain, several reported that their pain was bothersome.

001-001: It’s just when it hits direct…it can be pretty rough. You know, maybe a …maybe a 7 [out of 10]. …the next time it might be a 4. …It’s almost like—you was in a fist fight and they drilled you in the chest…That’s almost how it feels.

001-002: it was only painful when I laid down and it was a mild pain but …it didn’t even go away and so that’s what got my attention that … I shouldn’t be having pain when I lay on that side.

Cognitive interview results: BPI-sf “worst pain” item

Instructions, recall period and response options

All of the participants in the overall sample reported a clear understanding of the instructions for completing the “worst pain” and “average pain” items. There were no comprehension issues related to the recall period of the past 24 h. All participants asked correctly applied the past 24-h recall in evaluating worst pain and average pain, many stating that the question was “easy” when asked whether the question was “easy or difficult to answer” in a 24-h time period. All of the participants had a clear understanding of rating pain using a NRS with many responding that it was “easy” to select a numerical value for their level of pain.

BPI-sf “worst pain” item stem

The concept of “worst pain,” as assessed by item 3 of the BPI-sf, was explored in several sections of the cognitive interview and in relation to the assessment of the ease of interpretability and usability of the electronic version of the BPI-sf. Participants were asked to describe how they would rate their pain within the past 24 h, including “general pain,” “worst pain,” as assessed by BPI-sf item 3, and “average pain,” as assessed by BPI-sf item 5. When asked to provide a “think aloud” response to how they interpreted the BPI-sf “worst pain” item as it was read to them during the interview, a majority of pleural participants and all peritoneal participants found the item easy to answer and focused on pain that was their most severe or intense in the previous 24-h period (Table 3).

I would mark 0 if I had no pain at all. I would mark 10 if the pain was so intense that I was unable to essentially perform my life, you know, perform the tasks in my life (004-006).

So, what do you think about rating your “worst pain” over a 24 hour period?

“Oh, I would imagine you’d want to know how—how severe it could possibly be, right? You’re treating the severity, not just the fact that you have some pain. You want to know to what degree, so I get that.” (002-006)

Only one-third of the pleural participants were able to describe how they would rate their “worst pain”, but reported that they were not currently experiencing pain due to mesothelioma at the time of the interview. Amongst the pleural study participants not experiencing pain, the pain item was initially described as “difficult to respond to and explain,” and some described how they felt the last time they felt pain:

So, can you talk me through how you would answer a question about “worst pain”?

“I’d probably try to remember like the worst amount of pain that I had lately and like remember like when I was like, ow that hurts. And then be like, oh that sucked, and then I’d mark it down. And then, I’d try to like kind of compare it to like the “worst pain” ever. Like I don’t like giving very high numbers on pain stuff because I feel like a 10 should be like you’re basically dead from pain. So, [laughter], um, that being said, like I generally try to rate things on the lower scale because I don’t think I’ve ever experienced—like I feel like a 10 is something that I have not experienced yet because I feel like I’d die if I experienced a 10.” (003-004)

“Uh, so if you’d asked me two days ago maybe I would’ve said my pain in my stomach was a—a 2 or a 3 as a constant—a constant pain.” (002-010)

In order to better understand how participants conceptualized their “worst pain,” a subset of the sample was asked to compare and contrast this concept with “general pain” and “average pain.” Most of those participants said these questions were assessing different concepts and that “worst pain” was distinctive from “average” or “general” pain. A number of participants indicated that they would respond similarly to the two questions because “worst pain” and “general pain” tended to be comparable for them within a 24-h period.

“Um, on average I would say that like you just kind of think of like your everyday like day-to—like how it is generally like most of the time versus like the parts where it like hurts super bad.” (003-004)

Participants were asked to use their own words to explain “average” pain. Of the participants who were asked, all of these participants talked about trying to mentally calculate their worst and least pain to find an average of their pain. These participants noted that the concept of “worst pain” was more important and straightforward to assess rather than “average pain” because the intensity or severity could be rated more easily.

“…most of the time the pain is average and every once in a while, to me, I don’t know about other guys or other people, ah, sometimes I get sharp – sharp shooting pains, stabbing pains. Sometimes it lasts a minute, sometimes it lasts five minutes.” (002-005)

In your opinion, you see no difference between ‘average’ versus ‘worst’ or least?

Well there—there’s a mathematical difference, right? But—but if I say, what was your “worst pain” in the last 24 hours, and you say, it’s this, and, what was your least pain in the last 24 hours, and you say, that. And I—I—I essentially think Question 5 is a waste of space because you could calculate the average based on those two—those two answers (004-006).

All of the pleural and peritoneal participants who were asked stated that they correctly applied the past 24 h recall in evaluating “worst pain” and were able to easily use the NRS. These same participants reported no difficulty when asked whether the question was “easy or difficult to answer” based on a 24 h recall period.

Table 3 Item Mapping of participant quotations to BPI-sf “Worst Pain” (Item 3): Pleural participants (n = 19)

Usability of the electronic version

Nine pleural participants (including 1 participant with both pleural and peritoneal mesothelioma) participated in the usability portion of the study, including six men and three women, ranging in age from 22 to 84 years of age (mean 67 ± 20 years).

More than half of the pleural participants reported previous experience with technology, like computers and cell phones, and the remaining reported no or very little experience with technology. All of the participants indicated that they had no prior experience using an ePRO or similar handheld device to complete a questionnaire. When asked to provide their overall impressions about using the screen-based handheld device, nearly all participants reported positive impressions about the handheld device, including that it was “convenient,” “handy,” “fast,” or “easy” to use, and none described any problems using the handheld device.

“The questions were simple, direct. So, I didn’t think it was difficult at all.” (001-004)

What were your overall impressions about using this diary to answer questions? Uh, I like it actually. It’s handy. (003-004)

The one participant who reported a negative impression about the electronic format stated that the font size was too small to be read easily. When asked about the ease of reading text on the handheld device, most participants reported that it was easy to do so. Three participants indicated that a larger font would improve ease of reading for BPI-sf items.

It could probably be bigger—a little bit bigger. But I had to put my glasses on, but I mean [Laughter] at 69 you’ve got to wear glasses anyway. (004-001)

Would you change anything about the text that appears on the screen? If I didn’t have my glasses, the font could be a little larger. (002-002)

All participants reported that it was” quick,”” no problem, “or “easy” to move through and navigate the electronic format using the screen-based handheld device. Participants reported being able to easily change their answer when necessary. When participants were asked if they experienced any difficulty entering the answer that they wanted for any of the questions on the BPI-sf, all but one of the participants reported no difficulties. The one participant reporting difficulties stated that he did not understand how to answer the body diagram question on the BPI-sf and would have liked to have had more explanation of how to complete the ePRO version.

And how was it moving from screen-to-screen?

Easy, just—just touch ‘Next’. [laughter] As long as I can read N-E-X-T I can go to the next one. [laughter] And back is--goes back to the--back to—back to the previous one. I mean, that’s not hard. (002-012)

When prompted about what might improve the electronic format, suggestions from participants included having a bigger screen, having a brighter screen, having a stylus, and improving the screen sensitivity in response to finger taps.

Electronic format evaluation

After discussing their overall impressions of the screen-based handheld device, participants were asked to provide feedback about the experience of completing the paper-based and electronic formats of the BPI-sf. The majority of participants selected the same response on both the paper and electronic formats for each of the BPI-sf pain severity and impact questions (Table 4). One participant noted that for item 2, which assesses the location of pain via a body diagram, his ePRO answer was different from the paper format. This the participant selected multiple locations on the paper format of the body diagram to indicate areas of pain, but it was not clear to him how to select more than one location on the electronic format. Training on how to select multiple locations of pain is recommended in order to ensure this item is completed correctly on the handheld device.

Table 4 Comparison of Responses on BPI-sf Electronic vs. Paper Formats: Pleural Participants (n = 9)

There were no other differences noted by participants in appearance and usability of the formats that may have affected how they thought about a given question or selected an answer on the BPI-sf, although participants equally expressed a preference of one format over another.

I think the electronic is more to the point with its questions. It’s just, you know, boom. It’s just there. Where [the paper version] has got a little bit more to read or to look at where that one’s just more simplified. (001-004)


This study aimed to explore the pain experience in pleural and peritoneal mesothelioma and the interpretation of the concepts captured in the BPI-sf such as “worst pain.” The BPI-sf is one of the most commonly used instruments for pain assessment, however, no previous evidence had been collected directly from patients with mesothelioma on how they experience their pain.

The concept elicitation results suggest that the study participants experience discomfort to intense pain and can distinguish mesothelioma pain from treatment pain. Study participants readily distinguished between the “average” vs. “worst” pain concepts of the BPI-sf. In particular, support was found for the concept of “worst pain,” which patients consistently interpreted using words that reflected the item’s intended meaning and may be the more reliable assessment of pain severity. Furthermore, both participants with pleural and peritoneal mesothelioma found the “worst pain” item of the BPI-sf easy to interpret and respond to using a 24-h recall period. The authors could not identify any other study examining equivalence of paper and electronic version of “worst pain” NRS [20, 21].

An additional objective of the study was to assess the usability of the electronic version of the BPI-sf which has not previously been assessed among patients with MPM and to our knowledge, is the first qualitative study to evaluate the usability of the electronic format of the BPI-sf on a screen-based handheld device. The study results are consistent with prior qualitative studies supporting the content validity of the BPI-sf in patients with cancer-related pain. Gater and colleagues [22] conducted concept elicitation interviews followed by cognitive interviews in a sample of patients with castration-resistant prostate cancer with bone metastases focusing on the BPI-sf and other pain measures. Similarly, in exploring how patients interpret the concepts of “worst,” “average,” and “current” pain in a sample of patients with bone metastases, Harris and colleagues [23] found that patients tended to consider their “worst pain” even when asked to rate “average pain”, further highlighting the reliability of the “worst pain” item. Shi and colleagues [24] explored whether current pain, “worst pain”, or pain recalled from the past week better represented cancer patients’ overall experience. Findings suggested that “worst pain” recalled over the past week contributed most to the Pain Interference score, and that current pain tended to capture less severe pain than “worst” or “average” pain recalled from the past week.

Qualitative work has also been conducted focusing solely on the “worst pain” item in 26 patients (mean age = 68, range = 44–81) with metastatic castrate resistant prostate cancer from a non-randomized Phase II study expansion cohort. Interviews were conducted to assess the comprehension of the item as well as to elicit input on the response options on the 0–10 response scale to determine the intra-patient pain rating consistency. The findings indicated that the item is well understood and that the response scale is consistent among patients, with “2” describing relatively mild, noticeable and not limiting pain, “5” describing pain that is moderate and limits activity, and “8” as severe pain that is more or less incapacitating [16].

Results from the current study suggest that the ePRO format is easily used by participants with minimal experience with technology, as evidenced by relatively few response differences between paper and electronic versions, when appropriate participant training is provided. These results are consistent with other electronic diary usability studies conducted with older adults which found no significant issues with ease of use of electronic formats [25, 26]. There is some indication that additional training and use of larger font size may be important considerations for electronic devices used with older adults [27].

A strength of the current study is that it included 21 mesothelioma patients from geographically diverse regions of the US and from multiple medical centers. Additionally, the level of evidence on migration from paper-based to electronic format is consistent with that recommended by the ISPOR ePRO Good Research Practices Task Force Reports [17, 18]. In the case of this study, the adaptation of the TrialMax™ from the paper-based format was considered minor and therefore only qualitative evidence gathered via cognitive interview and usability testing was determined to be necessary. As outlined in best practice documents, small changes made to the format of an instrument during adaptation to electronic format do not require equivalence testing [18, 28]. The similarities found in responses between the paper and electronic versions of the BPI-sf in this study are supported by a systematic review by Gwaltney and colleagues, which identified equivalent scores produced by computer and paper measures [29].

The results of the study should be interpreted in light of the following limitations. First, participants with ECOG of 0 (asymptomatic) or 1 (symptomatic but ambulatory, able to perform non-strenuous activity) were included to match the clinical trial sample. An investigation of the BPI-sf among study participants with more severe, active disease was not within the scope of this study, and therefore the extent to which the results of the current study are generalizable to this more severe population are unknown. In addition, some of the participants in the current study may not have had as many symptoms as would be expected in the context of a treatment study. Pain was less commonly described by pleural participants in this study sample than may be expected for this disease (44% of those with pleural mesothelioma; 50% of those with peritoneal mesothelioma). Only a small number of participants with peritoneal mesothelioma were identified and interviewed, one of whom had both pleural and peritoneal mesothelioma. More research is needed to better understand the experience of pain in patients with different forms of mesothelioma. Finally, during the cognitive interview of the BPI-sf the paper version was always presented before the electronic version which may have introduced systematic bias towards participants comprehension of the BPI-sf items or experience with the device.

Future work on BPI-sf electronic device format may consider larger font size or improved contrast or brightness of the screen to improve readability. In addition, investigators should encourage ePRO vendors to include options to shade and select multiple pain locations on electronic versions of the BPI-sf figures so as to enhance pain evaluation in future clinical trials.


In summary, findings from this study provide strong support for the content validity of the BPI-sf in paper and electronic format for use in patients with pleural and peritoneal mesothelioma and add to existing support for the “worst pain” item as a key outcome measure in clinical trials with pleural mesothelioma patients.



Brief Pain Inventory - short form


Eastern Cooperative Oncology Group


electronic patient-reported outcome


Interactive voice response


Lung Cancer Symptom Scale - Mesothelioma


Malignant pleural mesothelioma


Numeric rating scale


  1. van Meerbeeck, J. P., Scherpereel, A., Surmont, V. F., & Baas, P. (2011). Malignant pleural mesothelioma: The standard of care and challenges for future management. Crit Rev Oncol Hematol, 78(2), 92–111.

    Article  PubMed  Google Scholar 

  2. Lanphear, B. P., & Buncher, C. R. (1992). Latent period for malignant mesothelioma of occupational origin. Journal of occupational medicine: official publication of the Industrial Medical Association, 34(7), 718–721.

    CAS  Google Scholar 

  3. Selikoff, I. J., Hammond, E. C., & Seidman, H. (1980). Latency of asbestos disease among insulation workers in the United States and Canada. Cancer, 46(12), 2736–2740.

    Article  CAS  Google Scholar 

  4. Larson, T., Melnikova, N., Davis, S. I., & Jamison, P. (2007). Incidence and descriptive epidemiology of mesothelioma in the United States, 1999-2002. Int J Occup Environ Health, 13(4), 398–403.

    Article  PubMed  Google Scholar 

  5. Park, E. K., Takahashi, K., Hoshuyama, T., Cheng, T. J., Delgermaa, V., Le, G. V., & Sorahan, T. (2011). Global magnitude of reported and unreported mesothelioma. Environ Health Perspect, 119(4), 514–518.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Stahel, R. A., Weder, W., Lievens, Y., Felip, E., & ESMO Guidelines Working Group. (2010). Malignant pleural mesothelioma: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Annals of oncology: official journal of the European Society for Medical Oncology, 21(Suppl 5), v126–v128.

    Article  Google Scholar 

  7. Moore, A. J., Parker, R. J., & Wiggins, J. (2008). Malignant mesothelioma. Orphanet journal of rare diseases, 3, 34.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Scherpereel, A., Astoul, P., Baas, P., Berghmans, T., Clayson, H., de Vuyst, P., Dienemann, H., Galateau-Salle, F., Hennequin, C., Hillerdal, G., Le Pechoux, C., Mutti, L., Pairon, J. C., Stahel, R., van Houtte, P., van Meerbeeck, J., Waller, D., Weder, W., & European Respiratory Society/European Society of Thoracic Surgeons Task Force. (2010). Guidelines of the European respiratory society and the European society of thoracic surgeons for the management of malignant pleural mesothelioma. Eur Respir J, 35(3), 479–495.

    Article  CAS  Google Scholar 

  9. Hollen, P. J., Gralla, R. J., Liepa, A. M., Symanowski, J. T., & Rusthoven, J. J. (2004). Adapting the lung cancer symptom scale (LCSS) to mesothelioma: Using the lcss-meso conceptual model for validation. Cancer, 101(3), 587–595.

    Article  PubMed  Google Scholar 

  10. Hollen, P. J., Gralla, R. J., Liepa, A. M., Symanowski, J. T., & Rusthoven, J. J. (2006). Measuring quality of life in patients with pleural mesothelioma using a modified version of the lung cancer symptom scale (LCSS): Psychometric properties of the LCSS-Meso. Supportive care in cancer : official journal of the Multinational Association of Supportive Care in Cancer, 14(1), 11–21.

    Article  Google Scholar 

  11. Cleeland, C. S. (2006). The measurement of pain from metastatic bone disease: Capturing the patient's experience. Clinical cancer research : an official journal of the American Association for Cancer Research, 12(20 Pt 2), 6236s–6242s.

    Article  Google Scholar 

  12. Song, C. Y., Lin, S. F., Huang, C. Y., Wu, H. C., Chen, C. H., & Hsieh, C. L. (2016). Validation of the brief pain inventory in patients with low back pain. Spine, 41(15), E937–E942.

    Article  PubMed  Google Scholar 

  13. Ehde, D. M., Nitsch, K. P., & Smiley, J. P. (2015). Measurement characteristics and clinical utility of the brief pain inventory-short form for individuals with multiple sclerosis. Rehabilitation psychology, 60(4), 365–366.

    Article  PubMed  Google Scholar 

  14. Naegeli, A. N., Tomaszewski, E. L., & Al Sawah, S. (2015). Psychometric validation of the brief pain inventory-short form in patients with systemic lupus erythematosus in the United States. Lupus, 24(13), 1377–1383.

    Article  CAS  PubMed  Google Scholar 

  15. Clark, M. J., Harris, N., Griebsch, I., Kaschinski, D., & Copley-Merriman, C. (2014). Patient-reported outcome labeling claims and measurement approach for metastatic castration-resistant prostate cancer treatments in the United States and European Union. Health Qual Life Outcomes, 12, 104.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Bennett, A., Eremenco, S., et al. (2013). Mode equivalence of interactive voice response (IVR) and paper versions of the brief pain inventory (BPI) worst pain item in metastatic castrate resistant prostate cancer (MCRPC) evaluated conceptually using qualitative methods. Value Health, 16(7), A601.

    Article  Google Scholar 

  17. Eremenco, S., Coons, S. J., Paty, J., Coyne, K., Bennett, A. V., Mc Entegart, D., & ISPOR PRO Mixed Modes Task Force. (2014). PRO data collection in clinical trials using mixed modes: Report of the ISPOR PRO mixed modes good research practices task force. Value Health, 17(5), 501–516.

    Article  PubMed  Google Scholar 

  18. Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., Lenderking, W. R., Cella, D., Basch, E., & ISPOR ePRO Task Force. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value Health, 12(4), 419–429.

    Article  PubMed  Google Scholar 

  19. Friese, S., & Ringmayr, T. (2013). Atlas.Ti 7 user guide and reference. Berlin: ATLAS.ti Scientific Software Development GmBH.

    Google Scholar 

  20. Marcano Belisario, J. S., Jamsek, J., Huckvale, K., O'Donoghue, J., Morrison, C. P., & Car, J. (2015). Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods. The Cochrane database of systematic reviews, 7, MR000042.

    Article  Google Scholar 

  21. Muehlhausen, W., Doll, H., Quadri, N., Fordham, B., O'Donohoe, P., Dogar, N., & Wild, D. J. (2015). Equivalence of electronic and paper administration of patient-reported outcome measures: A systematic review and meta-analysis of studies conducted between 2007 and 2013. Health Qual Life Outcomes, 13, 167.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Gater, A., Abetz-Webb, L., Battersby, C., Parasuraman, B., McIntosh, S., Nathan, F., & Piault, E. C. (2011). Pain in castration-resistant prostate cancer with bone metastases: A qualitative study. Health Qual Life Outcomes, 9, 88.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Harris, K., Li, K., Flynn, C., & Chow, E. (2007). Worst, average or current pain in the brief pain inventory: Which should be used to calculate the response to palliative radiotherapy in patients with bone metastases? Clin Oncol, 19(7), 523–527.

    Article  CAS  Google Scholar 

  24. Shi, Q., Wang, X. S., Mendoza, T. R., Pandya, K. J., & Cleeland, C. S. (2009). Assessing persistent cancer pain: A comparison of current pain ratings and pain recalled from the past week. J Pain Symptom Manag, 37(2), 168–174.

    Article  Google Scholar 

  25. Bellamy, N., Wilson, C., Hendrikz, J., Whitehouse, S. L., Patel, B., Dennison, S., Davis, T., & Group EDCS. (2011). Osteoarthritis index delivered by mobile phone (m-WOMAC) is valid, reliable, and responsive. J Clin Epidemiol, 64(2), 182–190.

    Article  PubMed  Google Scholar 

  26. Leidy, N. K., Wilcox, T. K., Jones, P. W., Murray, L., Winnette, R., Howard, K., Petrillo, J., Powers, J., Sethi, S., & EXACT-PRO Study Group. (2010). Development of the exacerbations of chronic obstructive pulmonary disease tool (EXACT): A patient-reported outcome (PRO) measure. Value Health, 13(8), 965–975.

    Article  PubMed  Google Scholar 

  27. Siek, K. A., Rogers, Y., & Connelly, K. H. (2005). Fat finger worries: How older and younger users physically interact with PDAs. In M. F. Costabile & F. Paternò (Eds.), Human-computer interaction - interact 2005. Interact 2005. Lecture notes in computer science, vol 3585. Berlin, Heidelberg: Springer.

    Google Scholar 

  28. Getz, K. A. (2006). The imperative to support site adoption of EDC. Appl Clin Trials, 15, 38–40.

    Google Scholar 

  29. Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value Health, 11(2), 322–333.

    Article  PubMed  Google Scholar 

Download references


This work was supported by funding from AstraZeneca. The authors thank Ren Yu for data analysis and Fritz Hamme for production assistance. Authors gratefully acknowledge the support patients who participated in this survey and clinical staff who assisted with recruitment.


Funding for this manuscript and research was provided by AstraZeneca.

Availability of data and materials

Data will not be shared. The conclusions of the paper are based on qualitative data. For confidentiality reasons, the full interview transcripts will not be shared.

Author information

Authors and Affiliations



HG, AS, ZB and SE research project: Conception, Organization and Execution. CS, PC and KH research project: Organization and Execution. CS, HG, AS, ZB and SE statistical analysis: Design and Execution. PC and KH statistical analysis: Review and Critique. CS, HG, AS, PC and SE manuscript: Writing of the first draft; ZB, KH manuscript: Review and Critique. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Heather L. Gelhorn.

Ethics declarations

Ethics approval and consent to participate

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This study was approved by MD Anderson 2013–0913; Roswll Park Cancer Institute 239,113; Scott & White 130,030; Moffitt Cancer Center 17,444.

Consent for publication

Not applicable.

Competing interests

Dr. Halling is currently employed and Ms. Cimms formerly employed by AstraZeneca. Dr. Gelhorn, Ms. Skalicky, and Ms. Balantac are current salaried employees and Dr. Sexton and Ms. Eremenco are former employees of Evidera, a consulting company, which has received funding from AZ for conducting the study. They are precluded from receiving payment/honoraria directly from companies for which Evidera consults. The authors have no other relevant affiliations or financial involvement with any organization or entity with financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:

Screen Shots of Electronic Version of Instrument. (DOCX 295 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gelhorn, H.L., Eremenco, S., Skalicky, A.M. et al. Content validity and ePRO usability of the BPI-sf and “worst pain” item with pleural and peritoneal mesothelioma. J Patient Rep Outcomes 2, 16 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: