Table 4 Panelists’ evaluation of the standard setting process
Strongly agree | Agree | Disagree/ Strongly Disagree | |
|---|---|---|---|
1. I understood the purpose of the standard setting exercise. | 62.5% | 37.5% | 0% |
2. The instructions and explanations provided by the facilitator were clear. | 25% | 62.5% | 12.5% |
3. The training on the standard setting method gave me the information I needed to complete my assignment. | 25% | 62.5% | 12.5% |
4. The Performance Descriptions that were developed prior to the meeting were accurate. | 25% | 62.5% | 12.5% |
5. I understood the concept of the borderline patient. | 37.5% | 62.5% | 0% |
6. The Performance Descriptions helped me determine how to rate each item. | 25% | 62.5% | 12.5% |
7. It was beneficial to have an opportunity for discussion and to review feedback. | 75% | 25% | 0% |
8. The opportunity to provide a second round of ratings (i.e., Round 2) helped me feel more confident about my final ratings. | 87.5% | 12.5% | 0% |
9. I felt engaged in the process. | 87.5% | 12.5% | 0% |
10. I felt comfortable sharing my ideas with the other panelists during the discussions. | 100% | 0% | 0% |
11. I am confident this standard setting process will produce fair cut scores. | 62.5% | 37.5% | 0% |
12. I would be comfortable defending this process to my peers. | 50% | 50% | 0% |
Very influential | Somewhat influential | Not influential | |
13. My perception of the severity of symptoms that the items were measuring | 87.5% | 12.5% | 0% |
14. The Performance Descriptions | 37.5% | 50% | 12.5% |
15. The average ratings of other panelists | 25% | 62.5% | 12.5% |
16. Group discussion after Round 1 | 50% | 50% | 0% |
17. My experience with patients | 75% | 25% | 0% |
Very useful | Somewhat useful | Not useful | |
18. Going through PowerPoint training slides prior to beginning the actual rating task | 25% | 75% | 0% |
19. Referencing the Performance Descriptions | 62.5% | 37.5% | 0% |
20. Group discussion after Round 1 | 100% | 0% | 0% |
Too much time | About right | Too little time | |
21. Reviewing the Performance Descriptions | 12.5% | 87.5% | 0% |
22. Round 1 of the rating task | 0% | 87.5% | 12.5% |
23. Group discussion after Round 1 to achieve consensus | 12.5% | 75% | 0% |