Journal of Surgical Education, 2021
Workplace-based assessments (WBAs) are used in multiple surgical specialties to facilitate feedback to residents as a form of formative assessment. The validity evidence to support this purpose is limited and has yet to include investigations of how users interpret the assess- ment and make rating decisions (response processes). This study aimed to explore the validity evidence based on response processes for a WBA in surgery.
Semi-structured interviews explored the rea- sonings and strategies used when answering questions in a surgical WBA, the System for Improving and Measur- ing Procedural Learning (SIMPL). Interview questions probed the interpretation of the three assessment ques- tions and their respective answer categories (level of autonomy, operative performance, case complexity). Researchers analyzed transcripts using directed qualita- tive content analysis to generate themes.
Single tertiary academic medical center.
Eight residents and 13 faculty within the Department of Otolaryngology—Head and Neck Sur- gery participating in a 6-month pilot of SIMPL.
We identified four overarching themes that that characterized faculty and resident response processes while completing SIMPL: (1) Faculty and resident users had similar content-level interpretations of the questions and corresponding answer choices; (2) Users employed a variety of cognitive, behavioral, and emotional processes to make rating decisions; (3) Contextual factors influenced ratings; and (4) Tensions during interpretation contributed to rating uncertainty.
Response processes are a key source of evidence to support the validity for the formative use of WBAs. Evaluating response process evidence should go beyond basic content-level analysis as contextual factors and tensions that arise during interpretation also play a large role in rating decisions. Additional work and a con- tinued critical lens are needed to ensure that WBAs can truly meet the needs for formative assessment. ( J Surg Ed 000:1!10. ! 2021 Published by Elsevier Inc. on behalf of Association of Program Directors in Surgery.)
The Society for Improving Medical Professional Learning (SIMPL) is a 501c3 nonprofit research collaborative lead by volunteers who are dedicated to working towards a continuously improving medical education system.