Are You Thinking What I’m Thinking? Exploring Response Process Validity Evidence for a Workplace-based Assessment for Operative Feedback

Nina W. Zhao, MD, MAEd, Lindsey M. Haddock, MD, MAEd, and Bridget C. O’Brien, PhD

Journal of Surgical Education, 2021

Abstract

 

OBJECTIVE:

Workplace-based assessments (WBAs) are used in multiple surgical specialties to facilitate feedback to residents as a form of formative assessment. The validity evidence to support this purpose is limited and has yet to include investigations of how users interpret the assess- ment and make rating decisions (response processes). This study aimed to explore the validity evidence based on response processes for a WBA in surgery.

 

DESIGN:

Semi-structured interviews explored the rea- sonings and strategies used when answering questions in a surgical WBA, the System for Improving and Measur- ing Procedural Learning (SIMPL). Interview questions probed the interpretation of the three assessment ques- tions and their respective answer categories (level of autonomy, operative performance, case complexity). Researchers analyzed transcripts using directed qualita- tive content analysis to generate themes.

 

SETTING:

Single tertiary academic medical center.

 

PARTICIPANTS:

Eight residents and 13 faculty within the Department of Otolaryngology—Head and Neck Sur- gery participating in a 6-month pilot of SIMPL.

 

RESULTS:

We identified four overarching themes that that characterized faculty and resident response processes while completing SIMPL: (1) Faculty and resident users had similar content-level interpretations of the questions and corresponding answer choices; (2) Users employed a variety of cognitive, behavioral, and emotional processes to make rating decisions; (3) Contextual factors influenced ratings; and (4) Tensions during interpretation contributed to rating uncertainty.

 

CONCLUSIONS:

Response processes are a key source of evidence to support the validity for the formative use of WBAs. Evaluating response process evidence should go beyond basic content-level analysis as contextual factors and tensions that arise during interpretation also play a large role in rating decisions. Additional work and a con- tinued critical lens are needed to ensure that WBAs can truly meet the needs for formative assessment. ( J Surg Ed 000:1!10. ! 2021 Published by Elsevier Inc. on behalf of Association of Program Directors in Surgery.)

 

Link: https://doi.org/10.1016/j.jsurg.2021.09.007