Constructing Learning Curves to Benchmark Operative Performance of General Surgery Residents Against a National Cohort of Peers

Williams RG, Sanfey H, Chen X, Dunnington GL. Ann Surg. 2012 July

Joseph D.Nicolas BA, Reiping Huang PhD, Ezra N.Teitelbaum MD, MEd, Karl Y. Bilimoria MD, MS, Yue-Yung Hu MD, MPH,

Journal of Surgical Education, Nov-Dec 2020

Abstract

OBJECTIVE:

No method or data exist to allow surgical trainees or their programs to contextualize their technical progress. The objective of this study was to create peer benchmarks for Cumulative Sum (CUSUM) charts based upon operative evaluations from a national cohort of general surgery residents.

DESIGN, SETTING, PARTICIPANTS:

In 2016-2018, faculty from 26 general surgery residency programs nationwide rated 328 residents’ operative performance on a case-by-case basis using a validated 5-point Likert scale. An individual case was considered a “misstep” if scoring below the national median score for that procedure in that postgraduate year (PGY). We constructed 2-sided observed-expected CUSUM charts to capture each resident’s cumulative performance over time relative to the national medians. Upper (failure) and lower (positive outlier) benchmarks were established based on the PGY-specific 75th percentile and median misstep rates; consistent/repeated missteps are reflected by crossing of the upper boundary. Procedures with ≤10 observations and residents who were evaluated <10 times for each PGY were excluded.

RESULTS:

Around 8,161 evaluations on 76 procedure types were analyzed. The individual misstep rate was lowest among PGY-3s at 13.3% and highest among PGY-4s at 28.6%. No interns had curves that crossed the failure boundary. 8.7% of PGY-2s and 8.9% of PGY-3s finished the year past the failure boundary. PGY-2s had the most positive outliers, with 28.3% of them demonstrating an outlying success performance beyond the lower boundary for at least once. PGY-5s most frequently failed, with 16.7% ever crossing the upper boundary and 11.1% remaining above it at graduation.

CONCLUSIONS:

CUSUM is a valid statistical approach for benchmarking individual residents’ operative performance against national peers as they progress through the year in real-time. With further validation, CUSUM could be used to set progression and/or graduation standards and objectively identify residents who might benefit from remediation.