Natural Language Processing and Assessment of Resident Feedback Quality
Quintin P. Solano, BS, Laura Hayward, BS, Zoey Chopra, BA, Kathryn Quanstrom, BA,
Daniel Kendrick, MD, Kenneth L. Abbott, MD, MS, Marcus Kunzmann, AB, Samantha Ahle, MD, MHS, Mary Schuller, MSEd, Erkin Ötles, MSE, and Brian C. George, MD, MAEd
Journal of Surgical Education, 2021
Abstract
OBJECTIVE:
To validate the performance of a natural lan- guage processing (NLP) model in characterizing the quality of feedback provided to surgical trainees.
DESIGN:
Narrative surgical resident feedback transcripts were collected from a large academic institution and classified for quality by trained coders. 75% of classified transcripts were used to train a logistic regression NLP model and 25% were used for testing the model. The NLP model was trained by uploading classified tran- scripts and tested using unclassified transcripts. The model then classified those transcripts into dichoto- mized high- and low- quality ratings. Model performance was primarily assessed in terms of accuracy and second- ary performance measures including sensitivity, specific- ity, and area under the receiver operating characteristic curve (AUROC).
SETTING:
A surgical residency program based in a large academic medical center.
PARTICIPANTS:
All surgical residents who received feedback via the Society for Improving Medical Profes- sional Learning smartphone application (SIMPL, Boston, MA) in August 2019.
RESULTS:
The model classified the quality (high vs. low) of 2,416 narrative feedback transcripts with an accuracy of 0.83 (95% confidence interval: 0.80, 0.86), sensitivity of 0.37 (0.33, 0.45), specificity of 0.97 (0.96, 0.98), and an area under the receiver operating characteristic curve of 0.86 (0.83, 0.87).
CONCLUSIONS:
The NLP model classified the quality of operative performance feedback with high accuracy and specificity. NLP offers residency programs the opportunity to efficiently measure feedback quality. This information can be used for feedback improvement efforts and ultimately, the education of surgical trainees.