Predicted Readiness Score Overview

The predicted readiness score is a model-based estimate of a trainee’s likelihood of earning a practice ready rating on his or her next SIMPL evaluation for that procedure. The model was developed using over 60,000 faculty-to-trainee evaluations for over 160 procedures. 

A predicted readiness score is based on the following factors:

  1. The specific procedure,
  2. The PGY-level of the trainee and the month of the academic year, and
  3. The cumulative number of practice ready evaluations by a trainee for a given procedure.
A screenshot of the predicted readiness score visualization
The Predicted Readiness Score on the trainee progress page of the SIMPL Reports Dashboard. The above trainees’ score would read as “there is a 77% chance that they will perform a laparoscopic cholecystectomy at a practice-ready level on their next attempt”

There is an important difference between a trainee’s predicted readiness and true readiness. True readiness is not directly knowable, but our confidence that we are quantifying readiness using SIMPL and the predicted readiness score increases as trainees request SIMPL evaluations and perform to a practice ready or higher standard over multiple evaluations.

The approach we used to develop the model allowed us to include factors like differences among trainees, variation among raters, and the overall complexity of a case in an efficient and robust way. We included variation from each of these factors to account for the uncertainty in each predicted readiness score while allowing the trainee’s cumulative experience to be the primary driver of changes in the score.

The predicted readiness score is explicitly predictive and can be interpreted in relation to a trainee’s future performance in the operating room. Moreover, as soon as a trainee has completed a single SIMPL evaluation for one of the 160+ procedures, he or she can generate a predicted readiness score.

Individual trainees can increase their readiness scores and reduce the overall uncertainty in their scores as they earn more prior practice ready ratings and as they progress in their training. For example, for a procedure where the typical trainee has a 50% chance of earning a practice ready rating, a trainee who has earned 10 practice ready ratings for that procedure over time has increased his or her predicted readiness score to 73%. The predicted probability score, therefore, provides a common scale that can be used to compare trainees’ progress toward different standards like a mastery standard of 95% or higher. Scores can also be used to quantify what the typical trainee’s likelihood is within a given PGY and month, which supports both absolute and relative benchmarks of performance.