Introduction
The objective of a residency recruitment process is to select the most appropriate candidate with the knowledge, skills and behaviors that will lead to success in residency and professional life.1 In 2024 there were 3034 applicants for 178 programs and 1695 anesthesiology PGY-1 positions.2 The median cost per PGY1 is estimated around $9899, of which $1042 is attributed to the interview.3 Thus, choosing the right applicant is crucial for the program to succeed and to maximize the return on investment.4 Predicting resident performance based on the material contained within a residency application is challenging.5–7 Generally, resident selection is based on a composite score that reflects cognitive predictors such as United States Medical Licensing Examination (USMLE) scores, medical school grades, class rank, honors grades in core subjects, and membership in the medical school honor society Alpha Omega Alpha (AOA).8 It also includes non-cognitive scores that can be gleamed by letters of recommendation (LOR), dean letters, personal statement, and interviews.1,9
Cognitive skills are associated with problem-solving and various forms of reasoning.10 These can be measured objectively by standardized tests, such as the USMLE Step 2 Clinical Knowledge (CK) exam, which are typically part of the residency application.11 In addition, non-cognitive skills and personal characteristics are predictive of good physicians.6,12 In resident selection there are few tools that are used to assess non-cognitive skills. Structured interviews can be used to measure these non-cognitive skills, by analyzing the job and creating standardized questions that aim at specific non-cognitive skills.5 There is a trend in anesthesiology residency towards placing increasing value on emotional intelligence, which can be gleamed by more non-cognitive factors.7 LORs are one of the documents which could potentially reflect these qualities in a candidate. However, studies have demonstrated that LORs often repeat the merits of an applicant’s cognitive metrics and do not specifically provide non-cognitive information predictive of future resident performance.13–16 According to the 2019 residency census, resident attrition is not as significant a problem in anesthesiology residency as it is in other specialties, and has declined since 2010.17 Although the attrition rate is low, residency program directors continue to look for comprehensive tool to select the most appropriate candidates.
Studies conducted in different specialties have yielded conflicting results in what programs value.18 Vinagre et al. found analyzing the response of 45 program directors that the top three factors influencing an interview offer in anesthesiology were the USMLE Step 1 score, the LORs, and finally, the Medical School Performance Evaluation (MSPE) combined with the deans letter. However, once the USMLE Step 1 exam became pass/fail, LORs have become the top factor, followed by USMLE Step 2 and the MSPE.19,20 Anesthesiology program directors also have a preference for the language used in LORs, specifically applicants with words focused on work or insight matched at a lower ranked program compared to applicants who had LORs focused on their anger.20 When looking at resident selection after Step 1 became pass/fail, program directors placed more value on Step 2 CK, while still ranking performance on sub-internships as very important.21 A retrospective review correlated Step 2, high interview performance, clerkship honors, and less service experience with higher ITE scores.22
Although some studies have shown that interviews are one of the most important factors in ranking candidates, others have shown a non-significant change in the programs final ranking.1,9 In particular, interviews have been criticized for their lack of standardization. There have been conflicting results on if it predicts residency performance, although behavioral interviews were more predictive of residency performance compared to traditional interviews.1 A survey of program directors in multiple specialties found that they valued interviewing as the most important screening tool in residency selection as it could gleam candidates noncognitive skills and compatibility with the program. They found that structured behavioral interviews were more reliable and valid compared to the traditional interview.23
The behavioral interview technique was introduced by an industrial psychologist, Dr. Tom Janz.24 The premise of the behavioral interview is that past behaviors are predictive of future behavior. In this type of interview, applicants are asked to narrative past situations to see if appropriate behaviors may be identified. Validity of this interview method has been studied and assessed in regard to future job performance.25 A high score in the behavioral interview predicts better job performance. Studies have demonstrated the importance of assessing non-cognitive skills among anesthesiologists.26 In an earlier study, researcher analyzed perioperative critical incidents reported by anesthesiologists and found that around 60% of the incidents involved non-cognitive personal attributes.26
In this qualitative study, we analyzed the role of behavioral-based interviews in predicting a successful residency. Behavioral-based interviews and traditional unstructured interviews conducted on the same individual were compared and the results of the interview were compared to the semi-annual and annual evaluations submitted by the teaching faculty and the program directors.
Aims
The aims of this study were to (1) identify the non-cognitive characteristics which can be elicited during anesthesiology resident selection interviews, (2) to compare these non-cognitive characteristics with those identified through resident evaluations, and (3) to correlate the non-cognitive factors that were predictive of a successful residency.
Methods
Ethical considerations
This qualitative study was designed to analyze the role of behavior-based interviews in selecting appropriate candidates in the field of anesthesiology. This study was conducted in the Department of Anesthesiology at Montefiore Medical Center. The Einstein IRB of Albert Einstein Medical College exempted this study according to the criterion (research conducted in established educational settings involving normal educational practices) under the Health and Human regulations 45 Code of Federal Regulation 46.101 (b) (4).
Sample Population
A critical case sampling method for selecting the study population was conducted. Evaluation sheets of six anesthesiology residents who are at least in their second year of residency were analyzed by the study team. The residency program director, who is not part of the research team, selected and divided the residents into two groups based on their annual evaluations. Group one, “the excellent group”, was comprised of three residents who received excellent or superior overall annual evaluations (five out of five scores) and group two, “the satisfactory group”, included three residents who received unsatisfactory, marginal, or satisfactory evaluations (three or less out of five scores).
The semiannual evaluations in the excellent group included “performs thoughtful and detailed preoperative evaluations with insightful use of diagnostic and laboratory tests” and “Evaluations are consistently excellent in the area of clinical judgment, uses sound principle of deductive reasoning. Excellent communication and interpersonal skills. Viewed as a consistently effective team member”. The evaluation narratives in the satisfactory group included “it has been pointed out he is slow to react to blood pressure issues. Also, when we were transferring a patient to ICU ventilator it was unclear if he knew the patient was not being ventilated” and “Asked to see a patient for preoperative evaluation and waited 2 days. He is not taking initiative to actively see patients.”
Narrative Material
A panel of four interviewers, who received informal training in conducting behavioral and traditional unstructured interviews, wrote the interview narratives (Table II and III). As employees of the Montefiore Medical Center they had previously received bias training by completing the Montefiore institutional video. Before every interview, the interviewer would be instructed and given a form that included a set list of questions that they could pick from, but not deviate (Table III). If conducting a behavioral interview, the questions were further subdivided into several non-cognitive skills, for which the interviewer had a rubric to score the resident on. Each applicant interviewed for only one structured interview and one behavioral interview. Each interviewer was instructed in only one style of interview. In both groups for each category a behavior score from A to F were given, where “A” demonstrated consistently the behaviors sought and “F” failed to demonstrate the sought behaviors.
To separate the residents into the two groups, the residents semi-annual scores were used, due to the standardized questions. Residents were evaluated at different time points and evaluations were based on the Accreditation Council for Graduate Medical Education (ACGME) core competencies. Each resident received monthly evaluations by the faculty which were based on the rotations completed or in progress. They also received semi-annual and annual evaluations by the program director based on the monthly evaluations and other influencing parameters such as verbal evaluations about performance. Baseline demographic data was gathered for these two groups. Researchers received de-identified interview narratives and evaluation sheets, together with a standard residency application packet.
Data analysis
Two investigators independently reviewed the narratives. Multiple readings were performed to gain a thorough understanding of the content and appropriately place the narratives within the established categories. Coding disagreements would arise when a interviewer would ask one question for two of the categories, to solve them the agreement became that it would be scored the same for the two categories.
A narrative approach was used to complete the content analysis of the interviews. The units of analyses were sentences and words written on interview narratives and evaluation sheets. Content analysis was used to analyze the narratives as suggested by Krippendorf (2004).27 Abstraction from the text to the categories will follow the working model of Graneheim and Lundman (2004).28 Methods of analysis also included word count within each category. Deductive analysis of the narratives explored the themes identified by the theoretical experience guiding the study and also the themes which were supported by the observation of the researchers.
The themes identified during the semi-annual evaluations were related to the non-cognitive factors that may be able to predict a successful anesthesiology residency. These were determined to be adaptability, decisiveness and judgment, teamwork, and stress management. Each narrative was primarily analyzed for the aforementioned categories. Table III shows the questions that were asked during the structured interview were aimed at making sure each of these categories was covered. The inductive analysis part of the study was focused on creating new meaningful categories. New meaningful information from the narratives were coded and condensed to sub-categories and then to categories.
Results
Twelve interview sheets belonging to the six study residents were analyzed. All the residents were at least in their CA-2 year and received semi–annual evaluations for the year 2011-2012. The baseline demographic characteristics of the residents were compared (Table I). Qualitative analysis of behavioral interview narratives clearly identified the non-cognitive characteristics of adaptability, decisiveness, time management, judgment, and the ability to work in a team (Table IV). It also identified other essential non-cognitive skills necessary for an anesthesiologist, such as ability to prioritize, study techniques, ability to destress.
The scores assigned by the interviewer were different for the two groups. In the excellent group in behavioral interview, 67% received a score of A and 33% received a B+, while in the traditional unstructured interview the scores for were consistently lower with all candidates receiving B. In the satisfactory group, for the behavioral interview the scores were evenly distributed (33%) among A, B and B- respectively, while in the traditional unstructured interview, 33% received a score of an A- and 67% received a B+.
The narrative part of the semiannual evaluations of the resident also identified the non-cognitive skills of adaptability, decisiveness, time management, judgment, and the ability to work in a team. Analysis of the semiannual evaluations also independently confirmed the presence or absence of desirable non-cognitive attributes in each group.
Additional non-cognitive skills identified in the behavioral interview narratives were diligence, enthusiasm, self-motivation, multi-tasking ability, thoughtfulness, and compassion. The non-cognitive skills identified by the traditional unstructured interview were diligence, decisiveness, flexibility, manual dexterity, lack of confidence, and poor organization skills. The behavioral interview method identified more non-cognitive characteristics when compared with the traditional unstructured interview technique.
The behavioral interview method identified more desirable non-cognitive characteristics when compared with the traditional unstructured interview technique. In our qualitative analysis of the resident interview narratives and semiannual evaluations, the behavioral interviews predicted the non-cognitive skills that are necessary for an anesthesiology resident in training. Additionally, traditional unstructured interviews were found to poorly predict residents’ non-cognitive skills and evaluation performances.
Discussion
Certain studies demonstrated a correlation between academic success and cognitive predictors.29–33 However, many studies have failed to demonstrate a consistent pattern of cognitive factors predicting successful residency and better job performance.34–37 Academic discussions regarding skill and related abilities focus measuring cognitive skills while ignoring or minimizing the value of non-cognitive skills. A survey conducted on plastic surgery program directors revealed that only 43% believed that current resident selection is adequate in identifying potentially problematic residents.38
One of the potential areas for intervention in assessing the non-cognitive skills of an applicant is modification of the interview technique. A paradigm shift from the traditional unstructured interview to the semi-structured behavioral interview gives the interviewer a much better opportunity to assess the applicant’s non-cognitive skills. As mentioned earlier, the premise of the behavioral interview is that past behaviors are predictive of future job performance. By changing the content and method by which the questions are asked, interviewer and applicant will engage in a more in-depth conversation, which allows exploration of both cognitive and non-cognitive skills. Multiple studies conducted in the area of traditional unstructured interview showed that these interviews failed to highlight factors which can predict job success.39–41 In contrast, studies conducted in the area of behavioral semi-structured interviews have demonstrated a positive correlation between behavioral interviews and predicting successful job performance.42–44 Our results are no exception. The interview narratives in the residents’ with excellent evaluations demonstrated desirable non-cognitive characteristics, while the interview narratives of the satisfactory evaluation group demonstrated average performance of the non-cognitive skills. This confirms the importance of conducting a behavioral interview in selecting residency applicants. Our behavioral interview method is easy to adapt for the programs at no extra cost.
Multiple Mini Interviews (MMI) are an alternative to traditional and behavioral interviews. MMIs require multiple stations, with each station measuring one to three attributes, this requires multiple interviewers, which does reduce the chance of bias.45 MMIs, although reliable, require more resources compared to behavioral interviews and literature reviews have found no statistical difference when scoring the interviewees.45 A survey conducted to understand the factors important to anesthesiology residency applicants found that they preferred an interview where they had the chance to meet the program director, faculty, and residents over MMIs.46
In future studies, having multiple people interviewing the applicants would be an interesting way to see if faculty and residents reach a consensus on the qualities shown by the narratives of the applicant. This could have the potential of reducing bias that is possible in a traditional interview. Additionally, understanding if having a clear scoring rubric, as standardization, would detract from having an in-depth conversation with applicants to gleam information on their motivation for the program or other non-cognitive skills that could work together with the skills that the program is looking for.
Limitations
As this was a pilot study, the small sample size analyzed necessitates further verification of our findings. A critical case sampling was performed to select the residents since this method yielded the most information and would have greatest impact on the development of knowledge. In addition, inherent in any interview process there are issues related to subjectivity, reliability, and validity. Even though these limitations are inevitable, steps were taken in the study design to reduce the influence of these limitations. Subjectivity, related to the interviewer bias was reduced by giving clear instructions and informal training to the interviewers. Reliability and validity of the interview are areas that still need improvement. The results demonstrated the validity of this method in anesthesiology, but it has not been verified in larger settings or in other specialties. Recording the interview dialogue to allow for multiple assessments of the qualitative data is a potential solution to some of these issues but may introduce extra stress for the interviewee.
Conclusion
Constructing a behavioral interview process can permit residency programs to evaluate applicant’s non-cognitive abilities and there by create a balanced selection process based on cognitive and non-cognitive skills. Additionally, this strategy may improve the selection of residents by identifying those individuals who are suited to the field of anesthesiology as they are adaptable, decisive, team-players with excellent judgment.
COI
No relevant COI to disclose
No funding provided. No conferences or other presentation for this work.
Acknowledgements
None