Using Teaching Excellence Surveys to Evaluate Improvements in Teaching Confidence

Background. There are many surveys to assess teaching excellence, but few validated tools to assess improvements in teaching confidence among faculty over time. We hypothesized that previously validated surveys for learner evaluation of faculty teaching excellence also can be used as a self-evaluation tool to assess changes in faculty teaching skills confidence over time. Methods. A cohort study was designed using a composite survey from two previously validated surveys (SETQ and CanMEDS) on teaching excellence. The composite survey was administered before and after a faculty development course on teaching excellence at the University of Kansas Medical Center in the Spring of 2012. Course “completers” attended more than 50% of the course and “non-completers” attended 50% or less of the course. Results. The overall mean change in survey result scores on a five-point Likert scale was nearly one point for “completers” (mean difference = 0.92, SD = 0.41) as opposed to 0.34 for “noncompleters” (SD = 0.34, p = 0.001). The Cronbach’s alpha coefficients for the pre-course surveys were 0.83 and 0.85 versus 0.88 and 0.83 for the post-course surveys, indicating a high internal consistency for both survey instruments. Conclusions. Measurable improvements in teaching skills confidence occur following faculty professional development courses. These improvements can be assessed more efficiently by using previously validated and reliable assessment tools in new and innovative ways. KS J Med 2015; 8(1):8-17. Introduction Clinical educators face many challenges in their roles as teachers of medicine. The obstacles are greater given the lack of formal training in teaching offered to most faculty prior to their first faculty appointment. To this end, there are a number of courses directed toward teaching faculty how to be excellent clinical teachers. However, it often can be difficult to justify the time, expense, and use of limited resources required to provide faculty with a formalized course to improve their teaching quality, especially when few valid evaluation tools exist to assess their effectiveness. In the recent past, a number of studies have defined and evaluated teaching excellence using subjective surveys of learners. New evaluation tools are being introduced and many of these tools have been validated. We proposed that previously validated surveys for learner evaluation of teaching excellence can be used as a self-evaluation tool for educators to assess changes and improvements in their own teaching confidence over time. Methods Study design, participants and setting. This cohort study was conducted at the University of Kansas Medical Center from January to September 2012 in conjunction with a faculty development course on teaching skills. A total of 28 faculty Kansas Journal of Medicine 2015 Teaching Excellence Surveys 9 members were invited to participate in a course called “Doctors as Educators” which consisted of twelve one-hour sessions. The format incorporated lectures, small group discussions, and faculty-learner practiceteaching presentations with feedback. Twenty-three faculty members enrolled; nineteen completed a self-assessment survey at the beginning and end of the course. Surveys were distributed by email and paper and reminders to complete the surveys were given verbally and by email. Four faculty members chose not to complete both surveys. Ten were present for more than 50% of the teaching sessions and were considered course “completers”. Nine were present for 50% or less of the sessions and were considered “non-completers.” Instrument development and data collection. The self-assessment survey was a composite of two previously-validated and reliable tools to assess teaching excellence: the Canadian Medical Education Directions for Specialists (CanMEDS) and the System for Evaluation of Teaching Qualities (SETQ) evaluation tools (Appendix). The wording of some questions was modified slightly to apply to a self-assessment rather than a learner assessment of a faculty member. A five-point Likert response scale (poor = 1, fair = 2, average = 3, good = 4, and excellent = 5) was used to answer the survey. A similar format was used by the original surveys. Demographics were collected for each faculty member enrolled in the course and included age, gender, race, years of faculty experience, graduation from a foreign medical school, completion of a chief resident year, participation in a prior faculty development course on teaching, and other formal training as a clinical educator. Faculty self-assessment questionnaires were kept confidential using random numerical identifiers during data analysis. This research project was approved by the University of Kansas School of Medicine Human Subject Committee. Data analysis. Data were coded and simple descriptive statistics were calculated for all applicable variables. The majority of faculty demographics were discrete, dichotomous variables and the remaining variables were transformed into dichotomous versions (mean values were used as the cut-off points) for consistency and clarity of the presentation. Fisher exact tests were performed to evaluate for any statistically significant demographic percent differences between faculty course “completers” versus “non-completers”. Paired t-tests also were performed to explore associations between continuous selfassessed overall improvements in faculty teaching quality (calculated as a change in the mean score for the questionnaire before and after the course). Instrument internal consistency reliability was examined by calculating the Cronbach’s alpha coefficient. Results Demographics. The faculty sample was comprised mainly of white providers (15/19, 79%), in their mid-thirties or beyond (11/19, 58%), with less than six years of faculty work experience (12/19, 63%). The sample was nearly gender-equal and most of the faculty were United States medical graduates (16/19, 84%). Only one (5%) faculty member of 19 had any formal training as a clinician-educator; six (32%) completed a chief residency year, and nearly half (8/19, 42%) had completed a faculty development course on teaching in the past. None of the demographic differences between “completers” and “non-completers” were statistically significant (Table 1). Survey results. Self-confidence ratings on preand post-course surveys for all enrolled faculty increased by over half a point on the five-point Likert scale (mean Kansas Journal of Medicine 2015 Teaching Excellence Surveys 10 difference = 0.65±0.33). However, when the group was divided into “completers” and “non-completers”, the overall selfconfidence survey rating for “completers” changed by nearly one point (mean difference = 0.92±0.41) after the course, compared to 0.34 for “non-completers” (SD = 0.34, p = 0.001), suggesting that faculty who completed the course developed significantly more confidence in their teaching skills (Figure 1). Table 1. Faculty demographics overall and by completers versus non-completers. Faculty Demographics Total N=19 Completers N=10 NonCompleters N=9 P Value Age < 36 years old, n (%)* Age ≥ 36 years old, n (%)* 8 (42.1%) 11 (57.9%) 3 (30%) 5 (55.6%) 0.37 Faculty experience < 6 years, n (%)* Faculty experience ≥ 6 years, n (%)* 12 (63.2%) 7 (36.8%) 8 (80%) 4 (44.4%) 0.17 Male, n (%) Female, n (%) 9 (47.4%) 10 (52.6%) 5 (50%) 4 (44.4%) 1.00 White, n (%) Non-white, n (%) 15 (78.9%) 4 (21.1%) 7 (70%) 8 (88.9%) 0.58 Foreign medical graduate, n (%) American medical graduate, n (%) 3 (15.8%) 16 (84.2%) 2 (20%) 1 (11.1%) 1.00 Completion of a chief residency, n (%) No completion of a chief residency, n (%) 6 (31.6%) 13 (68.4%) 4 (40%) 2 (10.5%) 0.63 Completion of a faculty development course on teaching, n (%) No completion of a faculty development course on teaching, n (%) 8 (42.1%) 11 (57.9%) 3 (30%) 5 (55.6%) 0.37 Completion of other formal training as a clinical educator, n (%) No completion of other formal training as a clinical educator, n (%) 1 (5.3%) 18 (94.7%) 0 (0%) 1 (11.1%) 1.00 *Cut-offs for provider demographics were chosen based on the mean values where applicable. Internal consistency of survey. The Cronbach’s alpha with standardized variables was used to investigate the internal consistency of each of the condensed surveys for all participants. A Cronbach’s alpha of at least 0.70 or greater is considered an indication of acceptable reliability. The Cronbach’s alpha coefficients for our precourse SETQ and CanMEDS surveys were 0.83 and 0.85, respectively; the Cronbach’s alpha coefficients for our postcourse surveys were 0.88 and 0.83, respectively, indicating a high internal consistency for all. 12 Discussion In academic medicine, it is crucial to train and retain clinically excellent medical educators. However, many physicians complete their residencies without any formal training on how to become a clinical educator. Without training, junior faculty can be overwhelmed by the teaching expectations. In theory, this can result in Kansas Journal of Medicine 2015 Teaching Excellence Surveys 11 Figure 1. Boxplot of mean difference in overall survey scores for completers versus noncompleters. poor evaluations on their teaching abilities, dissatisfying instruction as a faculty member, and ultimately, poorly trained future physicians. This void can be filled by creating easily-accessible, formal training programs for clinical educators. For programs like this to be supported, evidence of their success also must be shown and therein lays the challenge. As with others, the faculty development course showed a statistically significant improvement in teaching confidence in faculty “completers” as opposed to “noncompleters”, using a preand post-course survey. Though significant, surveys are inherently biased and few validated assessment tools have been published to evaluate improvements in teaching abilities over time in conjunction with faculty development courses. Furthermore, although 


Introduction
Clinical educators face many challenges in their roles as teachers of medicine.3][4] However, it often can be difficult to justify the time, expense, and use of limited resources required to provide faculty with a formalized course to improve their teaching quality, especially when few valid evaluation tools exist to assess their effectiveness. 5,62][13][14] We proposed that previously validated surveys for learner evaluation of teaching excellence can be used as a self-evaluation tool for educators to assess changes and improvements in their own teaching confidence over time.

Methods
Study design, participants and setting.This cohort study was conducted at the University of Kansas Medical Center from January to September 2012 in conjunction with a faculty development course on teaching skills.A total of 28 faculty members were invited to participate in a course called "Doctors as Educators" which consisted of twelve one-hour sessions.The format incorporated lectures, small group discussions, and faculty-learner practiceteaching presentations with feedback.Twenty-three faculty members enrolled; nineteen completed a self-assessment survey at the beginning and end of the course.Surveys were distributed by email and paper and reminders to complete the surveys were given verbally and by email.Four faculty members chose not to complete both surveys.Ten were present for more than 50% of the teaching sessions and were considered course "completers".Nine were present for 50% or less of the sessions and were considered "non-completers." Instrument development and data collection.The self-assessment survey was a composite of two previously-validated and reliable tools to assess teaching excellence: the Canadian Medical Education Directions for Specialists (CanMEDS) and the System for Evaluation of Teaching Qualities (SETQ) evaluation tools (Appendix). 4,12The wording of some questions was modified slightly to apply to a self-assessment rather than a learner assessment of a faculty member.A five-point Likert response scale (poor = 1, fair = 2, average = 3, good = 4, and excellent = 5) was used to answer the survey.A similar format was used by the original surveys.
Demographics were collected for each faculty member enrolled in the course and included age, gender, race, years of faculty experience, graduation from a foreign medical school, completion of a chief resident year, participation in a prior faculty development course on teaching, and other formal training as a clinical educator.Faculty self-assessment questionnaires were kept confidential using random numerical identifiers during data analysis.This research project was approved by the University of Kansas School of Medicine Human Subject Committee.
Data analysis.Data were coded and simple descriptive statistics were calculated for all applicable variables.The majority of faculty demographics were discrete, dichotomous variables and the remaining variables were transformed into dichotomous versions (mean values were used as the cut-off points) for consistency and clarity of the presentation.Fisher exact tests were performed to evaluate for any statistically significant demographic percent differences between faculty course "completers" versus "non-completers".Paired t-tests also were performed to explore associations between continuous selfassessed overall improvements in faculty teaching quality (calculated as a change in the mean score for the questionnaire before and after the course).
Instrument internal consistency reliability was examined by calculating the Cronbach's alpha coefficient.

Results
Demographics.The faculty sample was comprised mainly of white providers (15/19, 79%), in their mid-thirties or beyond (11/19, 58%), with less than six years of faculty work experience (12/19, 63%).The sample was nearly gender-equal and most of the faculty were United States medical graduates (16/19, 84%).Only one (5%) faculty member of 19 had any formal training as a clinician-educator; six (32%) completed a chief residency year, and nearly half (8/19, 42%) had completed a faculty development course on teaching in the past.None of the demographic differences between "completers" and "non-completers" were statistically significant (Table 1).
Survey results.Self-confidence ratings on pre-and post-course surveys for all enrolled faculty increased by over half a point on the five-point Likert scale (mean difference = 0.65±0.33).However, when the group was divided into "completers" and "non-completers", the overall selfconfidence survey rating for "completers" changed by nearly one point (mean difference = 0.92±0.41)after the course, compared to 0.34 for "non-completers" (SD = 0.34, p = 0.001), suggesting that faculty who completed the course developed significantly more confidence in their teaching skills (Figure 1).Internal consistency of survey.The Cronbach's alpha with standardized variables was used to investigate the internal consistency of each of the condensed surveys for all participants.A Cronbach's alpha of at least 0.70 or greater is considered an indication of acceptable reliability. 15The Cronbach's alpha coefficients for our precourse SETQ and CanMEDS surveys were 0.83 and 0.85, respectively; the Cronbach's alpha coefficients for our postcourse surveys were 0.88 and 0.83, respectively, indicating a high internal consistency for all. 4,12

Discussion
In academic medicine, it is crucial to train and retain clinically excellent medical educators.However, many physicians complete their residencies without any formal training on how to become a clinical educator. 1Without training, junior faculty can be overwhelmed by the teaching expectations.In theory, this can result in poor evaluations on their teaching abilities, dissatisfying instruction as a faculty member, and ultimately, poorly trained future physicians.This void can be filled by creating easily-accessible, formal training programs for clinical educators.For programs like this to be supported, evidence of their success also must be shown and therein lays the challenge.
As with others, the faculty development course showed a statistically significant improvement in teaching confidence in faculty "completers" as opposed to "noncompleters", using a pre-and post-course survey.Though significant, surveys are inherently biased and few validated assessment tools have been published to evaluate improvements in teaching abilities over time in conjunction with faculty development courses. 5,6urthermore, although a number of "static" tools to evaluate teaching abilities have been validated, none have been translated into use for evaluating "dynamic" improvements in teaching qualities over time.This study sought to fill this void by using two previously validated "static" surveys of teaching abilities as "dynamic" evaluation tools.
These previously validated tools were used in two novel ways.First, the evaluations were used to measure dynamic improvements in teaching confidence over time (as opposed to "a moment in time" assessment of teaching abilities).Second, the faculty assessed their own teaching abilities instead of their learners.When the internal consistency of each survey was tested, the Cronbach's alpha remained high for both.These findings supported the argument that validated tools on teaching excellence can be used in a variety of ways other than their original design to evaluate similar qualities.Future studies may focus on modifying previously validated tools to fit their needs as opposed to creating yet another survey.Another strength of this study was the use of a cohort design which allowed the authors to show a correlation between frequent exposure to a teaching skills curriculum and self-assessed improvement in teaching confidence.
Several limitations should be considered.First, the number of faculty who participated in this study was small.Thus, these results may not be generalizable to a larger population.Second, the faculty self-selected for registration into the course and all who registered were accepted.Therefore, there may have been a self-selection bias present with faculty who desired to improve their teaching abilities more likely to attend the course, reflect upon their attendance positively, and note more significant improvements in their teaching confidence following the course.However, an equal number of faculty members who registered for the course did not attend a significant portion of the classes and subsequently, did not note significant improvements in their teaching confidence on the survey.Third, self-assessment surveys are inherently subjective by nature.Direct observation of teaching performance may be a better method to assess teaching abilities but the time and resources required, along with the challenge of creating a valid tool for assessing teaching in a variety of different settings and formats, make this an unrealistic way to evaluate clinical educators.

Conclusions
For well-trained physicians in the future, faculty must be educated to teach the trainees.This study confirmed that measurable improvements in teaching confidence can occur among faculty following professional development courses.We have a responsibility to create curricula that improve teaching skills and confidence and validate the tools used to assess them.
Evaluations on improvements in teaching abilities following faculty development courses serve a number of roles: to assess improvements made by its learners, to uncover educators who are less effective and who may need more intensive instruction or remediation, and to acknowledge the effectiveness of the faculty development course itself.The ability to use previously validated assessment tools in new ways allows clinician-researchers and clinician-educators to redirect their focus away from development of these tools and toward new arenas in medical education and research.Ultimately, the benefits reach far beyond the faculty themselves.Increased focus on training and effectively evaluating medical educators will serve to benefit the learners they teach by providing them with a more expansive fund of knowledge, ultimately giving our future physicians a larger arsenal from which to diagnose and treat the patients who seek their expertise.

Figure 1 .
Figure 1.Boxplot of mean difference in overall survey scores for completers versus noncompleters.

Table 1 .
Faculty demographics overall and by completers versus non-completers.
*Cut-offs for provider demographics were chosen based on the mean values where applicable.