Determining the Measurement Quality of a Montessori High School Teacher Evaluation Survey
The purpose of this study was to conduct a psychometric validation of a course evaluation instrument, known as a student evaluation of teaching (SET), implemented in a Montessori high school. The authors demonstrate to the Montessori community how to rigorously examine the measurement and assessment quality of instruments used within Montessori schools. The Montessori high school community needs an SET that has been rigorously examined for measurement issues. The examined SET was developed by a Montessori high school, and the sample data were collected from Montessori high school students. Using a Rasch partial credit model, the results of the analysis identified several measurement issues, including multidimensionality, misfit items, and inappropriate item difficulty levels. A revised version of the SET underwent the same analysis procedure, and the results indicated that measurement issues persisted. The authors suggest several ways to improve the overall measurement quality of the instrument while keeping the Montessori foundation. Additional validation studies with a revised version of the SET will be needed before the instrument can be endorsed for full implementation in a Montessori setting.
American Montessori Society. (2015). Montessori schools. Retrieved from http://amshq.org/Montessori-Education/Introduction-to-Montessori/Montessori-Schools
Barker, D. (2011). A historical look at Montessori’s Erdkinder. Communications, 1-2, 96–112.
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). New York, NY: Lawrence Erlbaum Associates.
Bradley, K. D., Sampson, S. O., & Royal, K. D. (2006). Applying the Rasch rating scale model to gain insights into students’ conceptualisation of quality mathematics instruction. Mathematics Education Research Journal, 18, 11–26. doi:10.1007/BF03217433
de Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: The Guilford Press.
Donahoe, M., Cichucki, P., Coad-Bernard, S., Coe, B., & Scholtz, B. (2013). Best practices in Montessori secondary programs. Montessori Life, 25(2), 16–23.
Feldman, K. A. (1976). The superior college teacher from the students’ view. Research in Higher Education, 5, 243–288.
Kahn, D., & Pendleton, D. R. (2007). The whole-school Montessori handbook for teachers and administrators. Burton, OH: North American Montessori Teachers’ Association.
Kahn, D. (2011). Eight pictures at an exhibition: A Montessori retrospective on the discovery of the adolescent. Communications, 1-2, 15–41.
Kulik, J. A. (2001). Student ratings: Validity, utility, and controversy. New Directions for Institutional Research, 109, 9–25. doi:10.1002/ir.1
Linacre, J. M. (2015). A user guide to Winsteps Ministeps Rasch-model computer programs: Program manual 3.90.0. Available from Winsteps http://www.winsteps.com/manuals.htm
Linacre, J. M. (2016). Winsteps [Computer software]. Available from Winsteps http://www.winsteps.com
Marsh, H. W., & Dunkin, M. J. (1992). Students’ evaluations of university teaching: A multidimensional perspective. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 8, pp. 143–233). New York, NY: Agathon Press.
Mayes, C., & Williams, E. (2013). Nurturing the whole student: Five dimensions of teaching and learning. Lanham, MD: Rowman & Littlefield Education.
Miller, J. P. (2010). Whole child education. Canada: University of Toronto Press.
Miller, R. (1990). What are schools for? Holistic education in American culture. Brandon, VT: Holistic Education Press.
Montessori, M. (1973). From childhood to adolescence: Including Erdkinder and the function of the university. New York, NY: Schocken Books.
Montessori, M. (2011). Principles and practice in education. Communications, 1-2, 50–60. (Reprinted from First Lecture, Institute of Medical Psychology, London, November 10, 1936.)
Nardi, P. M. (2006). Doing survey research: A guide to quantitative methods. Boston, MA: Pearson Education.
National Center for Montessori in the Public Sector. (2014). USA Montessori census. Available from http://www.montessoricensus.org
North American Montessori Teachers’ Association. (2015). Curriculum downloads. Available from http://www.montessori-namta.org/Curriculum-Downloads
Royal, K. D., & Elahi, F. (2011). Psychometric properties of the Death Anxiety Scale (DAS) among terminally ill cancer patients. Journal of Psychosocial Oncology, 29, 359–371. http://www.tandfonline.com/doi/abs/10.1080/07347332.2011.582639
Toland, M. D. (2014). Practical guide to conducting an item response theory analysis. The Journal of Early Adolescence, 34, 120–151. doi:10.1177/0272431613511332
Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. Chicago, IL: MESA Press.
Wright, B. D. (1993). “Logits”? Rasch Measurement Transactions, 7, 228. Retrieved from https://www.rasch.org/rmt/rmt72e.htm
Wright, S. L., & Jenkins-Guarnieri, M. A. (2012). Student evaluations of teaching: Combining the meta-analyses and demonstrating further evidence for effective use. Assessment & Evaluation in Higher Education, 37, 683–699. http://dx.doi.org/10.1080/02602938.2011.563279
Copyright (c) 2017 Anthony Philip Setari, Kelly Bradley
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution Non-Commercial License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. Authors can view article download statistics for published articles within their accounts.
Journal of Montessori Research
The following is an agreement between the Author (the “Corresponding Author”) acting on behalf of all authors of the work (“Authors”) and the Journal of Montessori Research (the “Journal”) regarding your article (the “Work”) that is being submitted for consideration.
Whereas the parties desire to promote effective scholarly communication that promotes local control of intellectual assets, the parties for valuable consideration agree as follows.
A. CORRESPONDING AUTHOR’S GRANT OF RIGHTS
After being accepted for publication, the Corresponding Author grants to the Journal, during the full term of copyright and any extensions or renewals of that term, the following:
1. An irrevocable non-exclusive right to reproduce, republish, transmit, sell, distribute, and otherwise use the Work in electronic and print editions of the Journal and in derivative works throughout the world, in all languages, and in all media now known or later developed.
2. An irrevocable non-exclusive right to create and store electronic archival copies of theWork, including the right to deposit the Work in open access digital repositories.
3. An irrevocable non-exclusive right to license others to reproduce, republish, transmit,and distribute the Work under the condition that the Authors are attributed. (Currently this is carried out by publishing the content under a Creative Commons Attribution Non-Commercial 4.0 license (CC BY-NC.)
4. Copyright in the Work remains with the Authors.
B. CORRESPONDING AUTHOR’S DUTIES
1. When distributing or re-publishing the Work, the Corresponding Author agrees to credit the Journal as the place of first publication.
2. The Corresponding Author agrees to inform the Journal of any changes in contact information.
C. CORRESPONDING AUTHOR’S WARRANTY
The Corresponding Author represents and warrants that the Work is the Authors’ original work and that it does not violate or infringe the law or the rights of any third party and, specifically, that the Work contains no matter that is defamatory or that infringes literary or proprietary rights, intellectual property rights, or any rights of privacy. The Corresponding Author also warrants that he or she has the full power to make this agreement, and if the Work was prepared jointly, the Corresponding Author agrees to inform the Authors of the terms of this Agreement and to obtain their written permission to sign on their behalf. The Corresponding Author agrees to hold the Journal harmless from any breach of the aforestated representations.
D. JOURNAL’S DUTIES
In consideration of the Author’s grant of rights, the Journal agrees to publish the Work, attributing the Work to the Authors.
E. ENTIRE AGREEMENT
This agreement reflects the entire understanding of the parties. This agreement may be amended only in writing by an addendum signed by the parties. Amendments are incorporated by reference to this agreement.
ACCEPTED AND AGREED BY THE CORRESPONDING AUTHOR ON BEHALF OF ALL AUTHORS CONTRIBUTING TO THIS WORK