Assessing the Significance of Model Selection in Ecology
DOI:
https://doi.org/10.17161/eurojecol.v6i2.13747Keywords:
Model selection; Akaike’s Information Criterion; Permutation Test; Ibex; Reindeer.Abstract
Model Selection is a key part of many ecological studies, with Akaike’s Information Criterion (AIC) being by far the most commonly used technique for this purpose. Typically, a number of candidate models are defined a priori and ranked according to their expected out-of-sample performance. Model selection, however, only assesses the relative performance of the models and, as pointed out in a recent paper, a large proportion of ecology papers that use model selection do not assess the absolute fit of the ‘best’ model. In this paper, it is argued that assessing the absolute fit of the ‘best’ model alone does not go far enough. This is because a model that appears to perform well under model selection is also likely to appear to perform well under measures of absolute fit, even when there is no predictive value. A model selection permutation test is proposed that assesses the probability that the model selection statistic of the ‘best’ model could have occurred by chance alone, whilst taking account of dependencies between the models. It is argued that this test should always be performed as a part of formal model selection. The test is demonstrated on two real population modelling examples of ibex in northern Italy and wild reindeer in Norway.
References
Bargmann, T., Wheatcroft, E., Imperio, S. and Vetaas, O.R., 2020. Effects of weather and hunting on wild reindeer population dynamics in Hardangervidda National Park. Population Ecology.
Benjamini, Yoav, and Yosef Hochberg. 1995. “Controlling the False Discovery Rate: a Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society: Series B (Methodological) 57 (1): 289–300.
Bernardo, José M. 1979. “Expected Information as Expected Utility.” the Annals of Statistics: 686–690.
Bolker, Benjamin M, Mollie E Brooks, Connie J Clark, Shane W Geange, John R Poulsen, M Henry H Stevens, and Jada-Simone S White. 2009. “Generalized Linear Mixed Models: a Practical Guide for Ecology and Evolution.” Trends in Ecology & Evolution 24 (3): 127–135.
Bonferroni, Carlo E. 1936. “Teoria Statistica Delle Classi E Calcolo Delle Probabilita: Libreria Internazionale Seeber.”
Burnham, Kenneth P, and David R Anderson. 2001. “Kullback-Leibler Information as a Basis for Strong Inference in Ecological Studies.” Wildlife Research 28 (2): 111–119.
Burnham, K. P. & Anderson, D. R. (2004), ‘Multimodel inference: understanding aic and bic in model selection’, Sociological methods & research. 669 33(2), 261–304.
Claeskens, Gerda, and Nils Lid Hjort. 2008. Model Selection and Model Averaging. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press. doi:10.1017/CBO9780511790485.
Gneiting, Tilmann, and Adrian E Raftery. 2007. “Strictly Proper Scoring Rules, Prediction, and Estimation.” Journal of the American Statistical Association 102 (477): 359–378.
Good, I. J. 1952. “Rational Decisions.” Journal of the Royal Statistical Society: Series B 14: 107–114.
Holm, Sture. 1979. “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics: 65–70.
Imperio, Simona, Radames Bionda, Ramona Viterbi, and Antonello Provenzale. 2013. “Climate Change and Human Disturbance Can Lead to Local Extinction of Alpine Rock Ptarmigan: New Insight from the Western Italian Alps.” PloS One 8 (11): e81598.
J.Bröcker, and L. Smith. 2007. “Scoring Probabilistic Forecasts: the Importance of Being Proper.” Tellus A 22 (2).
Jacobson, Andrew R., Antonello Provenzale, Achaz von Hardenberg, Bruno Bassano, and Marco Festa-Bianchet. 2004. “CLIMATE FORCING AND DENSITY DEPENDENCE IN a MOUNTAIN UNGULATE POPULATION.” Ecology 85 (6): 1598–1610. doi:10.1890/02-0753. http://dx.doi.org/10.1890/02-0753.
Johnson, Jerald B, and Kristian S Omland. 2004. “Model Selection in Ecology and Evolution.” Trends in Ecology & Evolution 19 (2): 101–108.
Mac Nally, Ralph, Richard P. Duncan, James R. Thomson, and Jian D. L. Yen. 2017. “Model Selection Using Information Criteria, but Is the ‘Best’ Model Any Good?” Journal of Applied Ecology: n/a–n/a. doi:10.1111/1365-2664.13060. http://dx.doi.org/10.1111/1365-2664.13060.
Murphy, Allan H, and Edward S Epstein. 1989. “Skill Scores and Correlation Coefficients in Model Verification.” Monthly Weather Review 117 (3): 572–582.
Nakagawa, Shinichi. 2004. “A Farewell to Bonferroni: the Problems of Low Statistical Power and Publication Bias.” Behavioral Ecology 15 (6): 1044–1045.
Roulston, Mark S., and Leonard A. Smith. 2002. “Evaluating Probabilistic Forecasts Using Information Theory.” Monthly Weather Review 130: 1653–1660.
Šidák, Zbyněk. 1967. “Rectangular Confidence Regions for the Means of Multivariate Normal Distributions.” Journal of the American Statistical Association 62 (318): 626–633.
Stone, Mervyn. 1974. “Cross-Validatory Choice and Assessment of Statistical Predictions.” Journal of the Royal Statistical Society: Series B (Methodological) 36 (2): 111–133.
Symonds, Matthew RE, and Adnan Moussalli. 2011. “A Brief Guide to Model Selection, Multimodel Inference and Model Averaging in Behavioural Ecology Using Akaike’s Information Criterion.” Behavioral Ecology and Sociobiology 65 (1): 13–21.
Thieme, Horst R. 2018. Mathematics in Population Biology. Princeton University Press.
Wagenmakers, Eric-Jan., and Simon. Farrell. 2004. “AIC Model Selection Using Akaike Weights.” Psychonomic Bulletin and Review 11 (1): 192–196.
Westfall, Peter H, S Stanley Young, and S Paul Wright. 1993. “On Adjusting P-Values for Multiplicity.” Biometrics 49 (3): 941–945.
Wheatcroft, Edward. 2015. “Improving Predictability of the Future by Grasping Probability Less Tightly.” PhD thesis, The London School of Economics; Political Science (LSE).
Downloads
Published
Issue
Section
License
Copyright (c) 2021 Edward Wheatcroft
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright in their articles.
Articles in the European Journal of Ecology published 2020 and after are made available under a Creative Commons Attribution 4.0 license.
Articles in the European Journal of Ecology published 2015-2019 are made available under a Creative Commons Attribution-NonCommercial-NoDerivs 4.0 license.
