Hyperparameters and tuning strategies for random forest

Advanced Review

Philipp Probst, Marvin N. Wright, Anne‐Laure Boulesteix

Published Online: Jan 28 2019

DOI: 10.1002/widm.1301

Full article on Wiley Online Library: HTML PDF

Can't access this content? Tell your librarian.

The random forest (RF) algorithm has several hyperparameters that have to be set by the user, for example, the number of observations drawn randomly for each tree and whether they are drawn with or without replacement, the number of variables drawn randomly for each split, the splitting rule, the minimum number of samples that a node must contain, and the number of trees. In this paper, we first provide a literature review on the parameters' influence on the prediction performance and on variable importance measures. It is well known that in most cases RF works reasonably well with the default values of the hyperparameters specified in software packages. Nevertheless, tuning the hyperparameters can improve the performance of RF. In the second part of this paper, after a presenting brief overview of tuning strategies, we demonstrate the application of one of the most established tuning strategies, model‐based optimization (MBO). To make it easier to use, we provide the tuneRanger R package that tunes RF with MBO automatically. In a benchmark study on several datasets, we compare the prediction performance and runtime of tuneRanger with other tuning implementations in R and RF with default hyperparameters. This article is categorized under: Algorithmic Development > Biological Data Mining Algorithmic Development > Statistics Algorithmic Development > Hierarchies and Trees Technologies > Machine Learning

Boxplots of performance differences to ranger default. On the left side the boxplots with outliers are depicted and on the right side the same plots without outliers. For the error rate, the brier score and the logarithmic loss, low values are better, while for the area under the curve (AUC), high values are preferable. If the tuned measure equals the evaluation measure, the boxplot is displayed in gray

[ Normal View | Magnified View ]

Average runtime of the different algorithms on different datasets (upper plot: Unscaled, lower plot: Logarithmic scale). The datasets are ordered by the average runtime of the tuneRangerMMCE algorithm

[ Normal View | Magnified View ]

References

Belgiu,, M., & Drăguţ,, L. (2016). Random forest in remote sensing: A review of applications and future directions. ISPRS Journal of Photogrammetry and Remote Sensing, 114, 24–31.
Bergstra,, J., & Bengio,, Y. (2012). Random search for hyper‐parameter optimization. Journal of Machine Learning Research, 13, 281–305.
Bernard,, S., Heutte,, L., & Adam,, S. (2009). Influence of hyperparameters on random forest accuracy. In MCS, vol. 5519 of Lecture Notes in Computer Science (pp. 171–180). Springer.
Biau,, G., & Scornet,, E. (2016). A random forest guided tour. TEST, 25, 197–227.
Birattari,, M., Yuan,, Z., Balaprakash,, P., & Stützle,, T. (2010). F‐race and iterated F‐race: An overview. In Experimental methods for the analysis of optimization algorithms (pp. 311–336). Berlin, Germany: Springer.
Bischl,, B., Casalicchio,, G., Feurer,, M., Hutter,, F., Lang,, M., Mantovani,, R. G., van Rijn,, J. N., & Vanschoren,, J. (2017). OpenML benchmarking suites and the OpenML100. ArXiv preprint arXiv:1708.03731. Retreived from https://arxiv.org/abs/1708.03731
Bischl,, B., Lang,, M., Kotthoff,, L., Schiffner,, J., Richter,, J., Studerus,, E., … Jones,, Z. M. (2016). mlr: Machine learning in R. Journal of Machine Learning Research, 17, 1–5.
Bischl,, B., Richter,, J., Bossek,, J., Horn,, D., Thomas,, J., & Lang,, M. (2017). mlrMBO: A modular framework for model‐based optimization of expensive black‐box functions. ArXiv preprint arXiv:1703.03373. Retreived from https://arxiv.org/abs/1703.03373
Bischl,, B., Schiffner,, J., & Weihs,, C. (2013). Benchmarking local classification methods. Computational Statistics, 28, 2599–2619.
Bohachevsky,, I. O., Johnson,, M. E., & Stein,, M. L. (1986). Generalized simulated annealing for function optimization. Technometrics, 28, 209–217.
Boulesteix,, A.‐L., Bender,, A., Lorenzo Bermejo,, J., & Strobl,, C. (2012). Random forest gini importance favours snps with large minor allele frequency: Impact, sources and recommendations. Briefings in Bioinformatics, 13, 292–304.
Boulesteix,, A.‐L., Binder,, H., Abrahamowicz,, M., & Sauerbrei,, W. (2018). On the necessity and design of studies comparing statistical methods. Biometrical Journal, 60, 216–218.
Boulesteix,, A.‐L., Janitza,, S., Kruppa,, J., & König,, I. R. (2012). Overview of random forest methodology and practical guidance with emphasis on computational biology and bioinformatics. WIREs: Data Mining and Knowledge Discovery, 2, 493–507.
Boulesteix,, A.‐L., Wilson,, R., & Hapfelmeier,, A. (2017). Towards evidence‐based computational statistics: Lessons from clinical research on the role and design of real‐data benchmark studies. BMC Medical Research Methodology, 17, 138.
Breiman,, L. (1996). Out‐of‐bag estimation. Technical Report, UC Berkeley, Department of Statistics.
Breiman,, L. (2001). Random forests. Machine Learning, 45, 5–32.
Casalicchio,, G., Bossek,, J., Lang,, M., Kirchhoff,, D., Kerschke,, P., Hofner,, B., … Bischl,, B. (2017). OpenML: An R package to connect to the machine learning platform OpenML. Computational Statistics, 32, 1–15.
Criminisi,, A., Shotton,, J., & Konukoglu,, E. (2012). Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semi‐supervised learning. Foundations and Trends in Computer Graphics and Vision, 7, 81–227.
Díaz‐Uriarte,, R., & De Andres,, S. A. (2006). Gene selection and classification of microarray data using random forest. BMC Bioinformatics, 7, 3.
Fernández‐Delgado,, M., Cernadas,, E., Barro,, S., & Amorim,, D. (2014). Do we need hundreds of classifiers to solve real world classification problems. Journal of Machine Learning Research, 15, 3133–3181.
Ferri,, C., Hernández‐Orallo,, J., & Modroiu,, R. (2009). An experimental comparison of performance measures for classification. Pattern Recognition Letters, 30, 27–38.
Genuer,, R., Poggi,, J.‐M., & Tuleau,, C. (2008). Random forests: Some methodological insights. ArXiv preprint arXiv:0811.3619. Retreived from https://arxiv.org/abs/0811.3619
Genuer,, R., Poggi,, J.‐M., & Tuleau‐Malot,, C. (2010). Variable selection using random forests. Pattern Recognition Letters, 31, 2225–2236.
Geurts,, P., Ernst,, D., & Wehenkel,, L. (2006). Extremely randomized trees. Machine Learning, 63, 3–42.
Goldstein,, B. A., Polley,, E. C., & Briggs,, F. (2011). Random forests for genetic association studies. Statistical Applications in Genetics and Molecular Biology, 10, 32.
Grömping,, U. (2009). Variable importance assessment in regression: Linear regression versus random forest. The American Statistician, 63, 308–319.
Hastie,, T., Tibshirani,, R., & Friedman,, J. (2001). The Elements of Statistical Learning. Springer series in statistics. New York, NY: Springer New York Inc.
Hothorn,, T., Hornik,, K., & Zeileis,, A. (2006). Unbiased recursive partitioning: A conditional inference framework. Journal of Computational and Graphical Statistics, 15, 651–674.
Hothorn,, T., & Zeileis,, A. (2015). Partykit: A modular toolkit for recursive partytioning in R. Journal of Machine Learning Research, 16, 3905–3909.
Hutter,, F., Hoos,, H. H. and Leyton‐Brown,, K. (2011) Sequential model‐based optimization for general algorithm configuration, 507–523. Berlin and Heidelberg, Germany: Springer Berlin Heidelberg.
Janitza,, S., Binder,, H., & Boulesteix,, A.‐L. (2016). Pitfalls of hypothesis tests and model selection on bootstrap samples: Causes and consequences in biometrical applications. Biometrical Journal, 58, 447–473.
Janitza,, S., & Hornung,, R. (2018). On the overestimation of random forest`s out‐of‐bag error. PLoS One, 13, e0201904.
Jones,, D. R., Schonlau,, M., & Welch,, W. J. (1998). Efficient global optimization of expensive black‐box functions. Journal of Global Optimization, 13, 455–492.
Kuhn,, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28, 1–26.
Liaw,, A., & Wiener,, M. (2002). Classification and regression by randomForest. R News, 2, 18–22.
Lin,, Y., & Jeon,, Y. (2006). Random forests and adaptive nearest neighbors. Journal of the American Statistical Association, 101, 578–590.
Lunetta,, K. L., Hayward,, L. B., Segal,, J., & Van Eerdewegh,, P. (2004). Screening large‐scale association study data: Exploiting interactions using random forests. BMC Genetics, 5, 32.
Mantovani,, R. G., Rossi,, A. L., Vanschoren,, J., Bischl,, B., & Carvalho,, A. C. (2015). To tune or not to tune: Recommending when to adjust svm hyper‐parameters via meta‐learning. In Neural Networks (IJCNN), 2015 International Joint Conference (pp. 1–8). IEEE.
Martínez‐Muñoz,, G., & Suárez,, A. (2010). Out‐of‐bag estimation of the optimal sample size in bagging. Pattern Recognition, 43, 143–152.
Oshiro,, T. M., Perez,, P. S., & Baranauskas,, J. A. (2012). How many trees in a random forest? In Machine Learning and Data Mining in Pattern Recognition: 8th International Conference, MLDM 2012, Berlin, Germany, July 13–20, 2012, Proceedings, Vol. 7376, 154. Springer.
Probst,, P. (2017) OOBCurve: Out of bag learning curve. R package version 0.2.
Probst,, P. (2018) tuneRanger: Tune random forest of the ‘ranger’ package. R package version 0.1.
Probst,, P., Bischl,, B., & Boulesteix,, A.‐L. (2018). Tunability: Importance of hyperparameters of machine learning algorithms. ArXiv preprint arXiv:1802.09596. Retreived from https://arxiv.org/abs/1802.09596.
Probst,, P., & Boulesteix,, A.‐L. (2017). To tune or not to tune the number of trees in a random forest? Journal of Machine Learning Research, 18, 1–18.
Richter,, J. (2017) mlrHyperopt: Easy hyperparameter optimization with mlr and mlrMBO. R package version 0.0.1.
Schiffner,, J., Bischl,, B., Lang,, M., Richter,, J., Jones,, Z. M., Probst,, P., Pfisterer,, F., Gallo,, M., Kirchhoff,, D., Kühn,, T., Thomas,, J., & Kotthoff,, L. (2016). mlr tutorial. ArXiv preprint arXiv:1609.06146. Retreived from https://arxiv.org/abs/1609.06146
Scornet,, E. (2018). Tuning parameters in random forests. ESAIM: Proceedings And Surveys, 60, 144–162.
Segal,, M. R. (2004). Machine learning benchmarks and random forest regression. UCSF: Center for Bioinformatics and Molecular Biostatistics. Retrieved from https://escholarship.org/uc/item/35x3v9t4
Seibold,, H., Bernau,, C., Boulesteix,, A.‐L., & De Bin,, R. (2018). On the choice and influence of the number of boosting steps for high‐dimensional linear cox‐models. Computational Statistics, 33, 1195–1215.
Shmueli,, G. (2010). To explain or to predict? Statistical Science, 25, 289–310.
Strobl,, C., Boulesteix,, A.‐L., Zeileis,, A., & Hothorn,, T. (2007). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics, 8, 25.
van Rijn,, J. N., & Hutter,, F. (2018). Hyperparameter importance across datasets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery %26 Data Mining (pp. 2367–2376). ACM.
Vanschoren,, J., van Rijn,, J. N., Bischl,, B., & Torgo,, L. (2013). OpenML: Networked science in machine learning. SIGKDD Explorations, 15, 49–60.
Wright,, M. N., Dankowski,, T., & Ziegler,, A. (2017). Unbiased split variable selection for random survival forests using maximally selected rank statistics. Statistics in Medicine, 36, 1272–1284.
Wright,, M. N., & Ziegler,, A. (2017). Ranger: A fast implementation of random forests for high dimensional data in C++ and R. Journal of Statistical Software, 77, 1–17.
Wright,, M. N., Ziegler,, A., & König,, I. R. (2016). Do little interactions get lost in dark random forests? BMC Bioinformatics, 17, 145.
Ziegler,, A., & König,, I. R. (2014). Mining data with random forests: Current options for real‐world applications. WIREs: Data Mining and Knowledge Discovery, 4, 55–63.

Access to this WIREs title is by subscription only.

Recommend to Your
Librarian Now!

Hyperparameters and tuning strategies for random forest

Browse by Topic

Access to this WIREs title is by subscription only.

The latest WIREs articles in your inbox