How to cite this WIREs title:
WIREs Comp Stat

Bayesian and frequentist testing for differences between two groups with parametric and nonparametric two‐sample tests

Overview

Riko Kelter

Published Online: Jul 13 2020

DOI: 10.1002/wics.1523

Full article on Wiley Online Library: HTML PDF

Can't access this content? Tell your librarian.

Abstract Testing for differences between two groups is one of the scenarios most often faced by scientists across all domains and is particularly important in the medical sciences and psychology. The traditional solution to this problem is rooted inside the Neyman–Pearson theory of null hypothesis significance testing and uniformly most powerful tests. In the last decade, a lot of progress has been made in developing Bayesian versions of the most common parametric and nonparametric two‐sample tests, including Student's t‐test and the Mann–Whitney U test. In this article, we review the underlying assumptions, models and implications for research practice of these Bayesian two‐sample tests and contrast them with the existing frequentist solutions. Also, we show that in general Bayesian and frequentist two‐sample tests have different behavior regarding the type I and II error control, which needs to be carefully balanced in practical research. This article is categorized under: Statistical and Graphical Methods of Data Analysis > Bayesian Methods and Theory Statistical and Graphical Methods of Data Analysis > Monte Carlo Methods Statistical and Graphical Methods of Data Analysis > Markov Chain Monte Carlo

Left: Model for the Bayesian parametric two‐sample t‐test of Rouder et al. (2009), which is itself a special case of the model of Gronau et al. (2019). Data x_i and y_i is distributed and with grand mean μ, standard deviation σ and total effect α = δσ. The grand mean μ is assigned a flat prior p(μ) = 1, σ² is assigned Jeffreys prior p(σ²) = 1/σ² and the effect size δ is assigned a C(0, γ) prior; Right: Model of the Bayesian Wilcoxon rank sum test used for Gibbs sampling by van Doorn et al. (2020): The ranks and are ranks produced by the non‐observed latent variables and , each following a normal distribution with σ = 1 and shifted means and . The effect size δ is generated as , where g gets assigned a hyperprior itself

[ Normal View | Magnified View ]

Left: Model for the Bayesian parametric two‐sample t‐test of Kruschke (2013). Data y_[i|j] is the ith observation in the jth group, j = 1, 2. y_[i|j] is distributed y_[i|j] ∼ t_ν(μ_j, σ_j), where ν~exp(K), and ; Right: Two‐component Gaussian mixture‐model with known allocations of Kelter (2020c), where for i = 1, 2 and priors μ_i ∼ N(b₀, B₀) and are used

[ Normal View | Magnified View ]

Type II error rates for Bayesian and frequentist two‐sample tests under different distributions

[ Normal View | Magnified View ]

Type I error rates for Bayesian and frequentist two‐sample tests under different distributions

[ Normal View | Magnified View ]

Prior and posterior plot of the effect size δ for the nonparametric Bayesian Mann–Whitney U test of van Doorn et al. (2020) when using a prior on δ in the computer science education data set of Kelter et al. (2018)

[ Normal View | Magnified View ]

Top: Prior and posterior plot of the effect size δ for the parametric Bayesian two‐sample test of Gronau et al. (2019) when using a prior on δ in the kitchen rolls data set of Wagenmakers et al. (2015); Bottom left: Robustness check for BF₀₁ for varying prior width; Bottom right: Sequential analysis of how BF₁₀ changes when each observation is gradually incorporated into the analysis

[ Normal View | Magnified View ]

References

Benjamin,, D. J., Berger,, J. O., Johannesson,, M., Nosek,, B. A., Wagenmakers,, E.‐J., Berk,, R., … Johnson,, V. E. (2018). Redefine statistical significance. Nature Human Behaviour, 2, 6–10. Retrieved from http://www.nature.com/articles/s41562-017-0189-z
Berger,, J., Brown,, L., & Wolpert,, R. (1994). A unified conditional frequentist and Bayesian test for fixed and sequential hypothesis testing. The Annals of Statistics, 22, 1787–1807. Retrieved from http://projecteuclid.org/euclid.aos/1176345976
Berger,, J. O. (2003). Could Fisher, Jeffreys and Neyman have agreed on testing? Statistical Science, 18, 1–32. Retrieved from www.stat.duke.edu/
Berger,, J. O., Boukai,, B., & Wang,, Y. (1997). Unified frequentist and Bayesian testing of a precise hypothesis. Statistical Science, 12, 133–160.
Berger,, J. O., & Wolpert,, R. L. (1988). The likelihood principle. Hayward, CA: Institute of Mathematical Statistics. Retrieved from http://www.jstor.org/stable/4355509
Carpenter,, B., Guo,, J., Hoffman,, M. D., Brubaker,, M., Gelman,, A., Lee,, D., … Betancourt,, M. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76, 1–32.
Casella,, G., & Berger,, R. L. (2002). Statistical inference. Stamford, CT: Thomson Learning.
Cohen,, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Routledge.
Colquhoun,, D. (2017). The reproducibility of research and the misinterpretation of p‐values. Royal Society Open Science, 4, 171085. https://doi.org/10.1098/rsos.171085
Cox,, D. (1958). Some problems connected with statistical inference. The Annals of Mathematical Statistics, 29, 357–372. Retrieved from http://projecteuclid.org/euclid.aoms/1177706618
Cumming,, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29.
Escobar,, M. D., & West,, M. (1995). Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90, 577–588.
Fisher,, R. A. (1925). Statistical methods for research workers. Edinburgh, England: Oliver and Boyd, Hafner Publishing Company.
Frühwirth‐Schnatter,, S. (2006). Finite mixture and Markov switching models. Berlin: Springer.
Gabry,, J., Simpson,, D., Vehtari,, A., Betancourt,, M., & Gelman,, A. (2019). Visualization in Bayesian workflow. Journal of the Royal Statistical Society: Series A (Statistics in Society), 182, 389–402. https://doi.org/10.1111/rssa.12378
Gelman,, A., Lee,, D., & Guo,, J. (2015). Stan: A probabilistic programming language for Bayesian inference. Journal of Educational and Behavioral Statistics, 40, 530–543. https://doi.org/10.3102/1076998615606113
Geman,, S., & Geman,, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI‐6, 721–741.
Goddard,, S. D., & Johnson,, V. E. (2016). Restricted most powerful Bayesian tests for linear models. Scandinavian Journal of Statistics, 43, 1162–1177. https://doi.org/10.1111/sjos.12235
Gönen,, M., Johnson,, W. O., Lu,, Y., & Westfall,, P. H. (2005). The Bayesian two‐sample t test. The American Statistician, 59, 252–257.
Goodman,, S. N. (1999). Toward evidence‐based medical statistics. 2: The Bayes factor. Annals of Internal Medicine, 130, 1005–1013. https://doi.org/10.7326/0003-4819-130-12-199906150-00019
Gronau,, Q. F., Ly,, A., & Wagenmakers,, E.‐J. (2019). Informed Bayesian t‐tests. The American Statistician, 00, 1–7.
Held,, L., & Ott,, M. (2016). How the maximal evidence of p‐values against point hypotheses depends on sample size. The American Statistician, 70, 335–341. https://doi.org/10.1080/00031305.2016.1209128
Held,, L., & Ott,, M. (2018). On p‐values and Bayes factors. Annual Review of Statistics and its Application, 5, 393–419. https://doi.org/10.1146/annurev-statistics-031017-100307
Held,, L., & Sabanés Bové,, D. (2014). Applied statistical inference. Berlin, Heidelberg: Springer.
Hoffman,, M. D., & Gelman,, A. (2014). The no‐U‐turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15, 1351–1381. Retrieved from http://arxiv.org/abs/1111.4246
Ioannidis,, J. P. (2005). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association, 294, 218–228. https://doi.org/10.1001/jama.294.2.218
Jeffreys,, H. (1939). Theory of probability (1st ed.). Oxford, England: The Clarendon Press.
Jeffreys,, H. (1961). Theory of probability (3rd ed.). Oxford, England: Oxford University Press.
Kass,, R. E., & Raftery,, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.
Kelter,, R. (2019). bayest: Effect size targeted Bayesian two‐sample t‐tests via Markov chain Monte Carlo in Gaussian mixture models. Comprehensive R Archive Network. Retrieved from https://cran.r-project.org/web/packages/bayest/index.html
Kelter,, R. (2020a). Analysis of Bayesian posterior significance and effect size indices for the two‐sample t‐test to support reproducible medical research. BMC Medical Research Methodology, 20, 88.
Kelter,, R. (2020b). Bayesian alternatives to hypothesis significance testing in biomedical research: A non‐technical introduction to Bayesian inference with JASP. BMC Medical Research Methodology, 20, 142. https://doi.org/10.1186/s12874-020-00980-6
Kelter,, R. (2020c). bayest: An R package for effect‐size targeted Bayesian two‐sample t‐tests. Journal of Open Research Software, 8(1), 1–4. https://doi.org/10.5334/jors.290.
Kelter,, R., Kramer,, M., & Brinda,, T. (2018). Teachers perspectives on object‐oriented programming environments for secondary education. In Proceedings of the IFIP TC3 open conference on comp. Cham, Switzerland: International Federation for Information Processing (IFIP).
Kruschke,, J. K. (2013). Bayesian estimation supersedes the t‐test. Journal of Experimental Psychology: General, 142, 573–603.
Kruschke,, J. K. (2015). Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan (2nd ed.). Oxford, England: Academic Press.
Kruschke,, J. K. (2018). Rejecting or accepting parameter values in Bayesian estimation. Advances in Methods and Practices in Psychological Science, 1(2), 270–280.
Kruschke,, J. K., & Liddell,, T. (2018). The Bayesian new statistics: Hypothesis testing, estimation, meta‐analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin and Review, 25, 178–206.
Kruschke,, J. K., & Liddell,, T. M. (2017). The Bayesian new statistics: From a Bayesian perspective (pp. 1–28). Berlin: Springer.
Lee,, M. D., & Wagenmakers,, E.‐J. (2013). Bayesian cognitive modeling: A practical course. Amsterdam, Netherlands: Cambridge University Press.
Lehmann,, E. L. E. L., & Casella,, G. (1998). Theory of point estimation. New York, NY: Springer.
Liang,, F., Paulo,, R., Molina,, G., Clyde,, M. A., & Berger,, J. O. (2008). Mixtures of g priors for Bayesian variable selection. Journal of the American Statistical Association, 103, 410–423.
Lindley,, D. (1957). A statistical paradox. Biometrika, 44, 187–192.
Makowski,, D., Ben‐Shachar,, M. S., Chen,, S. H. A., & Lüdecke,, D. (2019). Indices of effect existence and significance in the Bayesian framework. Frontiers in Psychology, 10, 2767. https://doi.org/10.3389/fpsyg.2019.02767/full
Marin,, J.‐M., & Robert,, C. (2014). Bayesian essentials with R. New York, NY: Springer.
Matthews,, R., Wasserstein,, R., & Spiegelhalter,, D. (2017). The ASA`s p‐value statement, one year on. Significance, 14, 38–41. https://doi.org/10.1111/j.1740-9713.2017.01021.x
Matzke,, D., Nieuwenhuis,, S., van Rijn,, H., Slagter,, H. A., van der Molen,, M. W., & Wagenmakers,, E.‐J. (2015). The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration. Journal of Experimental Psychology: General, 144, e1–e15. https://doi.org/10.1037/xge0000038
McElreath,, R. (2016). Statistical rethinking: A Bayesian course with examples in R and Stan. Boca Raton, FL: Chapman %26 Hall, CRC Press. https://doi.org/10.3102/1076998616659752
McElreath,, R., & Smaldino,, P. E. (2015). Replication, communication, and the population dynamics of scientific discovery. PLoS One, 10, 1–16.
Mills,, J. (2017) Objective Bayesian hypothesis testing (Ph.D. thesis). University of Cincinnati. Retrieved from https://economics.ku.edu/sites/economics.ku.edu/files/files/Seminar/papers1718/april20.pdf
Neyman,, A. J., & Pearson,, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika, 20, 175–240.
Neyman,, J. (1957). “Inductive behavior” as a basic concept of philosophy of science. Review of the International Statistical Institute, 25, 7–22.
Neyman,, J., & Pearson,, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society A, 231, 289–337. Retrieved from http://rsta.royalsocietypublishing.org/content/231/694-706/289
Neyman,, J., & Pearson,, E. S. (1936). Contributions to the theory of testing statistical hypotheses. Statistical Research Memoirs, 1, 1–37. Retrieved from https://psycnet.apa.org/record/1936-05541-001
Nuijten,, M. B., Hartgerink,, C. H., van Assen,, M. A., Epskamp,, S., & Wicherts,, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48, 1205–1226.
Pereira,, C. A. d. B., & Stern,, J. M. (1999). Evidence and credibility: Full Bayesian significance test for precise hypotheses. Entropy, 1, 99–110.
Pereira,, C. A. d. B., Stern,, J. M., & Wechsler,, S. (2008). Can a significance test be genuinely Bayesian? Bayesian Analysis, 3, 79–100.
Quintana,, D. S., & Williams,, D. R. (2018). Bayesian alternatives for common ‐hypothesis significance tests in psychiatry: A non‐technical guide using JASP. BMC Psychiatry, 18, 178.
R Core Team. (2020). R: A Language and environment for statistical computing. Retrieved from https://www.r-project.org/
Richardson,, S., & Green,, P. (1997). On Bayesian analysis of mixtures with an unknown number of components. Journal of the Royal Statistical Society: Series B, 59, 731–792.
Robert,, C. P. (2016). The expected demise of the Bayes factor. Journal of Mathematical Psychology, 72, 33–37.
Rochon,, J., Gondan,, M., & Kieser,, M. (2012). To test or not to test: Preliminary assessment of normality when comparing two independent samples. BMC Medical Research Methodology, 12, 1–11.
Rouder,, J. N., Speckman,, P. L., Sun,, D., Morey,, R. D., & Iverson,, G. (2009). Bayesian t tests for accepting and rejecting the hypothesis. Psychonomic Bulletin and Review, 16, 225–237.
Samaniego,, F. J. (2010). A comparison of the Bayesian and frequentist approaches to estimation. In Springer series in statistics. New York, NY: Springer New York. https://doi.org/10.1007/978-1-4419-5941-6
Tanner,, M., & Wong,, W. H. (1987). The calculation of posterior distributions by data augmentation. Journal of the American Statistical Association, 82, 528–540.
Topolinski,, S., & Sparenberg,, P. (2012). Turning the hands of time. Social Psychological and Personality Science, 3, 308–314. https://doi.org/10.1177/1948550611419266
van Doorn,, J., Ly,, A., Marsman,, M., & Wagenmakers,, E.‐J. (2020). Bayesian rank‐based hypothesis testing for the rank sum test, the signed rank test, and Spearman`s rho. Journal of Applied Statistics, 1–23. Retrieved from https://www.tandfonline.com/doi/full/10.1080/02664763.2019.1709053.
Wagenmakers,, E.‐J., Beek,, T., Rotteveel,, M., Gierholz,, A., Matzke,, D., Steingroever,, H., … Pinto,, Y. (2015). Turning the hands of time again: A purely confirmatory replication study and a Bayesian analysis. Frontiers in Psychology, 6, 494. https://doi.org/10.3389/fpsyg.2015.00494.
Wagenmakers,, E.‐J., Lodewyckx,, T., Kuriyal,, H., & Grasman,, R. (2010). Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method. Cognitive Psychology, 60, 158–189.
Wang,, M., & Liu,, G. (2016). A simple two‐sample Bayesian t‐test for hypothesis testing. American Statistician, 70, 195–201.
Wasserstein,, R. L., & Lazar,, N. A. (2016). The ASA`s statement on p‐values: Context, process, and purpose. The American Statistician, 70, 129–133.
Wasserstein,, R. L., Schirm,, A. L., & Lazar,, N. A. (2019). Moving to a world beyond “p %3C 0.05”. The American Statistician, 73, 1–19.
Wetzels,, R., Raaijmakers,, J. G., Jakab,, E., & Wagenmakers,, E.‐J. (2009). How to quantify support for and against the hypothesis: A flexible WinBUGS implementation of a default Bayesian t test. Psychonomic Bulletin and Review, 16, 752–760.
Wilcox,, R. R. (1998). How many discoveries have been lost by ignoring modern statistical methods? American Psychologist, 53, 300–314.
Yuan,, Y., & Johnson,, V. E. (2008). Bayesian hypothesis tests using nonparametric statistics. Statistica Sinica, 18, 1185–1200.
Zellner,, A. (1980). Introduction. In Bayesian analysis in econometrics and statistics: Essays in honor of Harold Jeffreys. Amsterdam, Netherlands: North‐Holland Pub. Co..

Access to this WIREs title is by subscription only.

Recommend to Your
Librarian Now!

The latest WIREs articles in your inbox

Sign Up for Article Alerts