How to cite this WIREs title:
WIREs Comp Stat

Statistical nonparametric mapping: Multivariate permutation tests for location, correlation, and regression problems in neuroimaging

Advanced Review

Nathaniel E. Helwig

Published Online: Jan 22 2019

DOI: 10.1002/wics.1457

Full article on Wiley Online Library: HTML PDF

Can't access this content? Tell your librarian.

Nonparametric statistical inference via permutation testing is on the rise in neuroimaging research. This rise in popularity is likely in response to recent studies that have demonstrated limitations of parametric inference in certain situations. Nonparametric tests have the appeal of requiring fewer assumptions than their parametric counterparts, and are often touted as being more flexible and more useful for small samples. Furthermore, recent studies have demonstrated the robustness of nonparametric methods in situations when parametric inference fails. As a result, many nonstatistical neuroimaging researchers are likely to believe that nonparametric permutation tests are always a “safe choice” because the results do not depend on distributional assumptions and/or large sample approximations. Alas, this commonly held belief is not entirely accurate, given that nonparametric tests still do rely on assumptions and/or approximations for valid statistical inference. When these assumptions are met, nonparametric permutation tests have the potential to produce valid inferential results for the intended hypotheses. However, as I demonstrate, when these assumptions are violated, nonparametric permutation tests can produce invalid and/or misleading results, which have important implications for the use of such methods in neuroimaging research. All hope is not lost though, as recent theoretical developments in nonparametric statistics can improve current implementations of permutation tests in neuroimaging research. This article is categorized under: Statistical and Graphical Methods of Data Analysis > Nonparametric Methods Statistical and Graphical Methods of Data Analysis > Bootstrap and Resampling Data: Types and Structure > Image and Spatial Data

Six univariate distributions with mean μ = 0 and variance σ² = 1

[ Normal View | Magnified View ]

Timeline of the number of publications on permutation (black solid line) and bootstrap (gray dashed line) resampling methods for neuroimaging data. Obtained from a Scopus search for publications with titles, abstracts, and/or keywords containing the given term in combination with one or more of the terms—EEG, MEG, PET, CT, MRI, fMRI, DTI, Neuroimage, Neuroimaging, Brain image, Brain imaging

[ Normal View | Magnified View ]

Regression test. Type I error rate calculated across 10,000 replications for the parametric tests (red squares) and the permutation tests (black circles). The unfilled points denote the results using the (classic) F‐test statistic, whereas the filled points denote the results using the (Wald) W‐test statistic. The vertical bars denote approximate 99% confidence intervals, and the dotted horizontal lines denote the nominal rate of α = 0.05

[ Normal View | Magnified View ]

The X‐Y slice with Z = 0 (top) and the Z‐Y slice with X = 0 (bottom) for three trivariate distributions with μ_j = 0 and for j ∈ {X, Y, Z}. The X‐Z slice with Y = 0 is identical to the top row. Note that ρ_XZ = ρ_XY = 0 for all three distributions, ρ_ZY = 0 for the spherical uniform, and ρ_ZY = 1/2 for the multivariate normal and multivariate t

[ Normal View | Magnified View ]

Correlation test. Type I error rate calculated across 10,000 replications for the parametric tests (red squares) and the permutation tests (black circles). The unfilled points denote the results using the (classic) T statistic, whereas the filled points denote the results using the (studentized) T^* statistic. The vertical bars denote approximate 99% confidence intervals, and the dotted horizontal lines denote the nominal rate of α = 0.05

[ Normal View | Magnified View ]

Three bivariate distributions with μ_X = μ_Y = 0, , and ρ_XY = 0

[ Normal View | Magnified View ]

Two‐sample location test. Type I error rate calculated across 10,000 replications for the t‐test (red squares) and the permutation test (black circles). The unfilled points denote results using the (Student) T statistic, whereas the filled points denote results using the (Welch) T^* statistic. The vertical bars denote approximate 99% confidence intervals, and the dotted horizontal lines denote the nominal rate of α = 0.05. Note that the second group is twice as large as the first group, that is, n₂ = 2n₁

[ Normal View | Magnified View ]

One‐sample location test. Type I error rate calculated across 10,000 replications for the t‐test (red squares) and the permutation test (black circles). The vertical bars denote approximate 99% confidence intervals, and the dotted horizontal lines denote the nominal rate of α = 0.05

[ Normal View | Magnified View ]

References

Arndt,, S., Cizadlo,, T., Andreasen,, N., Heckel,, D., Gold,, S., & O`Leary,, D. (1996). Tests for comparing images based on randomization and permutation methods. Journal of Cerebral Blood Flow and Metabolism, 16(6), 1271–1279. https://doi.org/10.1097/00004647-199611000-00023
Basu,, D. (1980). Randomization analysis of experimental data: The fisher randomization test. Journal of the American Statistical Association, 75(371), 575–582. https://doi.org/10.2307/2287648
Benjamini,, Y., & Yu,, B. (2013). The shuffle estimator for explainable variance in fMRI experiments. The Annals of Applied Statistics, 7(4), 2007–2033. https://doi.org/10.1214/13-AOAS681
Blair,, R. C., Higgins,, J. J., Karniski,, W., & Kromrey,, J. D. (1994). A study of multivariate permutation tests which may replace Hotelling`s T2 test in prescribed circumstances. Multivariate Behavioral Research, 29(2), 141–163. https://doi.org/10.1207/s15327906mbr2902 2
Blair,, R. C., & Karniski,, W. (1994). Distribution‐free statistical analyses of surface and volumetric maps. In R. W. Thatcher,, M. Hallett,, E. R. John,, & M. Huerta, (Eds.), Functional neuroimaging: Technical foundations. San Diego, CA: Academic Press.
Brunner,, E., & Munzel,, U. (2000). The nonparametric Behrens‐fisher problem: Asymptotic theory and a small‐sample approximation. Biometrical Journal, 42(1), 17–25. https://doi.org/10.1002/(SICI)1521-4036(200001)42:1h17::AID-BIMJ17i3.0.CO;2-U
Bullmore,, E., Brammer,, M., Williams,, S. C. R., Rabe‐Hesketh,, S., Janot,, N., David,, A., … Sham,, R. H. P. (1996). Statistical methods of estimation and inference for functional MR image analysis. Magnetic Resonance in Medicine, 35(2), 261–277. https://doi.org/10.1002/mrm.1910350219
Bullmore,, E., Long,, C., Suckling,, J., Fadili,, J., Calvert,, G., Zelaya,, F., … Brammer,, M. (2000). Colored noise and computational inference in neurophysiological (fMRI) time series analysis: Resampling methods in time and wavelet domains. Human Brain Mapping, 12(2), 61–78. https://doi.org/10.1002/1097-0193(200102)12:2h61::AID-HBM1004i3.0.CO;2-W
Bullmore,, E., Suckling,, J., Overmeyer,, S., Rabe‐Hesketh,, S., Taylor,, E., & Brammer,, M. J. (1999). Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain. IEEE Transactions on Medical Imaging, 18(1), 32–42. https://doi.org/10.1109/42.750253
Chung,, E., & Romano,, J. P. (2013). Exact and asymptotically robust permutation tests. The Annals of Statistics, 41(2), 484–507. https://doi.org/10.1214/13-AOS1090
Chung,, E., & Romano,, J. P. (2016). Asymptotically valid and exact permutation tests based on two‐sample U‐statistics. Journal of Statistical Planning and Inference, 168, 97–105. https://doi.org/10.1016/j.jspi.2015.07.004
De Mazière,, P. A., & Van Hulle,, M. M. (2007). fMRI bold signal analysis using a novel nonparametric statistical method. Journal of Magnetic Resonance, 185(1), 138–151. https://doi.org/10.1016/j.jmr.2006.12.001
DiCiccio,, C. J., & Romano,, J. P. (2017). Robust permutation tests for correlation and regression coefficients. Journal of the American Statistical Association, 112(519), 1211–1220. https://doi.org/10.1080/01621459.2016.1202117
Ding,, P. (2016). On the conditional distribution of the multivariate t distribution. The American Statistician, 70(3), 293–295. https://doi.org/10.1080/00031305.2016.1164756
Draper,, N. R., & Stoneman,, D. M. (1966). Testing for the inclusion of variables in linear regression by a randomisation technique. Technometrics, 8(4), 695–699. https://doi.org/10.2307/1266641
Edgington,, E. S. (1964). Randomization tests. The Journal of Psychology, 57(2), 445–449. https://doi.org/10.1080/00223980.1964.9916711
Edgington,, E. S. (1969). Approximate randomization tests. The Journal of Psychology, 72(2), 143–149. https://doi.org/10.1080/00223980.1969.10543491
Edgington,, E. S., & Onghena,, P. (2007). Randomization tests (Fourth ed.). Boca Raton, FL: Chapman and Hall/CRC.
Eklund,, A., Andersson,, M., Josephson,, C., Johannesson,, M., & Knutsson,, H. (2012). Does parametric fMRI analysis with SPM yield valid results?—An empirical study of 1484 rest datasets. NeuroImage, 61(3), 565–578. https://doi.org/10.1016/j.neuroimage.2012.03.093
Eklund,, A., Nichols,, T. E., & Knutsson,, H. (2016). Cluster failure: Why fMRI inferences for spatial extent have inated false‐positive rates. Proceedings of the National Academy of Sciences of the United States of America, 113(28), 7900–7905. https://doi.org/10.1073/pnas.1602413113
Etévenon,, P., Bertaut,, A., Mitermite,, F., Eustache,, F., Lepaisant,, J., Lechevalier,, B., & Zarifian,, E. (1989). Inter‐ and intra‐individual probability maps in EEG cartography by use of nonparametric fisher tests. Brain Topography, 2(1), 81–89. https://doi.org/10.1007/BF01128846
Etévenon,, P., Peron‐Magnan,, P., Guillou,, S., Toussaint,, M., Gueguen,, B., Boulenger,, J., … Loo,, H. (1988). Caffeine and EEG mapping: Effects of a visuo‐spatial task in healthy volunteers [cafeine et cartographie EEG: Effets d`une tache visuospatiale chez des volontaires sains. strategie d`analyse des donnees electropharmacologiques]. Neurophysiologie Clinique/Clinical Neurophysiology, 18(4), 355–367. https://doi.org/10.1016/S0987-7053(88)80092-3
Etévenon,, P., Tortrat,, D., & Benkelfat,, C. (1985). Electroencephalographic cartography. Neuropsychobiology, 13(3), 141–146. https://doi.org/10.1159/000118177
Etévenon,, P., Tortrat,, D., Guillou,, S., & Wendling,, B. (1985). EEG cartography during visuo‐spatial task: Averaged maps and group statistics [cartographie EEG au cours d`une tache visuo‐spatiale: Cartes moyennes et statistiques de groupes]. Re‐vue d`Electroencephalographie et de/Neurophysiologie Clinique, 15(2), 139–147. https://doi.org/10.1016/S0370-4475(85)80018-6
Fisher,, R. A. (1925). Statistical methods for research workers. Edinburgh, England: Oliver and Boyd.
Fisher,, R. A. (1935). The design of experiments. Edinburgh, England: Oliver and Boyd.
Fisher,, R. A. (1936). “The coefficient of racial likeness” and the future of craniometry. The Journal of the Royal Anthropological Institute of Great Britain and Ireland, 66, 57–63. https://doi.org/10.2307/2844116
Fisher,, R. A. (1960). The design of experiments (Seventh ed.). Edinburgh, England: Oliver and Boyd.
Flandin,, G., & Friston,, K. J. (2017). Analysis of family‐wise error rates in statistical parametric mapping using random field theory. Human Brain Mapping. https://doi.org/10.1002/hbm.23839
Freedman,, D., & Lane,, D. (1983). A nonstochastic interpretation of reported significance levels. Journal of Business and Economic Statistics, 1(4), 292–298. https://doi.org/10.2307/1391660
Fritsch,, V., Mota,, B. D., Loth,, E., Varoquaux,, G., Banaschewski,, T., Barker,, G. J., … Thirion,, B. (2015). Robust regression for large‐scale neuroimaging studies. NeuroImage, 111, 431–441. https://doi.org/10.1016/j.neuroimage.2015.02.048
Ganjgahi,, H., Winkler,, A. M., Glahn,, D. C., Blangero,, J., Kochunov,, P., & Nichols,, T. E. (2015). Fast and powerful heritability inference for family‐based neuroimaging studies. NeuroImage, 115, 256–268. https://doi.org/10.1016/j.neuroimage.2015.03.005
Ge,, T., Feng,, J., Hibar,, D. P., Thompson,, P. M., & Nichols,, T. E. (2012). Increasing power for voxel‐wise genome‐wide association studies: The random field theory, least square kernel machines and fast permutation procedures. NeuroImage, 63(2), 858–873. https://doi.org/10.1016/j.neuroimage.2012.07.012
Good,, P. I. (2005). Permutation, parametric, and bootstrap tests of hypotheses. New York, NY: Springer‐Verlag. https://doi.org/10.1007/b138696
Hayasaka,, S., & Nichols,, T. E. (2004). Combining voxel intensity and cluster extent with permutation test framework. NeuroImage, 23(1), 54–63. https://doi.org/10.1016/j.neuroimage.2004.04.035
Hesterberg,, T. C. (2015). What teachers should know about the bootstrap: Resampling in the undergraduate statistics curriculum. The American Statistician, 69(4), 371–386. https://doi.org/10.1080/00031305.2015.1089789
Hoeffding,, W. (1952). The large‐sample power of tests based on permutations of observations. The Annals of Mathematical Statistics, 23(2), 169–192. https://doi.org/10.1214/aoms/1177729436
Holm,, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
Holmes,, A. P., Blair,, R. C., Watson,, J. D. G., & Ford,, I. (1996). Nonparametric analysis of statistic images from functional mapping experiments. Journal of Cerebral Blood Flow and Metabolism, 16(1), 7–22. https://doi.org/10.1097/00004647-199601000-00002
Janssen,, A. (1997). Studentized permutation tests for non‐i.i.d. hypotheses and the generalized Behrens‐fisher problem. Statistics %26 Probability Letters, 36(1), 9–21. https://doi.org/10.1016/S0167-7152(97)00043-6
Johnson,, N. J. (1978). Modified t tests and confidence intervals for asymmetrical populations. Journal of the American Statistical Association, 73(363), 536–544. https://doi.org/10.2307/2286597
Karniski,, W., Blair,, R. C., & Snider,, A. D. (1994). An exact statistical method for comparing topographic maps, with any number of subjects and electrodes. Brain Topography, 6(3), 203–210. https://doi.org/10.1007/BF01187710
Kempthorne,, O. (1955). The randomization theory of experimental inference. Journal of the American Statistical Association, 50(271), 946–967. https://doi.org/10.2307/2281178
Kessler,, D., Angstadt,, M., & Sripada,, C. S. (2017). Reevaluating “cluster failure” in fMRI using nonparametric control of the false discovery rate. Proceedings of the National Academy of Sciences of the United States of America, 114(17), E3372–E3373. https://doi.org/10.1073/pnas.1614502114
Lehmann,, E. L., & Stein,, C. (1949). On the theory of some non‐parametric hypotheses. The Annals of Mathematical Statistics, 20(1), 28–45. https://doi.org/10.2307/2236802
Locascio,, J. J., Jennings,, P. J., Moore,, C. I., & Corkin,, S. (1997). Time series analysis in the time domain and resampling methods for studies of functional magnetic resonance brain imaging. Human Brain Mapping, 5(3), 168–193. https://doi.org/10.1002/(SICI)1097-0193(1997)5:3h168::AID-HBM3i3.0.CO;2-1
Mann,, H. B., & Whitney,, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics, 18(1), 50–60. https://doi.org/10.1214/aoms/1177730491
Medina,, J., Kimberg,, D. Y., Chatterjee,, A., & Coslett,, H. B. (2010). Inappropriate usage of the Brunner–Munzel test in recent voxel‐based lesion‐symptom mapping studies. Neuropsychologia, 48(1), 341–343. https://doi.org/10.1016/j.neuropsychologia.2009.09.016
Meriaux,, S., Roche,, A., Thirion,, B., & Dehaene‐Lambertz,, G. (2006, April). Robust statistics for nonparametric group analysis in fMRI. In 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro (pp. 936–939). doi: https://doi.org/10.1109/ISBI.2006.1625073
Neuhaus,, G. (1993). Conditional rank tests for the two‐sample problem under random censorship. The Annals of Statistics, 21(4), 1760–1779. https://doi.org/10.1214/aos/1176349396
Nichols,, T. E. (2012). Multiple testing corrections, nonparametric methods, and random field theory. NeuroImage, 62(2), 811–815. https://doi.org/10.1016/j.neuroimage.2012.04.014
Nichols,, T. E., & Hayasaka,, S. (2003). Controlling the familywise error rate in functional neuroimaging: A comparative review. Statistical Methods in Medical Research, 12(5), 419–446. https://doi.org/10.1191/0962280203sm341ra
Nichols,, T. E., Holmes,, A. P.. (1996). SnPM—Statistical nonparametric mapping—A toolbox for SPM. Retrieved from http://warwick.ac.uk/snpm
Nichols,, T. E., & Holmes,, A. P. (2002). Nonparametric permutation tests for functional neuroimaging: A primer with examples. Human Brain Mapping, 15(1), 1–25. https://doi.org/10.1002/hbm.1058
Pantazis,, D., Nichols,, T. E., Baillet,, S., & Leahy,, R. M. (2003, July). Spatiotemporal localization of significant activation in MEG using permutation tests. In C. Taylor, & J. A. Noble, (Eds.), Information processing in medical imaging (pp. 512–523). Berlin, Heidelberg: Springer Berlin Heidelberg.
Pantazis,, D., Nichols,, T. E., Baillet,, S., & Leahy,, R. M. (2005). A comparison of random field theory and permutation methods for the statistical analysis of MEG data. NeuroImage, 25(2), 383–394. https://doi.org/10.1016/j.neuroimage.2004.09.040
Pearson,, E. S. (1937). Some aspects of the problem of randomization. Biometrika, 29(1/2), 53–64. https://doi.org/10.2307/2332406
Pernet,, C. R., Latinus,, M., Nichols,, T. E., & Rousselet,, G. A. (2015). Cluster‐based computational methods for mass univariate analyses of event‐related brain potentials/fields: A simulation study. Journal of Neuroscience Methods, 250, 85–93. https://doi.org/10.1016/j.jneumeth.2014.08.003
Pesarin,, F. (2001). Multivariate permutation tests: With applications in biostatistics. New York, NY: Wiley.
Petsche,, H., Linder,, K., Rappelsberger,, P., & Gruber,, G. (1988). The EEG: An adequate method to concretize brain processes elicited by music. Music Perception, 6(2), 133–159. https://doi.org/10.2307/40285422
Pitman,, E. J. G. (1937a). Significance tests which may be applied to samples from any populations. Supplement to the Journal of the Royal Statistical Society, 4(1), 119–130. https://doi.org/10.2307/2984124
Pitman,, E. J. G. (1937b). Significance tests which may be applied to samples from any populations. II. The correlation coefficient test. Supplement to the Journal of the Royal Statistical Society, 4(2), 225–232. https://doi.org/10.2307/2983647
Poline,, J.‐B., & Mazoyer,, B. M. (1993). Analysis of individual positron emission tomography activation maps by detection of high signal‐to‐noise‐ratio pixel clusters. Journal of Cerebral Blood Flow and Metabolism, 13(3), 425–437. https://doi.org/10.1038/jcbfm.1993.57
Politis,, D. (2016). T and χ2: Revisiting the classical tests for the 21st century classroom. IMS Bulletin, 45(4), 10–11.
Rappelsberger,, P., & Petsche,, H. (1988). Probability mapping: Power and coherence analyses of cognitive processes. Brain Topography, 1(1), 46–54. https://doi.org/10.1007/BF01129339
Raz,, J., Zheng,, H., Ombao,, H., & Turetsky,, B. (2003). Statistical tests for fMRI based on experimental randomization. NeuroImage, 19(2 Pt 1), 226–232. https://doi.org/10.1016/S1053-8119(03)00115-0
Roland,, P. E., Levin,, B., Kawashima,, R., & Akerman,, S. (1993). Three‐dimensional analysis of clustered voxels in 15o‐butanol brain activation images. Human Brain Mapping, 1(1), 3–19. https://doi.org/10.1002/hbm.460010103
Romano,, J. P. (1990). On the behavior of randomization tests without a group invariance assumption. Journal of the American Statistical Association, 85(411), 686–692. https://doi.org/10.1080/01621459.1990.10474928
Romano,, J. P., & Wolf,, M. (2005). Exact and approximate stepdown methods for multiple hypothesis testing. Journal of the American Statistical Association, 100(469), 94–108. https://doi.org/10.1198/016214504000000539
Romano,, J. P., & Wolf,, M. (2007). Control of generalized error rates in multiple testing. The Annals of Statistics, 35(4), 1378–1408. https://doi.org/10.1214/009053606000001622
Romano,, J. P., & Wolf,, M. (2016). Efficient computation of adjusted p‐values for resamplingbased stepdown multiple testing. Statistics %26 Probability Letters, 113, 38–40. https://doi.org/10.1016/j.spl.2016.02.012
Rorden,, C., Bonilha,, L., & Nichols,, T. E. (2007). Rank‐order versus mean based statistics for neuroimaging. NeuroImage, 35(4), 1531–1537. https://doi.org/10.1016/j.neuroimage.2006.12.043
Rorden,, C., Fridriksson,, J., & Karnath,, H.‐O. (2009). An evaluation of traditional and novel tools for lesion behavior mapping. NeuroImage, 44(4), 1355–1362. https://doi.org/10.1016/j.neuroimage.2008.09.031
Rorden,, C., Karnath,, H.‐O., & Bonilha,, L. (2007). Improving lesion‐symptom mapping. Journal of Cognitive Neuroscience, 19(7), 1081–1088. https://doi.org/10.1162/jocn.2007.19.7.1081
Salimi‐Khorshidi,, G., Smith,, S. M., & Nichols,, T. E. (2011). Adjusting the effect of nonstationarity in cluster‐based and TFCE inference. NeuroImage, 54(3), 2006–2019. https://doi.org/10.1016/j.neuroimage.2010.09.088
Scheffé,, H. (1943). Statistical inference in the non‐parametric case. The Annals of Mathematical Statistics, 14(4), 305–332. https://doi.org/10.1214/aoms/1177731355
Silver,, M., Montana,, G., Nichols,, T. E., & Alzheimer`s Disease Neuroimaging Initiative. (2011). False positives in neuroimaging genetics using voxel‐based morphometry data. NeuroImage, 54(2), 992–1000. https://doi.org/10.1016/j.neuroimage.2010.08.049
Smith,, S. M., & Nichols,, T. E. (2009). Threshold‐free cluster enhancement: Addressing problems of smoothing, threshold dependence and localisation in cluster inference. NeuroImage, 44(1), 83–98. https://doi.org/10.1016/j.neuroimage.2008.03.061
Student. (1908). The probable error of a mean. Biometrika, 6(1), 1–25. https://doi.org/10.2307/2331554
Thau,, K., Rappelsberger,, P., Lovrek,, A., Petsche,, H., Simhandl,, C., & Topitz,, A. (1988). Effect of lithium on the EEG of healthy males and females. Neuropsychobiology, 20(3), 158–163. https://doi.org/10.1159/000118491
Welch,, B. L. (1938). The significance of the difference between two means when the population variances are unequal. Biometrika, 39(3/4), 350–362. https://doi.org/10.2307/2332010
Welch,, B. L. (1947). The generalization of “Student`s” problem when several different population variances are involved. Biometrika, 34(1–2), 28–35. https://doi.org/10.1093/biomet/34.1-2.28
Westfall,, P. H., & Young,, S. S. (1993). Resampling‐based multiple testing. New York, NY: Wiley.
White,, H. (1980). A heteroscedasticity‐consistent covariance matrix and a direct test for heteroscedasticity. Econometrica, 48(4), 817–838. https://doi.org/10.2307/1912934
Wilcoxon,, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80–83. https://doi.org/10.2307/3001968
Winkler,, A. M., Ridgway,, G. R., Douaud,, G., Nichols,, T. E., & Smith,, S. M. (2016). Faster permutation inference in brain imaging. NeuroImage, 141, 502–516. https://doi.org/10.1016/j.neuroimage.2016.05.068
Winkler,, A. M., Ridgway,, G. R., Webster,, M. A., Smith,, S. M., & Nichols,, T. E. (2014). Permutation inference for the general linear model. NeuroImage, 92, 381–397. https://doi.org/10.1016/j.neuroimage.2014.01.060
Winkler,, A. M., Webster,, M. A., Brooks,, J. C., Tracey,, I., Smith,, S. M., & Nichols,, T. E. (2016). Non‐parametric combination and related permutation tests for neuroimaging. Human Brain Mapping, 37(4), 1486–1511. https://doi.org/10.1002/hbm.23115
Winkler,, A. M., Webster,, M. A., Vidaurre,, D., Nichols,, T. E., & Smith,, S. M. (2015). Multi‐level block permutation. NeuroImage, 123, 253–268. https://doi.org/10.1016/j.neuroimage.2015.05.092
Xu,, Y., Sudre,, G. P., Wang,, W., Weber,, D. J., & Kassa,, R. E. (2011). Characterizing global statistical significance of spatiotemporal hot spots in magnetoencephalography/electroencephalography source space via excursion algorithms. Statistics in Medicine, 30(23), 2854–2866. https://doi.org/10.1002/sim.4309
Yang,, X., Beason‐Held,, L., Resnick,, S. M., & Landman,, B. A. (2011). Biological parametric mapping with robust and non‐parametric statistics. NeuroImage, 57(2), 423–430. https://doi.org/10.1016/j.neuroimage.2011.04.046
Zhang,, H., Nichols,, T. E., & Johnson,, T. D. (2009). Cluster mass inference via random field theory. NeuroImage, 44(1), 51–61. https://doi.org/10.1016/j.neuroimage.2008.08.017

Access to this WIREs title is by subscription only.

Recommend to Your
Librarian Now!

The latest WIREs articles in your inbox

Sign Up for Article Alerts