How to cite this WIREs title:
WIREs Comp Stat

Aggregating predictions from experts: A review of statistical methods, experiments, and applications

Advanced Review

Nicholas G. Reich, Graham C. Gibson, Nutcha Wattanachit, Thomas McAndrew

Published Online: Jun 16 2020

DOI: 10.1002/wics.1514

Full article on Wiley Online Library: HTML PDF

Can't access this content? Tell your librarian.

Abstract Forecasts support decision making in a variety of applications. Statistical models can produce accurate forecasts given abundant training data, but when data is sparse or rapidly changing, statistical models may not be able to make accurate predictions. Expert judgmental forecasts—models that combine expert‐generated predictions into a single forecast—can make predictions when training data is limited by relying on human intuition. Researchers have proposed a wide array of algorithms to combine expert predictions into a single forecast, but there is no consensus on an optimal aggregation model. This review surveyed recent literature on aggregating expert‐elicited predictions. We gathered common terminology, aggregation methods, and forecasting performance metrics, and offer guidance to strengthen future work that is growing at an accelerated pace. This article is categorized under: Statistical Learning and Exploratory Methods of the Data Sciences > Clustering and Classification Statistical Learning and Exploratory Methods of the Data Sciences > Exploratory Data Analysis Statistical Learning and Exploratory Methods of the Data Sciences > Modeling Methods Statistical and Graphical Methods of Data Analysis > Multivariate Analysis

A consort diagram that describes the path from collected to analysis‐set article. The search term used to collect the initial set of articles is reported and all intermediate steps between initial and analysis‐set articles

[ Normal View | Magnified View ]

Complementary cumulative distribution of the total number of forecasts made per article (a), and the proportion of articles eliciting less than 10, 100, 10³, 10⁴, and 10⁵ forecasts. Articles collecting more than 10⁴ forecasts were simulations

[ Normal View | Magnified View ]

The complementary cumulative distribution (CCDF) of the number of experts elicited per article (a). The proportion of articles enrolling less than 10, less than 100, less than 10³, and less than 10⁴ expert forecasters (b). The small number of articles enrolling more than 10³ were simulation studies

[ Normal View | Magnified View ]

The annual proportion of the top 12 most prevalent word stems among all abstract text. Note: The words probability and probabilities were stemmed to probabl). For each year, word w frequency was divided by the frequency of all words present in all abstracts

[ Normal View | Magnified View ]

The top 5% most frequent words used in all analysis‐set abstracts. Expert, forecast, and judgment are the most frequent and likely related to the search words used to collect these articles

[ Normal View | Magnified View ]

The cumulative proportion (a) and individual number (b) of articles published per year. The earliest analysis‐set article was published in 1992 and most recent in 2018. A sharp increase in publication occurred at or near 2010

[ Normal View | Magnified View ]

References

Abernethy,, J. D., & Frongillo,, R. M. (2011). A collaborative mechanism for crowdsourcing prediction problems. In J. Shawe‐Taylor,, R. S. Zemel,, P. L. Bartlett,, F. Pereira,, & K. Q. Weinberger, (Eds.), Advances in neural information processing systems 24 (pp. 2600–2608). Red Hook, NY: Curran Associates Inc..
Abramson,, B., Brown,, J., Edwards,, W., Murphy,, A., & Winkler,, R. L. (1996). Hailfinder: A Bayesian system for forecasting severe weather. International Journal of Forecasting, 12(1), 57–71. https://doi.org/10.1016/0169-2070(95)00664-8
Adams,, R., White,, A., & Ceylan,, E. (2009). An Acceptability Predictor for Websites. In International Conference on Universal Access in Human‐Computer Interaction, pp. 628–634. Springer.
Alexander,, J. C., Jr. (1995). Refining the degree of earnings surprise: A comparison of statistical and analysts’ forecasts. Financial Review, 30(3), 469–506. https://doi.org/10.1111/j.1540-6288.1995.tb00842.x
Alho,, J. M. (1992). Estimating the strength of expert judgement: The case of US mortality forecasts. Journal of Forecasting, 11(2), 157–167. https://doi.org/10.1002/for.3980110206
Al‐Jarrah,, O. Y., Yoo,, P. D., Muhaidat,, S., Karagiannidis,, G. K., & Taha,, K. (2015). Efficient machine learning for big data: A review. Big Data Research, 2(3), 87–93. https://doi.org/10.1016/j.bdr.2015.04.001
Alvarado‐Valencia,, J., Barrero,, L. H., Önkal,, D., & Dennerlein,, J. T. (2017). Expertise, credibility of system forecasts and integration methods in judgmental demand forecasting. International Journal of Forecasting, 33(1), 298–313. https://doi.org/10.1016/j.ijforecast.2015.12.010
Amara,, R. C., & Lipinski,, A. J. (1971). Some views on the use of expert judgment. Technological Forecasting and Social Change, 3, 279–289. https://doi.org/10.1016/S0040-1625(71)80019-7
Armstrong,, J. S. (1983). Relative accuracy of Judgemental and extrapolative methods in forecasting annual earnings. Journal of Forecasting, 2(4), 437–447. https://doi.org/10.1002/for.3980020411
Armstrong,, J. S. (1985). Long‐range forecasting: From crystal ball to computer. New York, NY: Wiley.
Armstrong,, J. S. (2001a). Combining forecasts. In Principles of forecasting. International series in operations research and management science (pp. 417–439). New York, NY: Springer.
Armstrong,, J. S. (2001b). Judgmental bootstrapping: Inferring experts’ rules for forecasting. In Principles of forecasting. International series in operations research and management science (pp. 171–192). New York, NY: Springer.
Ayyub,, B. M. (2001). Elicitation of expert opinions for uncertainty and risks. Boca Raton, FL: CRC Press.
Baecke,, P., De Baets,, S., & Vanderheyden,, K. (2017). Investigating the added value of integrating human judgement into statistical demand forecasting systems. International Journal of Production Economics, 191, 85–96. https://doi.org/10.1016/j.ijpe.2017.05.016
Baldwin,, P. (2015). Weighting components of a composite score using Naïve expert judgments about their relative importance. Applied Psychological Measurement, 39(7), 539–550. https://doi.org/10.1177/0146621615584703
Baran,, S., & Lerch,, S. (2018). Combining predictive distributions for the statistical post‐processing of ensemble forecasts. International Journal of Forecasting, 34(3), 477–496. https://doi.org/10.1016/j.ijforecast.2018.01.005
Baron,, J., Mellers,, B. A., Tetlock,, P. E., Stone,, E., & Ungar,, L. H. (2014). Two reasons to make aggregated probability forecasts more extreme. Decision Analysis, 11(2), 133–145. https://doi.org/10.1287/deca.2014.0293
Bassetti,, F., Casarin,, R., & Ravazzolo,, F. (2018). Bayesian nonparametric calibration and combination of predictive distributions. Journal of the American Statistical Association, 113(522), 675–685. https://doi.org/10.1080/01621459.2016.1273117
Bates,, J. M., & Granger,, C. W. (1969). The combination of forecasts. Journal of the Operational Research Society, 20(4), 451–468. https://doi.org/10.1057/jors.1969.103
Berrocal,, V. J., Raftery,, A. E., & Gneiting,, T. (2007). Combining spatial statistical and ensemble information in probabilistic weather forecasts. Monthly Weather Review, 135(4), 1386–1402. https://doi.org/10.1175/MWR3341.1
Bolger,, D., & Houlding,, B. (2017). Deriving the probability of a linear opinion pooling method being superior to a set of alternatives. Reliability Engineering %26 System Safety, 158, 41–49.
Borsuk,, M. E. (2004). Predictive assessment of fish health and fish kills in the Neuse River estuary using elicited expert judgment. Human and Ecological Risk Assessment, 10(2), 415–434. https://doi.org/10.1080/10807030490438454
Brabham,, D. C. (2013). Crowdsourcing. Cambridge, MA: MIT Press.
Brito,, M., & Griffiths,, G. (2016). A Bayesian approach for predicting risk of autonomous underwater vehicle loss during their missions. Reliability Engineering %26 System Safety, 146, 55–67. https://doi.org/10.1016/j.ress.2015.10.004
Brito,, M., Griffiths,, G., Ferguson,, J., Hopkin,, D., Mills,, R., Pederson,, R., & MacNeil,, E. (2012). A behavioral probabilistic risk assessment framework for managing autonomous underwater vehicle deployments. Journal of Atmospheric and Oceanic Technology, 29(11), 1689–1703. https://doi.org/10.1175/JTECH-D-12-00005.1
Bunn,, D., & Wright,, G. (1991). Interaction of judgemental and statistical forecasting methods: Issues %26 analysis. Management Science, 37(5), 501–518. https://doi.org/10.1287/mnsc.37.5.501
Bunn,, D. W. (1979). The synthesis of predictive models in marketing research. Journal of Marketing Research, 16(2), 280–283.
Bunn,, D. W. (1985). Statistical efficiency in the linear combination of forecasts. International Journal of Forecasting, 1(2), 151–163. https://doi.org/10.1016/0169-2070(85)90020-2
Cabello,, E., Conde,, C., Diego,, I., Moguerza,, J., & Redchuk,, A. (2012). Combination and selection of traffic safety expert judgments for the prevention of driving risks. Sensors, 12(11), 14711–14729. https://doi.org/10.3390/s121114711
Cai,, M., Lin,, Y., Han,, B., Liu,, C., & Zhang,, W. (2016). On a simple and efficient approach to probability distribution function aggregation. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47(9), 2444–2453. https://doi.org/10.1109/TSMC.2016.2531647
Capistrán,, C., & Timmermann,, A. (2009). Forecast combination with entry and exit of experts. Journal of Business %26 Economic Statistics, 27(4), 428–440. https://doi.org/10.1198/jbes.2009.07211
Che,, D., Liu,, Q., Rasheed,, K., & Tao,, X. (2011). Decision tree and ensemble learning algorithms with their applications in bioinformatics. In Software Tools and Algorithms for Biological Systems (pp. 191–199). New York, NY: Springer.
Clemen,, R. T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5(4), 559–583. https://doi.org/10.1016/0169-2070(89)90012-5
Clemen,, R. T. (2008). Comment on Cooke`s classical method. Reliability Engineering %26 System Safety, 93(5), 760–765. https://doi.org/10.1016/j.ress.2008.02.003
Clemen,, R. T., & Winkler,, R. L. (1986). Combining economic forecasts. Journal of Business %26 Economic Statistics, 4(1), 39–46. https://doi.org/10.2307/1391385
Clemen,, R. T., & Winkler,, R. L. (1999). Combining probability distributions from experts in risk analysis. Risk Analysis, 19(2), 187–203. https://doi.org/10.1111/j.1539-6924.1999.tb00399.x
Clemen,, R. T., & Winkler,, R. l. (2007). Aggregating probability distributions. In W. Edwards,, R. Miles,, & D. von Winterfeldt, (Eds.), Advances in Decision Analysis: From Foundations to Applications. Cambridge, UK: Cambridge University Press.
Cooke,, R., Mendel,, M., & Thijs,, W. (1988). Calibration and information in expert resolution; a classical approach. Automatica, 24(1), 87–93. https://doi.org/10.1016/0005-1098(88)90011-8
Cooke,, R. (1991). Experts in uncertainty: Opinion and subjective probability in science. New York: Oxford University Press.
Cooke,, R. M. (2014). Validating expert judgment with the classical model. In C. Martini, & M. Boumans, (Eds.), Experts and Consensus in Social Science (pp. 191–212). Cham: Springer.
Cooke,, R. M. (2015). The aggregation of expert judgment: Do good things come to those who weight? Risk Analysis, 35(1), 12–15. https://doi.org/10.1111/risa.12353
Cooke,, R. M., Wittmann,, M. E., Lodge,, D. M., Rothlisberger,, J. D., Rutherford,, E. S., Zhang,, H., & Mason,, D. M. (2014). Out‐of‐sample validation for structured expert judgment of Asian carp establishment in Lake Erie. Integrated Environmental Assessment and Management, 10(4), 522–528. https://doi.org/10.1002/ieam.1559
Craig,, P. S., Goldstein,, M., Rougier,, J. C., & Seheult,, A. H. (2001). Bayesian forecasting for complex systems using computer simulators. Journal of the American Statistical Association, 96(454), 717–729. https://doi.org/10.1198/016214501753168370
Cranor,, L. F. (2008). A framework for reasoning about the human in the loop. In Proceedings of the 1st Conference on Usability, Psychology, and Security, UPSEC`08. USENIX Association.
C Forlines,, S Miller,, L Guelcher,, R Bruzzi,. (2014). Crowdsourcing the Future: Predictions Made with a Social Network. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI`14, pp. 3655–3664, New York, NY: Association for Computing Machinery.
Dawid,, A., DeGroot,, M., Mortera,, J., Cooke,, R., French,, S., Genest,, C., … Winkler,, R. (1995). Coherent combination of experts’ opinions. Test, 4(2), 263–313. https://doi.org/10.1007/BF02562628
Dawid,, A. P. (2007). The geometry of proper scoring rules. Annals of the Institute of Statistical Mathematics, 59(1), 77–93. https://doi.org/10.1007/s10463-006-0099-8
de Groot,, A. D. (2014). Thought and Choice in Chess (Vol. 4). Berlin: Walter de Gruyter GmbH %26 Co KG.
DeGroot,, M. H. (1988). A Bayesian view of assessing uncertainty and comparing expert opinion. Journal of Statistical Planning and Inference, 20(3), 295–306. https://doi.org/10.1016/0378-3758(88)90094-8
Dietterich,, T. G. (2002). Ensemble learning. The Handbook of Brain Theory and Neural Networks, 2, 110–125.
Džeroski,, S., & Ženko,, B. (2004). Is combining classifiers with stacking better than selecting the best one? Machine Learning, 54(3), 255–273. https://doi.org/10.1023/B:MACH.0000015881.36452.6
Edmundson,, R. (1990). Decomposition; a strategy for judgemental forecasting. Journal of Forecasting, 9(4), 305–314. https://doi.org/10.1002/for.3980090403
Ericsson,, K. A., & Ward,, P. (2007). Capturing the naturally occurring superior performance of experts in the laboratory: Toward a science of expert and exceptional performance. Current Directions in Psychological Science, 16(6), 346–350. https://doi.org/10.1111/j.1467-8721.2007.00533.x
Evans,, J., Gray,, G., Sielken,, R., Smith,, A., Valdezflores,, C., & Graham,, J. (1994). Use of probabilistic expert judgment in uncertainty analysis of carcinogenic potency. Regulatory Toxicology and Pharmacology, 20(1), 15–36. https://doi.org/10.1006/rtph.1994.1034
Failing,, L., Horn,, G., & Higgins,, P. (2004). Using expert judgment and stakeholder values to evaluate adaptive management options. Ecology and Society, 9(1), 13.
Farrington‐Darby,, T., & Wilson,, J. R. (2006). The nature of expertise: A review. Applied Ergonomics, 37(1), 17–32. https://doi.org/10.1016/j.apergo.2005.09.001
Fischer,, I., & Harvey,, N. (1999). Combining forecasts: What information do judges need to outperform the simple average? International Journal of Forecasting, 15(3), 227–246. https://doi.org/10.1016/S0169-2070(98)00073-9
Franses,, P. H. (2011). Averaging model forecasts and expert forecasts: Why does it work? Interfaces, 41(2), 177–181. https://doi.org/10.1287/inte.1100.0554
French,, S. (2011). Aggregating expert judgement. Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. Serie A. Matematicas, 105(1), 181–206. https://doi.org/10.1007/s13398-011-0018-6
Galton,, F. (1907). Vox populi. Nature, 75(7), 450–451. https://doi.org/10.1038/075450a0
Garratt,, A., Henckel,, T., & Vahey,, S. P. (2019). Empirically‐transformed linear opinion pools. CAMA Working Papers 2019‐47. Centre for Applied Macroeconomic Analysis, Crawford School of Public Policy.
Genest,, C., & Zidek,, J. V. (1986). Combining probability distributions: A critique and an annotated bibliography. Statistical Science, 1(1), 114–135. https://doi.org/10.1214/ss/1177013825
Genre,, V., Kenny,, G., Meyler,, A., & Timmermann,, A. (2013). Combining expert forecasts: Can anything beat the simple average? International Journal of Forecasting, 29(1), 108–121. https://doi.org/10.1016/j.ijforecast.2012.06.004
Glahn,, B., Peroutka,, M., Wiedenfeld,, J., Wagner,, J., Zylstra,, G., Schuknecht,, B., & Jackson,, B. (2009). MOS uncertainty estimates in an ensemble framework. Monthly Weather Review, 137(1), 246–268. https://doi.org/10.1175/2008MWR2569.1
Gneiting,, T., & Raftery,, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. https://doi.org/10.1198/016214506000001437
Gneiting,, T., & Ranjan,, R. (2011). Comparing density forecasts using threshold‐and Quantile‐weighted scoring rules. Journal of Business %26 Economic Statistics, 29(3), 411–422. https://doi.org/10.1198/jbes.2010.08110
Gneiting,, T., & Ranjan,, R. (2013). Combining predictive distributions. Electronic Journal of Statistics, 7, 1747–1782. https://doi.org/10.1214/13-EJS823
Graefe,, A. (2015). Accuracy gains of adding vote expectation surveys to a combined forecast of US presidential election outcomes. Research %26 Politics, 2(1), 1–5. https://doi.org/10.1177/2053168015570416
Graefe,, A. (2018). Predicting elections: Experts, polls, and fundamentals. Judgment and Decision making, 13(4), 334.
Graefe,, A., Armstrong,, J. S., Jones,, R. J., Jr., & Cuzán,, A. G. (2014a). Combining forecasts: An application to elections. International Journal of Forecasting, 30(1), 43–54. https://doi.org/10.1016/j.ijforecast.2013.02.005
Graefe,, A., Armstrong,, J. S., Jones,, R. J., Jr., & Cuzán,, A. G. (2014b). Accuracy of combined forecasts for the 2012 presidential election: The PollyVote. PS: Political Science %26 Politics, 47(2), 427–431. https://doi.org/10.1017/S1049096514000341
Granger,, C. W., & Ramanathan,, R. (1984). Improved methods of combining forecasts. Journal of Forecasting, 3(2), 197–204. https://doi.org/10.1002/for.3980030207
Gu,, W., Saaty,, T. L., & Whitaker,, R. (2016). Expert system for ice hockey game prediction: Data mining with human judgment. International Journal of Information Technology and Decision Making, 15(04), 763–789. https://doi.org/10.1142/S0219622016400022
Guangliang,, S. (1996). A multi‐hierarchical comprehensive evaluation model and its application. Systems Engineering, 14(2), 64–67.
Hammond,, K. R., & Stewart,, T. R. (2001). The essential Brunswik: Beginnings, explications, applications. Oxford: Oxford University Press.
Hanea,, A. M., McBride,, M. F., Burgman,, M. A., & Wintle,, B. C. (2018). The value of performance weights and discussion in aggregated expert judgments. Risk Analysis, 38(9), 1781–1794. https://doi.org/10.1111/risa.12992
Hathout,, M., Vuillet,, M., Peyras,, L., Carvajal,, C., & Diab,, Y. (2016). Uncertainty and expert assessment for supporting evaluation of levees safety. In 3rd European Conference on Flood Risk Management (FLOODrisk 2016), p. 6.
Helmer,, O. (1967, March). Analysis of the Future: The Delphi Method [Technical report]. Santa Monica, CA: The RAND Corporation.
Hogarth,, R. M. (1975). Cognitive processes and the assessment of subjective probability distributions. Journal of the American Statistical Association, 70(350), 271–289. https://doi.org/10.2307/2285808
Holzinger,, A. (2016). Interactive machine learning for health informatics: When do we need the human‐in‐the‐loop? Brain Informatics, 3(2), 119–131. https://doi.org/10.1007/s40708-016-0042-6
Hora,, S. C., Fransen,, B. R., Hawkins,, N., & Susel,, I. (2013). Median aggregation of distribution functions. Decision Analysis, 10(4), 279–291. https://doi.org/10.1287/deca.2013.0282
Hora,, S. C., & Kardeş,, E. (2015). Calibration, sharpness and the weighting of experts in a linear opinion pool. Annals of Operations Research, 229(1), 429–450. https://doi.org/10.1007/s10479-015-1846-0
Howe,, J. (2006). The rise of crowdsourcing. Wired Magazine, 14(6), 1–4.
Huang,, A., Qiao,, H., Wang,, S., & Liu,, J. (2016). Improving forecasting performance by exploiting expert knowledge: Evidence from Guangzhou port. International Journal of Information Technology and Decision Making, 15(02), 387–401. https://doi.org/10.1142/S0219622016500085
Hurley,, W., & Lior,, D. (2002). Combining expert judgment: On the performance of trimmed mean vote aggregation procedures in the presence of strategic voting. European Journal of Operational Research, 140(1), 142–147. https://doi.org/10.1016/S0377-2217(01)00226-0
Jana,, D. K., Pramanik,, S., Sahoo,, P., & Mukherjee,, A. (2019). Interval type‐2 fuzzy logic and its application to occupational safety risk performance in industries. Soft Computing, 23(2), 557–567. https://doi.org/10.1007/s00500-017-2860-8
Jin,, W., Lu,, Q., & Gan,, W. (2007). Research progress on the durability design and life prediction of concrete structures. Journal of Building Structures, 28(1), 7–13.
Johnson,, F. A., Alhainen,, M., Fox,, A. D., Madsen,, J., & Guillemain,, M. (2018). Making do with less: Must sparse data preclude informed harvest strategies for European waterbirds? Ecological Applications, 28(2), 427–441. https://doi.org/10.1002/eap.1659
Jolliffe,, I. T., & Stephenson,, D. B. (2012). Forecast verification: A Practitioner`s guide in atmospheric science. Hoboken, NJ: John Wiley %26 Sons.
Kabak,, Ö., & Ülengin,, F. (2008). Aggregating forecasts to obtain fuzzy demands. In Computational intelligence in decision and control (pp. 73–78). Singapore: World Scientific.
Keeney,, R. L. (1976). A group preference Axiomatization with cardinal utility. Management Science, 23(2), 140–145.
Kläs,, M., Nakao,, H., Elberzhager,, F., & Münch,, J. (2010). Support planning and controlling of early quality assurance by combining expert judgment and defect data a case study. Empirical Software Engineering, 15(4), 423–454. https://doi.org/10.1007/s10664-009-9112-1
Kleiber,, W., Raftery,, A. E., Baars,, J., Gneiting,, T., Mass,, C. F., & Grimit,, E. (2011). Locally calibrated probabilistic temperature forecasting using Geostatistical model averaging and local Bayesian model averaging. Monthly Weather Review, 139(8), 2630–2649. https://doi.org/10.1175/2010MWR3511.1
Kleinmuntz,, B. (1990). Why we still use our heads instead of formulas: Toward an integrative approach. Psychological Bulletin, 107(3), 296–310. https://doi.org/10.1037/0033-2909.107.3.296
Kune,, R., Konugurthi,, P. K., Agarwal,, A., Chillarige,, R. R., & Buyya,, R. (2016). The anatomy of big data computing. Software: Practice and Experience, 46(1), 79–105. https://doi.org/10.1002/spe.2374
Kurowicka,, D., Bucura,, C., Cooke,, R., & Havelaar,, A. (2010). Probabilistic inversion in priority setting of emerging Zoonoses. Risk Analysis, 30(5), 715–723. https://doi.org/10.1111/j.1539-6924.2010.01378.x
Lawrence,, M., Goodwin,, P., O`Connor,, M., & Önkal,, D. (2006). Judgmental forecasting: A review of Progress over the last 25 years. International Journal of Forecasting, 22(3), 493–518. https://doi.org/10.1016/j.ijforecast.2006.03.007
Lawrence,, M., & O`Connor,, M. (1992). Exploring judgemental forecasting. International Journal of Forecasting, 8(1), 15–26. https://doi.org/10.1016/0169-2070(92)90004-S
Leal,, J., Wordsworth,, S., Legood,, R., & Blair,, E. (2007). Eliciting expert opinion for economic models: An applied example. Value in Health, 10(3), 195–203. https://doi.org/10.1111/j.1524-4733.2007.00169.x
Li,, G. (2017). Human‐in‐the‐loop data integration. Proceedings of the VLDB Endowment, 10(12), 2006–2017. https://doi.org/10.14778/3137765.3137833
Li,, W., Liu,, Y.‐J., & Yang,, Z. (2012). Preliminary strategic environmental assessment of the great Western development strategy: Safeguarding ecological security for a new Western China. Environmental Management, 49(2), 483–501. https://doi.org/10.1007/s00267-011-9794-1
Lintott,, C. J., Schawinski,, K., Slosar,, A., Land,, K., Bamford,, S., Thomas,, D., … Vandenberg,, J. (2008). Galaxy zoo: Morphologies derived from visual inspection of galaxies from the Sloan digital sky survey. Monthly Notices of the Royal Astronomical Society, 389(3), 1179–1189. https://doi.org/10.1111/j.1365-2966.2008.13689.x
Loper,, E. & Bird,, S. (2002). NLTK: The Natural Language Toolkit.
Mak,, B., Bui,, T., & Blanning,, R. (1996). Aggregating and updating experts’ knowledge: An experimental evaluation of five classification techniques. Expert Systems with Applications, 10(2), 233–241. https://doi.org/10.1016/0957-4174(95)00049-6
Makridakis,, S., & Winkler,, R. L. (1983). Averages of forecasts: Some empirical results. Management Science, 29(9), 987–996.
Mantyka‐Pringle,, C. S., Martin,, T. G., Moffatt,, D. B., Linke,, S., & Rhodes,, J. R. (2014). Understanding and predicting the combined effects of climate change and land‐use change on freshwater macroinvertebrates and fish. Journal of Applied Ecology, 51(3), 572–581. https://doi.org/10.1111/1365-2664.12236
Martin,, T. G., Burgman,, M. A., Fidler,, F., Kuhnert,, P. M., Low‐Choy,, S., McBride,, M., & Mengersen,, K. (2012). Eliciting expert knowledge in conservation science. Conservation Biology, 26(1), 29–38. https://doi.org/10.1111/j.1523-1739.2011.01806.x
McLaughlin,, R. L. (1973). The forecasters’ batting averages. Business Economics, 3, 58–59.
Meehl,, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. Minneapolis, MN: University of Minnesota Press.
Mellers,, B., Ungar,, L., Baron,, J., Ramos,, J., Gurcay,, B., Fincher,, K., … Tetlock,, P. E. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25(5), 1106–1115. https://doi.org/10.1177/0956797614524255
Morales‐Nápoles,, O., Paprotny,, D., Worm,, D., Abspoel‐Bukman,, L., & Courage,, W. (2017). Characterization of precipitation through copulas and expert judgement for risk assessment of infrastructure. ASCE‐ASME Journal of Risk and Uncertainty in Engineering Systems, Part A: Civil Engineering, 3(4), 04017012. https://doi.org/10.1061/AJRUA6.0000914
Moran,, K. R., Fairchild,, G., Generous,, N., Hickmann,, K. S., Osthus,, D., Priedhorsky,, R., … Valle,, S. Y. D. (2016). Epidemic forecasting is messier than weather forecasting: The role of human behavior and internet data streams in epidemic forecast. The Journal of Infectious Diseases, 214, S404–S408. https://doi.org/10.1093/infdis/jiw375
Morgan,, M. G. (2014). Use (and abuse) of expert elicitation in support of decision making for public policy. Proceedings of the National Academy of Sciences of the United States of America, 111(20), 7176–7184. https://doi.org/10.1073/pnas.1319946111
Morris,, P. A. (1974). Decision analysis expert use. Management Science, 20(9), 1233–1241.
Mu,, L. & Xianming,, W. (1999). Multi‐hierarchical durability assessment of existing reinforced‐concrete structures. In Proceedings of the 8th International Conference on Durability of Building Materials and Components, pp. 49–69.
Murphy,, A. H., & Winkler,, R. L. (1974a). Probability forecasts: A survey of National Weather Service forecasters. Bulletin of the American Meteorological Society, 55(12), 1449–1453.
Murphy,, A. H., & Winkler,, R. L. (1974b). Subjective probability forecasting experiments in meteorology: Some preliminary results. Bulletin of the American Meteorological Society, 55(10), 1206–1216. https://doi.org/10.1175/1520-0477(1974)055%3C1206:SPFEIM%3E2.0.CO;2
Murphy,, A. H., & Winkler,, R. L. (1984). Probability forecasting in meteorology. Journal of the American Statistical Association, 79(387), 489–500. https://doi.org/10.2307/2288395
Neves,, L. & Frangopol,, D. (2008). Life‐cycle performance of structures: Combining expert judgment and results of inspection. In Proceedings of the 1st International Symposium on Life‐Cycle Civil Engineering, pp. 409–414.
Normand,, S.‐L. T., McNeil,, B. J., Peterson,, L. E., & Palmer,, R. H. (1998). Eliciting expert opinion using the Delphi technique: Identifying performance indicators for cardiovascular disease. International Journal for Quality in Health Care, 10(3), 247–260. https://doi.org/10.1093/intqhc/10.3.247
O`Connor,, M., Remus,, W., & Griggs,, K. (1993). Judgemental forecasting in times of change. International Journal of Forecasting, 9(2), 163–172. https://doi.org/10.1016/0169-2070(93)90002-5
O`Hagan,, A., Buck,, C. E., Daneshkhah,, A., Eiser,, J. R., Garthwaite,, P. H., Jenkinson,, D. J., … Rakow,, T. (2006). Uncertain judgements: Eliciting experts’ probabilities. Hoboken, NJ: John Wiley %26 Sons.
Petrovic,, D., Xie,, Y., & Burnham,, K. (2006). Fuzzy decision support system for demand forecasting with a learning mechanism. Fuzzy Sets and Systems, 157(12), 1713–1725. https://doi.org/10.1016/j.fss.2006.03.011
Polley,, E. C. & van der Laan,, M. J. (2010). Super learner in prediction. U.C. Berkeley Division of Biostatistics Working Paper Series. Working Paper 266. Retrieved from https://biostats.bepress.com/ucbbiostat/paper266.
Powell,, C. (2003). The Delphi technique: Myths and realities. Journal of Advanced Nursing, 41(4), 376–382. https://doi.org/10.1046/j.1365-2648.2003.02537.x
Prill,, R. J., Saez‐Rodriguez,, J., Alexopoulos,, L. G., Sorger,, P. K., & Stolovitzky,, G. (2011). Crowdsourcing network inference: The DREAM predictive signaling network challenge. Science Signaling, 4, mr7. https://doi.org/10.1126/scisignal.2002212
Ranjan,, R., & Gneiting,, T. (2010). Combining probability forecasts. Journal of the Royal Statistical Society, Series B: Statistical Methodology, 72(1), 71–91. https://doi.org/10.1111/j.1467-9868.2009.00726.x
Rantilla,, A. K. & Budescu,, D. V. (1999). Aggregation of expert opinions. In Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. p. 11.
Ren‐jun,, Z. & Xian‐zhong,, D. (2002). Optimal combined load forecast based on the improved analytic hierarchy process. In Proceedings of the IEEE International Conference on Power System Technology, Vol. 2, pp. 1096–1100.
Rikers,, R. M., & Paas,, F. (2005). Recent advances in expertise research. Applied Cognitive Psychology, 19(2), 145–149. https://doi.org/10.1002/acp.1108
Sakkis,, G., Androutsopoulos,, I., Paliouras,, G., Karkaletsis,, V., Spyropoulos,, C. D., & Stamatopoulos,, P. (2001). Stacking classifiers for anti‐spam filtering of E‐mail. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing.
Sarin,, R. K. (2013). Median aggregation, scoring rules, expert forecasts, choices with binary attributes, portfolio with dependent projects, and information security. Decision Analysis, 10(4), 277–278. https://doi.org/10.1287/deca.2013.0284
Satopää,, V. A., Jensen,, S. T., Mellers,, B. A., Tetlock,, P. E., & Ungar,, L. H. (2014). Probability aggregation in time‐series: Dynamic hierarchical modeling of sparse expert beliefs. The Annals of Applied Statistics, 8(2), 1256–1280 https://doi.org/10.1214/14-AOAS739
Seifert,, M., & Hadida,, A. L. (2013). On the relative importance of linear model and human judge(s) in combined forecasting. Organizational Behavior and Human Decision Processes, 120(1), 24–36. https://doi.org/10.1016/j.obhdp.2012.08.003
Shin,, J., Coh,, B.‐Y., & Lee,, C. (2013). Robust future‐oriented technology portfolios: Black–Litterman approach. R%26D Management, 43(5), 409–419. https://doi.org/10.1111/radm.12022
Song,, H., Gao,, B. Z., & Lin,, V. S. (2013). Combining statistical and judgmental forecasts via a web‐based tourism demand forecasting system. International Journal of Forecasting, 29(2), 295–310. https://doi.org/10.1016/j.ijforecast.2011.12.003
Spence,, M. T., & Brucks,, M. (1997). The moderating effects of problem characteristics on experts’ and novices’ judgments. Journal of Marketing Research, 34(2), 233–247. https://doi.org/10.2307/3151861
Stone,, M. (1961). The opinion Pool. The Annals of Mathematical Statistics, 32(4), 1339–1342.
Syarif,, I., Zaluska,, E., Prugel‐Bennett,, A., & Wills,, G. (2012). Application of bagging, boosting and stacking to intrusion detection. In International Workshop on Machine Learning and Data Mining in Pattern Recognition. pp. 593–602. Springer.
Tartakovsky,, D. M. (2007). Probabilistic risk analysis in subsurface hydrology. Geophysical Research Letters, 34(5), 114–127. https://doi.org/10.1029/2007GL029245
Ting,, K. M., & Witten,, I. H. (1999). Issues in stacked generalization. Journal of Artificial Intelligence Research, 10, 271–289. https://doi.org/10.1613/jair.594
Ungar,, L. H., Mellers,, B. A., Satopaa,, V., Tetlock,, P., & Baron,, J. (2012). The good judgment project: A large scale Test of different methods of combining expert predictions. In AAAI Fall Symposium on Machine Aggregation of Human Judgment.
van der Laan,, M. J., Polley,, E. C., & Hubbard,, A. E. (2007). Super learner. Statistical Applications in Genetics and Molecular Biology, 6(1). https://doi.org/10.2202/1544-6115.1309
Wallis,, K. F. (2011). Combining forecasts—forty years later. Applied Financial Economics, 21(1–2), 33–41. https://doi.org/10.1080/09603107.2011.523179
Wallsten,, T. S., Budescu,, D. V., Erev,, I., & Diederich,, A. (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making, 10(3), 243–268. https://doi.org/10.1002/(SICI)1099-0771(199709)10:3%3C243::AID-BDM268%3E3.0.CO;2-M
Wallsten,, T. S., Budescu,, D. V., & Tsao,, C. J. (1997). Combining linguistic probabilities. Psychologische Beitrage, 39(1–2), 27–55.
Wang,, C., Chen,, M.‐H., Schifano,, E., Wu,, J., & Yan,, J. (2016). Statistical methods and computing for big data. Statistics and its Interface, 9(4), 399–414. https://doi.org/10.4310/SII.2016.v9.n4.a1
Wang,, G., Hao,, J., Ma,, J., & Jiang,, H. (2011). A comparative assessment of ensemble learning for credit scoring. Expert Systems with Applications, 38(1), 223–230. https://doi.org/10.1016/j.eswa.2010.06.048
Wang,, L., & Zhang,, X. (2018). Bayesian analytics for estimating risk probability in PPP waste‐to‐energy projects. Journal of Management in Engineering, 34(6), 04018047.
Wang,, X., Du,, C., & Cao,, Z. (2008). Probabilistic inversion techniques in quantitative risk assessment for power system load forecasting. In 2008 IEEE International Conference on Information and Automation, pp. 718–723.
Willett,, P. (2006). The porter stemming algorithm: Then and now. Program: Electronic Library and Information Systems, 40(3), 219–223. https://doi.org/10.1108/00330330610681295
Winkler,, R. L. (1968). The consensus of subjective probability distributions. Management Science, 15(2), B–61.
Winkler,, R. L. (1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66(336), 675–685. https://doi.org/10.1080/01621459.1971.10482329
Winkler,, R. L. (1981). Combining probability distributions from dependent information sources. Management Science, 27(4), 479–488. https://doi.org/10.1287/mnsc.27.4.479
Winkler,, R. L., Grushka‐Cockayne,, Y., Lichtendahl,, K. C., Jr., & Jose,, V. R. R. (2019). Probability forecasts and their combination: A research perspective. Decision Analysis, 16(4), 239–260 https://doi.org/10.1287/deca.2019.0391
Wolpert,, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241–259. https://doi.org/10.1016/S0893-6080(05)80023-1
Yaniv,, I., & Hogarth,, R. M. (1993). Judgmental versus statistical prediction: Information asymmetry and combination rules. Psychological Science, 4(1), 58–62. https://doi.org/10.1111/j.1467-9280.1993.tb00558.x
Yousuf,, M. I. (2007). Using experts’ opinions through Delphi technique. Practical Assessment, Research %26 Evaluation, 12(4), 1–8. https://doi.org/10.7275/rrph-t210
Yu,, F., Seff,, A., Zhang,, Y., Song,, S., Funkhouser,, T., & Xiao,, J. (2015). LSUN: Construction of a large‐scale image dataset using deep learning with humans in the loop. ArXiv, 2015, abs/1506.03365.
Zio,, E. (1996). On the use of the analytic hierarchy process in the aggregation of expert judgments. Reliability Engineering %26 System Safety, 53(2), 127–138. https://doi.org/10.1016/0951-8320(96)00060-9
Zio,, E., & Apostolakis,, G. (1997). Accounting for expert‐to‐expert variability: A potential source of bias in performance assessments of high‐level radioactive waste repositories. Annals of Nuclear Energy, 24(10), 751–762. https://doi.org/10.1016/S0306-4549(96)00052-7

Access to this WIREs title is by subscription only.

Recommend to Your
Librarian Now!

The latest WIREs articles in your inbox

Sign Up for Article Alerts