Skip to main content
Loading...

Using prediction polling to harness collective intelligence for disease forecasting

During infectious disease epidemics, public health workers require an early warning, situational awareness, and predictive information. Sentinel surveillance, laboratory reporting, and case identification are all traditional methods of infectious disease surveillance that provide crucial information for outbreak response, management, and decision-making.

During infectious disease epidemics, public health workers require an early warning, situational awareness, and predictive information. Sentinel surveillance, laboratory reporting, and case identification are all traditional methods of infectious disease surveillance that provide crucial information for outbreak response, management, and decision-making. However, real-time and predictive outbreak data is frequently lacking, making it difficult for practitioners to respond effectively before an outbreak reaches its height. Given these and other problems with standard disease surveillance, it could be worthwhile to look into complementary methodologies that could supplement illness reporting and provide more forward-looking or predictive outbreak data.
Prediction polls, which aggregate individual forecasts statistically using recency-based subsetting, differential weighting based on historical performance, and recalibration, are an alternative approach to crowdsourcing forecasts for infectious disease surveillance. Prediction polls are different from traditional "opinion polling" in that they are used to make predictions about future outcomes of interest. A custom online forecasting platform allows a broad group of experts to anticipate infectious illness outcomes to explore the efficacy of crowdsourced knowledge for disease surveillance. Types of disease outcomes, questions, and settings that would result in reliable projections were examined in this study. The final goal was to build a foundation of evidence for using crowdsourced forecasting to confidently give information to decision-makers that can enhance traditional surveillance and modeling efforts while also improving response to infectious disease outbreaks.


Public health experts, medical professionals, epidemiologists, risk assessment experts, microbiologists, individuals with on-the-ground knowledge of disease outbreak conditions, public health graduate students, and others interested in infectious disease outbreaks were among those recruited. Forecasting, on the other hand, was available to everyone interested. A total of 562 volunteer participants participated over 15 months in a long-term experiment in a crowd- forecasting infectious disease outbreaks, answering 61 questions with a total of 217 possible answers about 19 diseases.


Crowd forecasts aggregated using best-practice adaptive algorithms are well-calibrated, accurate, timely, and outperform all individual forecasters, which is consistent with the "wisdom of crowds" phenomena. Participants were asked to indicate a probability for each feasible answer for each question. Participants were provided a current (but sub-optimal) aggregate of the collective (crowd) forecast for comparison after making their initial forecast in a question, and they could alter their own at any time. Participants might also share their projections and rationales in the discussion thread if they so desire.


In a nutshell, crowd forecasts proved to be quite reliable and accurate across all 61 settled IFPs. The formulation of forecasting questions was the most difficult aspect of deploying this infectious illness prediction technology. Questions had to be carefully crafted to be straightforward and simple enough to just have a few alternative outcomes. Crowd projections were found to be extremely reliable, accurate, and timely. Crowd forecasts aggregated using best-practice algorithms surpassed all individual forecasters, the majority of whom had professional experience in public health, demonstrating the "wisdom of crowds" phenomenon. However, infectious disease prediction polling should be investigated further, with a particular focus on establishing the best participant make-up.

Read more:
Sell TK, Warmbrod KL, Watson C, Trotochaud M, Martin E, Ravi SJ, Balick M, Servan-Schreiber E. Using prediction polling to harness collective intelligence for disease forecasting. BMC Public Health. 2021 Nov 20;21(1):2132. doi: 10.1186/s12889-021-12083-y. PMID: 34801014; PMCID: PMC8605461.
 

 

Leave a Comment