superforecaster

{{Short description|Forecasters whose results are more accurate than average}}

A superforecaster is a person who makes forecasts that can be shown by statistical means to have been consistently more accurate than the general public or experts. Superforecasters sometimes use modern analytical and statistical methodologies to augment estimates of base rates of events; research finds that such forecasters are typically more accurate than experts in the field who do not use analytical and statistical techniques,{{Sfnp|Adonis|2020}} though this has been overstated in some sources.{{cite web |title=Can Policymakers Trust Forecasters? |url=https://ifp.org/can-policymakers-trust-forecasters/#asking-the-experts |website=Institute for Progress |access-date=26 August 2024}} The term "superforecaster" is a trademark of Good Judgment Inc.{{Cite web |title=Trademark Electronic Search System (TESS) |url=https://tmsearch.uspto.gov/bin/gate.exe?f=doc&state=4806:vyqtyx.9.1 |access-date=2023-01-05 |website=tmsearch.uspto.gov}}

Etymology

The term is a combination of the prefix {{Linktext|super}}, meaning "over and above"{{cite web |title=Super Definition & Meaning |url=https://www.merriam-webster.com/dictionary/super |website=Merriam-Webster |access-date=1 November 2023 |archive-url=https://web.archive.org/web/20231101043310/https://www.merriam-webster.com/dictionary/super |archive-date=1 November 2023}} or "of high grade or quality", and {{Linktext|forecaster}}, meaning one who predicts an outcome that might occur in the future.

History

Origins of the term are attributed to Philip E. Tetlock with results from The Good Judgment Project and subsequent book with Dan Gardner Superforecasting: The Art and Science of Prediction.{{Sfnp|Tetlock|Gardner|2015}}

In December 2019 a Central Intelligence Agency analyst writing under the pseudonym "Bobby W." suggested the Intelligence community should study superforecaster research on how certain individuals with "particular traits" are better forecasters and how they should be leveraged.{{Sfnp|Bobby W.|2019|p=14}}

In February 2020 Dominic Cummings agreed with Tetlock and others in implying that study of superforecasting was more effective than listening to political pundits.{{Sfnp|BBC News|2020}}

Superforecasters

={{Anchor|A-CHAMP}}Science=

Superforecasters estimate a probability of an occurrence, and review the estimate when circumstances contributing to the estimate change. This is based on both personal impressions, public data, and incorporating input from other superforecasters, but attempts to remove bias in their estimates.{{Sfnp|BBC News|2020|loc=What is the science behind it?}} In The Good Judgment Project one set of forecasters were given training on how to translate their understandings into a probabilistic forecast, summarised into an acronym "CHAMP" for Comparisons, Historical trends, Average opinions, Mathematical models, and Predictable biases.{{Sfnp|Harford|2014|loc=How to be a superforecaster}}

A study published in 2021 used a Bias, Information, Noise (BIN) model to study the underlying processes enabling accuracy among superforecasters. The conclusion was that superforecasters' ability to filter out "noise" played a more significant role in improving accuracy than bias reduction or the efficient extraction of information.{{cite journal | url=https://pubsonline.informs.org/doi/abs/10.1287/mnsc.2020.3882 | doi=10.1287/mnsc.2020.3882 | title=Bias, Information, Noise: The BIN Model of Forecasting | date=2021 | last1=Satopää | first1=Ville A. | last2=Salikhov | first2=Marat | last3=Tetlock | first3=Philip E. | last4=Mellers | first4=Barbara | journal=Management Science | volume=67 | issue=12 | pages=7599–7618 | url-access=subscription }}

=Effectiveness=

In the Good Judgment Project, "the top forecasters... performed about 30 percent better than the average for intelligence community analysts who could read intercepts and other secret data".David Ignatius. [https://www.washingtonpost.com/opinions/david-ignatius-more-chatter-than-needed/2013/11/01/1194a984-425a-11e3-a624-41d661b0bb78_story.html "More chatter than needed"]. The Washington Post. 1 November 2013.Horowitz MC, Ciocca J, Kahn L, Ruhl C. "Keeping Score: A New Approach to Geopolitical Forecasting" [https://global.upenn.edu/sites/default/files/perry-world-house/Keeping%20Score%20Forecasting%20White%20Paper.pdf (PDF)]. Perry World House, University of Pennsylvania. 2021, p.9.

Training forecasters with specialised techniques may increase forecaster accuracy: in the Good Judgment Project, one group was given training in the "CHAMP" methodology, which appeared to increase forecasting accuracy.{{Sfnp|Harford|2014|loc=How to be a superforecaster}}

Superforecasters sometimes predict that events are less than 50% likely to happen, but they still happen: Bloomberg notes that they made a prediction of 23% for a leave vote in the month of the June 2016 Brexit referendum. On the other hand, the BBC notes that they accurately predicted Donald Trump's success in the 2016 Republican Party primaries.{{Sfnp|BBC News|2020|loc=How successful is it?}}

Superforecasters also made a number of accurate and important forecasts about the coronavirus pandemic, which "businesses, governments and other institutions" have drawn upon. In addition, they have made "accurate predictions about world events like the approval of the United Kingdom’s Brexit vote in 2020, Saudi Arabia’s decision to partially take its national gas company public in 2019, and the status of Russia’s food embargo against some European countries also in 2019".Tara Law. [https://time.com/5848271/superforecasters-covid-19/ "'Superforecasters' Are Making Eerily Accurate Predictions About COVID-19. Our Leaders Could Learn From Their Approach."] TIME. 11 June 2020.

Aid agencies are also using superforecasting to determine the probability of droughts becoming famines,{{Sfnp|Adonis|2020}} while the Center for a New American Security has described how superforecasters aided them in predicting future Colombian government policy.Cochran KM, Tozzi G. [https://www.cnas.org/publications/reports/getting-it-righter-faster "Getting it Righter, Faster: The Role of Prediction in Agile Government Decisionmaking"]. Center for a New American Security. 2017. Goldman Sachs drew upon superforecasters' vaccine forecasts during the coronavirus pandemic to inform their analyses.Hatzius J, Struyven D, Bhushan S, Milo D. [https://www.goldmansachs.com/insights/pages/gs-research/macro-outlook-2021/report.pdf "V(accine)-Shaped Recovery"]. Goldman Sachs Economics Research. 7 November 2020.

The Economist notes that in October 2021, Superforecasters accurately predicted events that occurred in 2022, including "election results in France and Brazil; the lack of a Winter Olympics boycott; the outcome of America's midterm elections, and that global Covid-19 vaccinations would reach 12bn doses in mid-2022". However, they did not forecast the emergence of the Omicron variant.[https://www.economist.com/the-world-ahead/2022/11/18/what-the-superforecasters-predict-for-major-events-in-2023 What the “superforecasters” predict for major events in 2023]. The Economist. 18 November 2022 The following year, The Economist wrote that all eight of the Superforecasters’ predictions for 2023 were correct, including on global GDP growth, Chinese GDP growth, and election results in Nigeria and Turkey.[https://www.economist.com/the-world-ahead/2023/11/13/what-the-superforecasters-predict-for-major-events-in-2024 What the “superforecasters” predict for major events in 2024]. The Economist. 13 November 2023

In February 2023, Superforecasters made better forecasts than readers of the Financial Times on eight out of nine questions that were resolved at the end of the year.[https://www.ft.com/content/e56d9709-025a-4c29-8d44-f08c0b7a9d2d The art of superforecasting: how FT readers fared against the experts in 2023]. Financial Times. 26 December 2023 In July 2024, the Financial Times reported that Superforecasters "have consistently outperformed financial markets in predicting the Fed's next move".[https://www.ft.com/content/4dedd574-517f-4f19-b972-ab1ca0275b32 Alternative data: Can superforecasters beat the market?]. Financial Times. 18 July 2024 In February 2025, the Financial Times reported that "superforecasters continue to have the edge over the futures market in anticipating what the FOMC [Federal Open Market Committee] will do."[https://www.ft.com/content/b214eb6d-4827-40b6-a096-8cc9de1acb7c Superforecasters continue to beat the market]. Financial Times. 20 February 2025.

=Traits=

One of Tetlock's findings from the Good Judgment Project was that cognitive and personality traits were more important than specialised knowledge when it came to predicting the outcome of various world events typically more accurately than intelligence agencies.{{Sfnp|Burton|2015}} In particular, a 2015 study found that key predictors of forecasting accuracy were "cognitive ability [IQ], political knowledge, and open-mindedness".Mellers B, Stone E, Atanasov P, Rohrbaugh N, Metz SE, Ungar L, et al. "The psychology of intelligence analysis: drivers of prediction accuracy in world politics" [https://www.apa.org/pubs/journals/releases/xap-0000040.pdf (PDF)]. Journal of Experimental Psychology: Applied. 2015;21(1):1-14. Superforecasters "were better at inductive reasoning, pattern detection, cognitive flexibility, and open-mindedness". In the Good Judgment Project, the superforecasters "scored higher on both intelligence and political knowledge than the already well-above-average group of forecasters" who were taking part in the tournament.Mellers B, Stone E, Atanasov P, Rohrbaugh N, Metz SE, Ungar L, et al. "The psychology of intelligence analysis: drivers of prediction accuracy in world politics" [https://www.apa.org/pubs/journals/releases/xap-0000040.pdf (PDF)]. Journal of Experimental Psychology: Applied. 2015;21(1):1-14.

={{Anchor|A-superforecasters}}People=

Regina Joseph, Good Judgment Project superforecaster,{{Cite book |last= |first= |title=Superforecasting: The Art and Science of Prediction |publisher=Crown |year=2015 |isbn=9780804136693 |language=en}}{{Cite AV media |url=https://www.youtube.com/watch?si=PL3rQUsCyy6tA-8q&v=4qMYBeOu1RE&feature=youtu.be |title=Chechnya Abuse & The FBI Firing: VICE News Tonight Full Episode (HBO) |date=2017-05-19 |last=VICE News |access-date=2024-08-27 |via=YouTube}} technologist and founding Editor-in-Chief of Blender Magazine,{{Cite book |last=LLC |first=New York Media |url=https://books.google.com/books/about/New_York_Magazine.html?hl=nl&id=beQCAAAAMBAJ#v=onepage&q=High%20Tech%20Boom%20Town&f=false |title=New York Magazine |date=1995-11-13 |publisher=New York Media, LLC |language=en}}{{Cite AV media |url=https://www.youtube.com/watch?si=oZ6b6QtGVCotw7Qe&v=ehFxJoqS640&feature=youtu.be |title=WHO KNEW The Smartest People In The Room - Regina Joseph & David Hughes |date=2021-04-27 |last=WHO KNEW |access-date=2024-08-27 |via=YouTube}}{{Citation |title=Blender (magazine) |date=2024-08-27 |work=Wikipedia |url=https://en.m.wikipedia.org/wiki/Blender_(magazine) |access-date=2024-08-27 |language=en}} former Futures Division leader and Defence/Security Senior Research Fellow at Clingendael Institute,{{Cite web |last=Joseph |first=Regina |title=Clingendael Futures |url=https://www.clingendael.org/sites/default/files/2016-02/Transnational%20Crime%20and%20Southern%20Europe.pdf |website=Clingendael Futures}} forecasting science researcher{{Cite AV media |url=https://www.youtube.com/watch?v=ESO8_c_zJS4&list=PLUxduCQo6_X3oChP5z08pqp66-mG4Z9SA&index=36 |title=Forecasting Practices and Processes 5 |date=2022-08-15 |last=International Institute of Forecasters |access-date=2024-08-27 |via=YouTube}}{{Cite AV media |url=https://www.youtube.com/watch?si=hru60I7B80V6bKDQ&v=HnW9fRi_liI&feature=youtu.be |title=RP2 Day 2 Lightning Round 7: Social, Behavioral, Economic & Governance |date=2022-01-18 |last=NSF PREPARE |access-date=2024-08-27 |via=YouTube}} and inventor{{Cite web |last=USPTO.report |title=Systems and Methods for Bias-Sensitive Crowd-Sourced Analytics Patent Application |url=https://uspto.report/patent/app/20170309193 |access-date=2024-08-27 |website=USPTO.report |language=en}}{{Cite web |last=USPTO.report |title=Systems and Methods for Multi-Source Reference Class Identification, Base Rate Calculation, and Prediction Patent Application |url=https://uspto.report/patent/app/20210034651 |access-date=2024-08-27 |website=USPTO.report |language=en}}
Elaine Rich, a superforecaster who participated in the Good Judgement Project.{{sfnp|Nilaya|2015|loc=Guests}}
Andrew Sabisky, who resigned from his position as advisor to the United Kingdom government at Downing Street, with chief advisor Dominic Cummings telling journalists "read Philip Tetlock's Superforecasters, instead of political pundits who don't know what they're talking about".{{Sfnp|BBC News|2020}}
Nick Hare, former head of futures and analytical methods at the Ministry of Defence (MoD).{{Sfnp|Burton|2015}}
Reed Roberts, a former PhD student in Chemistry.{{Sfnp|Burton|2015}}
Jonathon Kitson{{Cite web|title=Superforecasting: The Future's Chequered Past and Present|url=https://whynow.co.uk/read/superforecasting-the-futures-chequered-past-and-present/|access-date=2021-07-17|website=whynow|date=8 February 2021 |language=en}}
Jean-Pierre Beugoms{{Cite web|title=Superforecaster Profiles|url=https://goodjudgment.com/about/our-team/superforecaster-profiles/|access-date=2021-07-17|website=Good Judgment|language=en-US}}
Dan Mayland
Kjirste Morrell
Dominic Smith

Criticism

The concept of superforecasting has been criticised from multiple angles. Nassim Nicholas Taleb has been a particularly strong critic, arguing among other claims that forecasting is not useful to decision makers and that the lack of financial gain accrued by superforecasters is a sign that their actual predictive powers are lacking.{{cite journal |last1=Taleb |first1=Nassim Nicholas |last2=Richman |first2=Ronald |last3=Carreira |first3=Marcos |last4=Sharpe |first4=James |title=The probability conflation: A reply to Tetlock et al. |journal=International Journal of Forecasting |date=1 April 2023 |volume=39 |issue=2 |pages=1026–1029 |doi=10.1016/j.ijforecast.2023.01.005 }} Counter-terrorism expert Suzanne Raine criticises it for placing too much emphasis on "what is going to happen" rather than "what is happening" and "how can the future be changed".{{cite web |last1=Raine |first1=Suzanne |title=Superforecasting will not save us |url=https://engelsbergideas.com/essays/superforecasting-will-not-save-us/ |website=Engelsberg ideas |access-date=26 August 2024 |language=sv}}

References

{{Cite news|last=Adonis|first=Andrew|date=20 February 2020|title=In praise of (some) superforecasters

|url=https://www.theneweuropean.co.uk/top-stories/andrew-adonis-on-dominic-cummings-superforecasters-and-andrew-sabisky-1-6523611|url-status=live

|archive-date=20 February 2020|archive-url=https://archive.today/20200220152204/https://www.theneweuropean.co.uk/top-stories/andrew-adonis-on-dominic-cummings-superforecasters-and-andrew-sabisky-1-6523611

|website=The New European|access-date=20 February 2020|authorlink=Andrew Adonis}}

{{cite news|author=BBC News|date=18 February 2020|title=Andrew Sabisky: What is superforecasting?

|url=https://www.bbc.co.uk/news/uk-politics-51545541|website=BBC News|access-date=18 February 2020|url-status=live

|archive-url=https://web.archive.org/web/20200218170959/https://www.bbc.co.uk/news/uk-politics-51545541|archive-date=18 February 2020}}

{{cite journal|author=Bobby W.|date=December 2019|title=The Limits of Prediction—or, How I Learned to Stop Worrying About Black Swans and Love Analysis

|url=https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/csi-studies/studies/vol-63-no-4/pdfs/Limits-of-Prediction.pdf

|journal=Studies in Intelligence|volume=63|issue=4|url-status=dead

|archive-url=https://web.archive.org/web/20200219000100/https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/csi-studies/studies/vol-63-no-4/pdfs/Limits-of-Prediction.pdf|archive-date=19 February 2020}}

{{cite web|last=Burton|first=Tara Isabella|date=20 January 2015

|archive-url=https://archive.today/20200219045532/https://www.bbc.com/future/article/20190903-linguistic-fluency-proficiency-second-language-learning|archive-date=19 February 2020}}

{{cite magazine|last=Harford|first=Tim|date=5 September 2014|title=How to see into the future

|url=https://www.ft.com/content/3950604a-33bc-11e4-ba62-00144feabdc0|url-status=live

|magazine=Financial Times Magazine|publisher=FT Group|access-date=18 February 2020

|archive-url=https://web.archive.org/web/20190923203302/https://www.ft.com/content/3950604a-33bc-11e4-ba62-00144feabdc0|archive-date=23 September 2019}}

{{cite web|last=Nilaya|first=Josh|date=29 September 2015|title=U.S. Intelligence Dabbles in Forecasting the Future

|url=https://www.wnpr.org/post/us-intelligence-dabbles-forecasting-future#stream/0|website=Connecticut Public Radio Home|access-date=21 February 2020

|archive-url=https://web.archive.org/web/20190111112421/http://www.wnpr.org/post/us-intelligence-dabbles-forecasting-future|archive-date=11 January 2019|url-status=live}}