The Good Judgment Project#Media coverage

The Good Judgment Project (GJP) is an organization dedicated to "harnessing the wisdom of the crowd to forecast world events". It was co-created by Philip E. Tetlock (author of Superforecasting: The Art and Science of Prediction and Expert Political Judgment), decision scientist Barbara Mellers, and Don Moore, all professors at the University of Pennsylvania.{{cite web|title=Welcome to the Good Judgment Project|url=http://www.gjopen.com/|publisher=The Good Judgment Project|accessdate=May 5, 2014}}{{cite web|date=July 27, 2011|title=Who's who in the Good Judgment Project|url=http://goodjudgmentproject.com/|publisher=The Good Judgment Project|accessdate=May 5, 2014}}{{cite news|last=Brooks|first=David|date=March 21, 2013|title=Forecasting Fox|newspaper=New York Times|url=https://www.nytimes.com/2013/03/22/opinion/brooks-forecasting-fox.html|accessdate=May 5, 2014}}

The project began as a participant in the Aggregative Contingent Estimation (ACE) program of the Intelligence Advanced Research Projects Activity (IARPA).{{cite web|title=The Project|url=http://www.goodjudgmentproject.com/about_project.html|url-status=dead|publisher=The Good Judgment Project|accessdate=May 5, 2014|archiveurl=https://web.archive.org/web/20140506201121/http://www.goodjudgmentproject.com/about_project.html|archivedate=May 6, 2014}}{{cite news|last=Horowitz|first=Michael|date=November 26, 2013|title=Good judgment in forecasting international affairs (and an invitation for season 3)|newspaper=Washington Post|url=https://www.washingtonpost.com/blogs/monkey-cage/wp/2013/11/26/good-judgment-in-forecasting-international-affairs-and-an-invitation-for-season-3/|accessdate=May 5, 2014}} It then extended its crowd wisdom to commercial activities, recruiting forecasters and aggregating the predictions of the most historically accurate among them to forecast future events.{{Cite web|title=About Superforecasting {{!}} Unprecedented Accurate & Precise Forecasting|url=https://goodjudgment.com/about/|access-date=2022-02-17|website=Good Judgment|language=en-US}}{{Cite web|last=Matthews|first=Dylan|date=2022-02-16|title=How can we prevent major conflicts like a Russia-Ukraine war?|url=https://www.vox.com/future-perfect/2022/2/16/22935927/russia-ukraine-great-power-conflict|access-date=2022-02-17|website=Vox|language=en}} Predictions are scored using Brier scores.{{cite web|last=Dickenson|first=Matt|date=November 12, 2013|title=Prediction and Good Judgment: Can ICEWS Inform Forecasts?|url=http://predictiveheuristics.com/2013/11/12/prediction-and-good-judgment-can-icews-inform-forecasts/|publisher=Predictive Heuristics|accessdate=May 24, 2014}} The top forecasters in GJP are "reportedly 30% better than intelligence officers with access to actual classified information."

History

The Good Judgment Project began in July 2011 in collaboration with the Aggregative Contingent Estimation (ACE) Program at IARPA (IARPA-ACE).{{cite web|url = http://goodjudgmentproject.com/blog/?p=4|title = The idea behind the Good Judgment Project|date = July 27, 2011|accessdate = May 5, 2014|publisher = The Good Judgment Project|url-status = dead|archiveurl = https://web.archive.org/web/20140506202243/http://goodjudgmentproject.com/blog/?p=4|archivedate = May 6, 2014}} The first contest began in September 2011. GJP was one of many entrants in the IARPA-ACE tournament, which posed around 100 to 150 questions each year on geopolitical events. The GJP research team gathered a large number of talented amateurs (rather than geopolitical subject matter experts), gave them basic tutorials on forecasting best practice and overcoming cognitive biases, and created an aggregation algorithm to combine the individual predictions of the forecasters. GJP won both seasons of the contest, and were 35% to 72% more accurate than any other research team.{{Cite web|title=The first championship season|url=https://goodjudgment.com/resources/the-superforecasters-track-record/the-first-championship-season/|access-date=2022-02-17|website=Good Judgment|language=en-US}} Starting with the summer of 2013, GJP were the only research team IARPA-ACE was still funding, and GJP participants had access to the Integrated Conflict Early Warning System.

People

The co-leaders of the GJP include Philip Tetlock, Barbara Mellers and Don Moore. The website lists a total of about 30 team members, including the co-leaders as well as David Budescu, Lyle Ungar, Jonathan Baron, and prediction-markets entrepreneur Emile Servan-Schreiber.{{cite web|url=http://www.goodjudgmentproject.com/|title = Team|accessdate = May 5, 2014|publisher = The Good Judgment Project}} The advisory board included Daniel Kahneman, Robert Jervis, J. Scott Armstrong, Michael Mauboussin, Carl Spetzler and Justin Wolfers.{{cite web| title = Freakonomics| work = Sign Up for a Prediction Tournament| date = 2011-08-04|url=http://freakonomics.com/2011/08/04/sign-up-for-a-prediction-tournament/}} The study employed several thousand people as volunteer forecasters. Using personality-trait tests, training methods and strategies the researchers at GJP were able to select forecasting participants with less cognitive bias than the average person; as the forecasting contest continued the researchers were able to further down select these individuals in groups of so-called superforecasters. The last season of the GJP enlisted a total of 260 superforecasters.{{Citation needed|date=February 2022}}

Research

A significant amount of research has been conducted based on the Good Judgment Project by the people involved with it.{{multiref2

| {{Cite journal|last1=Atanasov|first1=Pavel|last2=Rescober|first2=Phillip|last3=Stone|first3=Eric|last4=Swift|first4=Samuel A.|last5=Servan-Schreiber|first5=Emile|last6=Tetlock|first6=Philip|last7=Ungar|first7=Lyle|last8=Mellers|first8=Barbara|date=2016-04-22|title=Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls|url=https://pubsonline.informs.org/doi/abs/10.1287/mnsc.2015.2374|journal=Management Science|volume=63|issue=3|pages=691–706|doi=10.1287/mnsc.2015.2374|issn=0025-1909|url-access=subscription}}

| {{cite web |url=http://www.aaai.org/ocs/index.php/FSS/FSS12/paper/viewFile/5570/5871/ |title=The Good Judgment Project: A Large Scale Test of Different Methods of Combining Expert Predictions |last1=Ungar |first1=Lyle |last2=Mellers |first2=Barbara |last3=Satopää |first3=Ville |last4=Baron |first4=Jon |last5=Tetlock |first5=Philip E. |author5-link=Philip E. Tetlock |last6=Ramos |first6=Jaime |last7=Swift |first7=Sam |publisher=Association for the Advancement of Artificial Intelligence}}

| {{cite report |ssrn=2418980 |title=Using the 'Wisdom of (Expert) Crowds' to Forecast Mass Atrocities |last=Ulfelder |first=Jay |date=March 27, 2014 |doi=10.2139/ssrn.2418980 }}

| {{Cite journal|last1=Atanasov|first1=P.|last2=Witkowski|first2=J.|last3=Ungar|first3=L.|last4=Mellers|first4=B.|last5=Tetlock|first5=P.|date=2020|title=Small steps to accuracy: Incremental belief updaters are better forecasters|journal=Organizational Behavior and Human Decision Processes |volume=160 |pages=19–35 |doi=10.1016/j.obhdp.2020.02.001|s2cid=216273470 |doi-access=free}}}} The results show that harnessing a blend of statistics, psychology, training and various levels of interaction between individual forecasters, consistently produced the best forecast for several years in a row.{{cite journal |doi=10.1177/0956797614524255 |issn=1467-9280 |volume=25 |issue=5 |pages=1106–1115 |last1=Mellers |first1=Barbara |last2 =Ungar |first2=Lyle |last3=Baron |first3=Jonathan |last4=Ramos |first4=Jaime |last5=Gurcay |first5=Burcu |last6=Fincher |first6=Katrina |last7=Scott |first7=Sydney E. |last8=Moore |first8=Don |last9=Atanasov |first9=Pavel |last10=Swift |first10=Samuel A. |last11=Murray |first11=Terry |last12=Stone |first12=Eric |last13=Tetlock |first13=Philip E. |title=Psychological strategies for winning a geopolitical forecasting tournament |journal=Psychological Science |date=2014-05-01 |pmid=24659192|s2cid=42143367 |url=https://escholarship.org/uc/item/4rg4n9vr }}

Good Judgment Inc.

A commercial spin-off of the Good Judgment Project started to operate on the web in July 2015 under the name Good Judgment Inc. Their services include forecasts on questions of general interest, custom forecasts, and training in Good Judgment's forecasting techniques.{{cite web| title=Meet the Elite Team of Superforecasters Who Have Turned Future-Gazing Into a Science| url=https://www.entrepreneur.com/article/395236| last=Brody| first=Liz| date=January 1, 2022| magazine=Entrepreneur}} Starting in September 2015, Good Judgment Inc has been running a public forecasting tournament at the Good Judgment Open site. Like the Good Judgment Project, Good Judgment Open has questions about geopolitical and financial events, although it also has questions about US politics, entertainment, and sports.{{Cite web|title=How the Good Judgment Project's Superforecasters Use Data to Make Predictions| last=Gossett| first=Stephen |url=https://builtin.com/data-science/superforecasters-good-judgement| date=August 6, 2020| access-date=2022-06-07|website=builtin.com|language=en}}{{Cite web|title=Good Judgment® Open|url=https://www.gjopen.com/|access-date=2022-02-17|website=www.gjopen.com}}

Media coverage

GJP has repeatedly been discussed in The Economist.{{cite news|url=https://www.economist.com/blogs/freeexchange/2011/09/perils-prediction|title = The perils of prediction: Adventures in punditry|date = September 2, 2011|accessdate = May 6, 2014|newspaper = The Economist}}{{cite news|url=https://www.economist.com/blogs/freeexchange/2011/09/monetary-policy-0|title = Monetary policy: How likely is deflation?|date = September 13, 2011|accessdate = May 6, 2014|newspaper = The Economist}}{{cite news|url=https://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts|title = International: Who's good at forecasts? How to sort the best from the rest|date = November 18, 2013|accessdate = May 6, 2014|newspaper = The Economist}}{{cite news| url=https://www.economist.com/the-world-ahead/2021/11/10/the-experts-best-bets| title=The experts' best bets| date=November 10, 2021| accessdate=June 7, 2022| newspaper=The Economist}} GJP has also been covered in The New York Times, The Washington Post,{{cite news|url=https://www.washingtonpost.com/opinions/david-ignatius-more-chatter-than-needed/2013/11/01/1194a984-425a-11e3-a624-41d661b0bb78_story.html|title = More chatter than needed|last = Ignatius|first = David|authorlink = David Ignatius|date = November 1, 2013|accessdate = May 6, 2014|newspaper = Washington Post}}{{cite web|url=http://www.businessinsider.com/good-judgement-project-accurate-predictions-2014-4|title = Huge Experiment Finds Regular Folks Predict World Events Better Than CIA Agents|last = Bender|first = Jeremy|date = April 3, 2014|accessdate = May 6, 2014|website = Business Insider}} and Co.Exist.{{cite web|url=http://www.fastcoexist.com/3028840/the-surprising-accuracy-of-crowdsourced-predictions-about-the-future|title = The Surprising Accuracy Of Crowdsourced Predictions About The Future. Do you know whether Turkey will get a new constitution? It turns out you do: A group of well-informed citizens can predict future events more often than any foreign policy expert or CIA analyst.|date = April 21, 2014|accessdate = May 6, 2014|publisher = Co.exist}} NPR aired a segment on The Good Judgment Project by the title "So You Think You're Smarter Than a CIA Agent", on April 2, 2014.{{cite web

| last = Spiegel

| first = Alix

| title = So You Think You're Smarter Than A CIA Agent

| work = NPR.org

| accessdate = 2014-08-18

| url = https://www.npr.org/blogs/parallels/2014/04/02/297839429/-so-you-think-youre-smarter-than-a-cia-agent

}} The Financial Times published an article on the GJP on September 5, 2014.{{Cite news

| issn = 0307-1766

| last = Harford

| first = Tim

| title = How to see into the future

| work = Financial Times

| accessdate = 2014-09-05

| date = 2014-09-05

| url = http://www.ft.com/intl/cms/s/2/3950604a-33bc-11e4-ba62-00144feabdc0.html#axzz3CTBRBAGH}} Washingtonian published an article that mentioned the GJP on January 8, 2015.{{cite web |last=Hamilton|first=Keegan

| title = How US Agencies Are Using the Web to Pick Our Brains

| work = Washingtonian

|date=8 January 2015

| accessdate = 2015-01-24

| url = http://www.washingtonian.com/blogs/capitalcomment/national-security/us-agencies-are-using-the-web-to-pick-our-brains.php

}} The BBC and The Washington Post published articles on the GJP respectively on January 20, 21, and 29, 2015.{{cite web |last = Burton|first=Tara|date = 2015-01-20

| title = Could you be a 'super-forecaster'?

| work = BBC Future

| accessdate = 2015-01-21

| url = http://www.bbc.com/future/story/20150120-are-you-a-super-forecaster

}}{{Cite news

| issn = 0190-8286

| last = Jensen

| first = Nathan

| title = Experts see a Republican Senate and fast-track authority for Obama as keys to new trade agreements

| newspaper = The Washington Post

| accessdate = 2015-01-21

| date = 2015-01-21

| url = https://www.washingtonpost.com/blogs/monkey-cage/wp/2015/01/21/experts-see-a-republican-senate-and-fast-track-authority-for-obama-as-keys-to-new-trade-agreements/

}}{{Cite news

| issn = 0190-8286

| last = Mellers

| first = Barbara

|author2=Michael C. Horowitz

| title = Does anyone make accurate geopolitical predictions?

| newspaper = The Washington Post

| accessdate = 2015-01-30

| date = 2015-01-29

| url = https://www.washingtonpost.com/blogs/monkey-cage/wp/2015/01/29/does-anyone-make-accurate-geopolitical-predictions/?al}}

The Almanac of Menlo Park published a story on the GJP on January 29, 2015.{{cite web| title = Feature story: Bob Sawyer of Woodside discovers his latent talent in forecasting| accessdate = 2015-03-17| url = http://www.almanacnews.com/news/2015/01/29/feature-story-bob-sawyer-of-woodside-discovers-his-latent-talent-in-forecasting}} An article on the GJP appeared on the portal of the Philadelphia Inquirer, Philly.com, on February 4, 2015.{{cite web

| last = Dribben

| first = Melissa

|author2=Inquirer Staff Writer

| title = Fortune telling: Crowds surpass pundits

| work = Philly.com

| accessdate = 2015-02-06

| date = 2015-02-04

| url = http://www.philly.com/philly/news/science/20150204_Fortune_telling__Crowds_surpass_pundits.html}} The book Wiser: Getting Beyond Groupthink to Make Groups Smarter has a section detailing the involvement of the GJP in the tournament run by IARPA.{{cite book |publisher=Harvard Business Review Press |isbn=978-1-4221-2299-0 |last1=Sunstein |first1=Cass R. |author1-link=Cass Sunstein |last2=Hastie |first2=Reid |title=Wiser: Getting Beyond Groupthink to Make Groups Smarter |date=2014-12-23}} Psychology Today published online a short article summarizing the paper by Mellers, et al., that wraps up the main findings of the GJP.{{cite web |title=Who's Best at Predicting the Future? (and How to Get Better) |work=Psychology Today |accessdate=2015-07-11 |url=https://www.psychologytoday.com/blog/the-sports-mind/201505/whos-best-predicting-the-future-and-how-get-better}}{{cite journal |doi=10.1177/1745691615577794 |issn=1745-6916 |volume=10 |issue=3 |pages=267–281 |last1=Mellers |first1=Barbara |last2=Stone |first2=Eric |last3=Murray |first3=Terry |last4=Minster |first4=Angela |last5=Rohrbaugh |first5=Nick |last6=Bishop |first6=Michael |last7=Chen |first7=Eva |last8=Baker |first8=Joshua |last9=Hou |first9=Yuan |last10=Horowitz |first10=Michael |last11=Ungar |first11=Lyle |last12=Tetlock |first12=Philip |title=Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions |journal=Perspectives on Psychological Science |date=2015-05-01 |pmid=25987508|s2cid=3118872 | url=https://www.researchgate.net/publication/277087515}}

The project spawned a 2015 book by Tetlock and coauthored by Dan Gardner, Superforecasting - The Art and Science of Prediction, which divulges the main findings of the research conducted with the data from the GJP.{{Cite book| publisher = Crown| isbn = 978-0-8041-3669-3| last1 = Tetlock| first1 = Philip E.| last2 = Gardner| first2 = Dan| title = Superforecasting - The Art and Science of Prediction| location = New York| date = 2015-09-29}} Co-author Gardner had already published a book in 2010, that quoted previous research by Tetlock that seeded the GJP effort.{{Cite book| publisher = McClelland & Stewart| last = Gardner| first = Dan| title = Future Babble: Why Expert Predictions Fail - and Why We Believe Them Anyway| date = 2010-10-12}} A book review in the September 26, 2015, print edition of the Economist discusses the main concepts.{{Cite news| issn = 0013-0613| title = Unclouded vision| newspaper = The Economist| accessdate = 2015-09-24| date = 2015-09-26| url = https://www.economist.com/news/books-and-arts/21666098-forecasting-talent-luckily-it-can-be-learned-unclouded-vision}} A Wall Street Journal article depicts it as: "The most important book on decision making since Daniel Kahneman’s Thinking, Fast and Slow."{{cite news|last1=Zweig|first1=Jason|title=Can You See the Future? Probably Better Than Professional Forecasters|url=https://blogs.wsj.com/moneybeat/2015/09/25/can-you-see-the-future-probably-better-than-professional-forecasters/|accessdate=September 25, 2015|newspaper=The Wall Street Journal|quote=I think Philip Tetlock’s 'Superforecasting: The Art and Science of Prediction,' ... is the most important book on decision making since Daniel Kahneman's 'Thinking, Fast and Slow.'}} The Harvard Business Review paired it with the book How Not to Be Wrong: The Power of Mathematical Thinking by Jordan Ellenberg.{{cite web| last = Frick| first = Walter| title = Question Certainty| work = Harvard Business Review| accessdate = 2015-09-26| url = https://hbr.org/2015/10/question-certainty}}

On September 30, 2015, NPR aired an episode of the Colin McEnroe Show centering on the GJP and the book Superforecasting; guests on the show were Tetlock, IARPA Director Jason Matheny, and superforecaster Elaine Rich.{{cite news|last1=McEnroe|first1=Colin|last2=Wolf|first2=Chion|url=http://wnpr.org/post/us-intelligence-dabbles-forecasting-future#stream/0|accessdate=October 1, 2015|title=The Colin McEnroe Show|agency=National Public Radio|publisher=WNPR}}