Results 1 to 7 of 7

Thread: Punditry and prediction

  1. #1
    Senior Contributor
    Join Date
    09 Oct 10
    Posts
    1,074

    Punditry and prediction

    I want to draw attention to some scientific work on political punditry. The research tested several hundred political pundits having them make long term predictions on matters closely and not so closely related to their fields of interest. They could choose three options based broadly on probabilities scales.

    The psychologist Philip Kaufmann has concluded they performed worse than "dart throwing monkeys" and would have been better off just assigning equal probabilty to each option to all predictions The research found that performance was barley better than random, and that those that were the most famous or most specialised performed the poorest.

    This has led to the development of the Good Judgement Project and has significant consequences in both economics and geopolitics.

    These findings were reported widely in the media and came to the attention of Intelligence Advanced Research Projects Activity (IARPA) inside the United States intelligence community—a fact that was partly responsible for the 2011 launch of a four-year geopolitical forecasting tournament that engaged tens of thousands of forecasters and drew over one million forecasts across roughly 500 questions of relevance to U.S. national security, broadly defined.

    Since 2011, Tetlock and his wife/research partner Barbara Mellers have been co-leaders of the Good Judgment Project (GJP), a research collaborative that emerged as the winner of the IARPA tournament.[3] The original aim of the tournament was to improve geo-political and geo-economic forecasting. Illustrative questions include “What is the chance that a member will withdraw from the European Union by a target date?” or “What is the likelihood of naval clashes claiming over 10 lives in the East China Sea?” or “How likely is the head of state of Venezuela to resign by a target date?” The tournament challenged GJP and its competitors at other academic institutions to come up with innovative methods of recruiting gifted forecasters, methods of training forecasters in basic principles of probabilistic reasoning, methods of forming teams that are more than the sum of their individual parts and methods of developing aggregation algorithms that most effectively distill the wisdom of the crowd.[3][4][5][6][7][8]

    Among the more surprising findings from the tournament were:

    the degree to which simple training exercises improved the accuracy of probabilistic judgments as measured by Brier scores;[3][4]
    the degree to which the best forecasters could learn to distinguish many degrees of uncertainty along the zero to 1.0 probability scale (many more distinctions than the traditional 7-point verbal scale used by the National Intelligence Council);[4][9]
    the consistency of the performance of the elite forecasters (superforecasters) across time and categories of questions;[4][5][6]
    the power of a log-odds extremizing aggregation algorithm to out-perform competitors;[7][10] and
    the apparent ability of GJP to generate probability estimates that were "reportedly 30% better than intelligence officers with access to actual classified information."
    Tetlock and Mellers[11] see forecasting tournaments as a possible mechanism for helping intelligence agencies escape from blame-game (or accountability) ping-pong in which agencies find themselves whipsawed between clashing critiques that they were either too slow to issue warnings (false negatives such as 9/11) and too fast to issue warnings (false positives). They argue that tournaments are ways of signaling that an organization is committed to playing a pure accuracy game –and generating probability estimates that are as accurate as possible (and not tilting estimates to avoid the most recent “mistake”).
    https://en.wikipedia.org/wiki/Philip_E._Tetlock

    They have a website which allow individuals and groups to engage in prediction forecasting on a huge variety of topics across the world - economic, defense and political in nature.
    https://www.gjopen.com/questions?filter=featured

    Kaufmann states that humans tend to focus on singular big ideas and develop coherent stories based on them which gives them confidence on future forecasting (just look at how experts performed). Also, we develop coherent stories on past events with hindsight which creates the illusion that past events were really predictable all along and this feeds into the delusion of future event forecasting. The reality is that future prediction is dependent on the certainty of events that are usually highly unpredictable (that seems absurdly simple) but everyone still slips into the delusion that they can outperform a monkey with darts once discussion gets going on future events.

    .

  2. #2
    Senior Contributor
    Join Date
    09 Oct 10
    Posts
    1,074
    Forgot this
    Tetlock conducted a set of small scale forecasting tournaments between 1984 and 2003. The forecasters were 284 experts from a variety of fields, including government officials, professors, journalists, and others, with many opinions, from Marxists to free-marketeers.

    The tournaments solicited roughly 28,000 predictions about the future and found the forecasters were often only slightly more accurate than chance, and usually worse than basic extrapolation algorithms, especially on longer–range forecasts three to five years out. Forecasters with the biggest news media profiles were also especially bad. This work suggests that there is a perverse inverse relationship between fame and accuracy.
    https://en.wikipedia.org/wiki/Philip_E._Tetlock

  3. #3
    Senior Contributor DonBelt's Avatar
    Join Date
    01 Oct 08
    Location
    Taxachusetts, somewhere between Boston and Wista
    Posts
    1,065
    Somehow I find this kind of a relief.

  4. #4
    Turbanator Senior Contributor Double Edge's Avatar
    Join Date
    11 Sep 10
    Location
    Bangalore
    Posts
    5,970
    Most of the time its hard enough just to figure out what happened let alone predict anything.

    An understanding of the past gives us better tools to understand the future. To then go make predictions on the basis of it, how long out ? short term is possible, longer term is harder.

    so what then to make of the Chinese long game way of thinking ?

  5. #5
    Official Thread Jacker Senior Contributor gunnut's Avatar
    Join Date
    27 Jan 06
    Location
    DPRK, Demokratik People's Republik of Kalifornia
    Posts
    23,781
    I am so glad these same "experts" are also involved in saving the planet from "climate change."
    "Only Nixon can go to China." -- Old Vulcan proverb.

  6. #6
    Senior Contributor DOR's Avatar
    Join Date
    08 Mar 11
    Location
    London
    Posts
    2,399
    Quote Originally Posted by Double Edge View Post
    Most of the time its hard enough just to figure out what happened let alone predict anything.

    An understanding of the past gives us better tools to understand the future. To then go make predictions on the basis of it, how long out ? short term is possible, longer term is harder.

    so what then to make of the Chinese long game way of thinking ?
    The Chinese long-game is a myth. Look at financial reform, for just one example: Lots of discussion over many years, but when it comes time to make a move, little happens.

  7. #7
    Senior Contributor
    Join Date
    09 Oct 10
    Posts
    1,074
    Quote Originally Posted by Double Edge View Post

    how long out ? short term is possible, longer term is harder.
    Tetlocks original work focused on 3 to 5 year time ranges and he drew negative conclusions on expert ability to forecast. Later work focused on shorter ranges, inside twelve months.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. My prediction for Nov. 6th...
    By gunnut in forum American Politics & Economy
    Replies: 17
    Last Post: 07 Nov 12,, 11:14
  2. snaps prediction
    By snapper in forum American Politics & Economy
    Replies: 10
    Last Post: 19 Sep 12,, 17:45
  3. My prediction on the economy
    By gunnut in forum International Economy
    Replies: 30
    Last Post: 14 Nov 08,, 18:36
  4. WAB Prediction Market - Brigadier Ray to 100,000 posts
    By Shek in forum World Affairs Board Pub
    Replies: 29
    Last Post: 12 May 08,, 17:29
  5. U.S. Envoy Offers Grim Prediction on Iraq Pullout
    By JAD_333 in forum The Middle East and North Africa
    Replies: 14
    Last Post: 14 Jul 07,, 21:45

Share this thread with friends:

Share this thread with friends:

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •