O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
New methods of REF assessment: Prediction Markets Jisc Research Analytics Webinar 1 July 2019 Jackie Thompson University o...
The Problem
Prediction markets: replication studies
The market interface https://www.refpredictionmarkets.com/screencast.html
Prediction Market specs • 21 participants – psychology department • Postdocs and teaching/research academics • Financial i...
Rationale • Condenses crowd wisdom • Dynamically weighted • Incentives • Fine-grained rankings • Burden
Assessment methods • Close reading 1 (Internal panel) • Close reading 2 (external rater) • Prediction market • Machine lea...
Results: Correlations Spearman's rho Internal CR External CR Prediction Market Machine Learning Internal CR 1 .534** .732*...
5 7 9 11 13 0.4 0.6 0.8 1 5 7 9 11 13 0.4 0.6 0.8 1 5 7 9 11 13 5 7 9 11 13 External CR Internal CR Prediction Market Exte...
Further plans • User feedback mostly positive • Rolling out beyond Bristol soon (contact jackie.thompson@bristol.ac.uk)
Thank you! Research Contributors: Marcus Munafò (U. Bristol) Ian Penton-Voak (U. Bristol) Anna Dreber Almenberg (Stockholm...
Próximos SlideShares
Carregando em…5
×

Prediction markets

82 visualizações

Publicada em

The aim of this project was to develop and evaluate a prediction market tool that higher education institutions can use to rank outputs for potential REF submissions as part of their internal REF (Research Excellence Framework) planning. A prediction market is a bit like the stock market, except instead of investing in companies, participants invest in the outcomes of future events (in this case, ratings of research outputs). This presentation will give some background to the project and details of the prediction markets that have been tested with Units of Assessment at the University of Bristol. It will include a demo of the tool used and the lessons learned from the first round of markets.  

Publicada em: Dados e análise
no profile picture user

  • Entre para ver os comentários

  • Seja a primeira pessoa a gostar disto

Prediction markets

  1. 1. New methods of REF assessment: Prediction Markets Jisc Research Analytics Webinar 1 July 2019 Jackie Thompson University of Bristol
  2. 2. The Problem
  3. 3. Prediction markets: replication studies
  4. 4. The market interface https://www.refpredictionmarkets.com/screencast.html
  5. 5. Prediction Market specs • 21 participants – psychology department • Postdocs and teaching/research academics • Financial incentives • 11 days • 29 papers
  6. 6. Rationale • Condenses crowd wisdom • Dynamically weighted • Incentives • Fine-grained rankings • Burden
  7. 7. Assessment methods • Close reading 1 (Internal panel) • Close reading 2 (external rater) • Prediction market • Machine learning
  8. 8. Results: Correlations Spearman's rho Internal CR External CR Prediction Market Machine Learning Internal CR 1 .534** .732*** .655*** External CR 1 .601*** .739*** Prediction Market 1 .643*** Machine Learning 1
  9. 9. 5 7 9 11 13 0.4 0.6 0.8 1 5 7 9 11 13 0.4 0.6 0.8 1 5 7 9 11 13 5 7 9 11 13 External CR Internal CR Prediction Market External CR Prediction Market Machine Learning 0.4 0.6 0.8 1 1 2 3 4 5 7 9 11 13 1 2 3 4 5 7 9 11 13 1 2 3 4
  10. 10. Further plans • User feedback mostly positive • Rolling out beyond Bristol soon (contact jackie.thompson@bristol.ac.uk)
  11. 11. Thank you! Research Contributors: Marcus Munafò (U. Bristol) Ian Penton-Voak (U. Bristol) Anna Dreber Almenberg (Stockholm School of Economics) Felix Holzmeister (U. Innsbruck)

×