Quiz: Real-World Applications

Question 1

Which of the following best describes Google's experience with internal prediction markets?

A) Markets consistently underperformed official forecasts but provided useful diversity of opinion B) Markets outperformed official forecasts, were particularly good at surfacing bad news, and showed detectable optimism biases C) Markets performed identically to official forecasts but at lower cost D) Markets were abandoned quickly due to low participation

Answer: B Explanation: Google's internal markets, studied by Bo Cowgill, outperformed official forecasts for product launches and operational targets. They were notably effective at surfacing "bad news" that didn't travel well through hierarchies. However, they showed detectable biases, with employees being optimistic about their own projects and about Google overall.

Question 2

HP's prediction market experiments with printer sales forecasting demonstrated that:

A) Markets require at least 1,000 participants to outperform traditional forecasts B) Markets with as few as 12-20 participants outperformed sophisticated statistical forecasting models in most test cases C) Markets only work for technology products, not consumer goods D) Real-money incentives are essential for corporate prediction market accuracy

Answer: B Explanation: HP's experiments, led by Kay-Yut Chen and Charles Plott, showed that in 6 out of 8 test cases, markets with only 12-20 participants outperformed HP's official sales forecasts produced by experienced analysts using sophisticated statistical models. This challenged the assumption that large participant pools are necessary.

Question 3

The Brier score is defined as $\frac{1}{n}\sum_{i=1}^{n}(p_i - o_i)^2$. If a prediction market assigns probabilities of 0.80 and 0.30 to two events, and both events occur, what is the Brier score?

A) 0.085 B) 0.265 C) 0.55 D) 0.125

Answer: B Explanation: Brier = [(0.80 - 1)^2 + (0.30 - 1)^2] / 2 = [0.04 + 0.49] / 2 = 0.53 / 2 = 0.265.

Question 4

In the Good Judgment Project, "superforecasters" were found to:

A) Have special access to classified information B) Represent roughly 2% of participants and outperform intelligence analysts with classified access by approximately 30% C) Use sophisticated statistical models rather than judgment D) Perform well individually but poorly in teams

Answer: B Explanation: The GJP found that approximately 2% of participants --- "superforecasters" --- were dramatically better than others. These individuals outperformed professional intelligence analysts who had access to classified information by about 30% as measured by Brier scores. When grouped into teams, their performance improved further.

Question 5

The extremizing transformation $p_{\text{agg}} = \frac{\bar{p}^a}{\bar{p}^a + (1-\bar{p})^a}$ is used to:

A) Reduce the variance of crowd forecasts B) Push the aggregate forecast toward 0.50 to reflect uncertainty C) Correct for the tendency of averaged forecasts to be insufficiently extreme D) Weight forecasters by their past accuracy

Answer: C Explanation: When individual forecasts are averaged, the result tends to be pulled toward 0.50 because averaging attenuates signals. The extremizing transformation with $a > 1$ pushes the average away from 0.50 toward the more extreme values, correcting for this attenuation. In the GJP, values around $a \approx 2.5$ were found to be optimal.

Question 6

Which of the following was NOT a key finding from COVID-19 pandemic prediction markets?

A) Forecasting platforms generally produced better vaccine timeline estimates than early expert consensus B) Markets incorporated new information faster than institutional forecasts C) Markets consistently outperformed epidemiological models for case count predictions D) Markets provided natural probability distributions quantifying uncertainty

Answer: C Explanation: While pandemic prediction markets showed strengths in vaccine timeline forecasting, speed of information incorporation, and uncertainty quantification, they did not consistently outperform epidemiological models for case count predictions. Thin participation and domain expertise gaps limited their accuracy on technical epidemiological questions.

Question 7

The fed funds futures-implied probability of a rate hike is calculated as:

$$P(\text{hike}) = \frac{r_{\text{futures}} - r_{\text{current}}}{r_{\text{hike}} - r_{\text{current}}}$$

If the current rate is 5.25%, futures imply 5.31%, and a hike would be to 5.50%, what is the implied probability?

A) 12% B) 24% C) 36% D) 48%

Answer: B Explanation: P(hike) = (5.31 - 5.25) / (5.50 - 5.25) = 0.06 / 0.25 = 0.24, or 24%.

Question 8

The DARPA Policy Analysis Market (PAM) was cancelled because:

A) It failed to produce accurate forecasts in pilot testing B) Congressional opposition characterized it as a "terrorism futures market" C) The technology was not mature enough to support large-scale trading D) Intelligence agencies preferred their existing forecasting methods

Answer: B Explanation: PAM was cancelled in 2003 after congressional critics called it a "terrorism futures market." The cancellation is widely considered an overreaction, as the proposed market would not have allowed betting on specific terrorist attacks and could have provided valuable intelligence signals. The episode set back prediction market development by years.

Question 9

In the context of corporate prediction markets, which combination of features is most likely to produce accurate forecasts?

A) Mandatory participation, long time horizons, questions within a single department B) Voluntary participation, short-to-medium time horizons, cross-functional questions with clear resolution criteria C) Real-money incentives, questions about company stock price, participation limited to senior management D) Anonymous trading, purely random question selection, no executive sponsorship

Answer: B Explanation: The chapter identifies several features that contribute to successful corporate markets: voluntary participation (mandatory participation leads to random trading), short-to-medium time horizons (1-6 months generate the most engagement), cross-functional questions (these aggregate information from multiple departments), and clear resolution criteria (to avoid ambiguity). Executive sponsorship is also important.

Question 10

The Replication Markets project found that prediction markets:

A) Were less accurate than expert surveys at identifying non-replicating studies B) Were better calibrated than surveys of the same experts, with a Brier score of approximately 0.17 vs. 0.21 for surveys C) Could identify non-replicating studies with 95% accuracy D) Were useful only for psychology studies, not for other sciences

Answer: B Explanation: The Replication Markets project found that prediction markets outperformed expert surveys even when composed of the same experts, suggesting that the market mechanism itself adds value beyond opinion collection. Markets achieved a Brier score of approximately 0.17 vs. 0.21 for surveys, and correctly identified non-replicating studies roughly 70% of the time.

Question 11

Weather derivatives function as prediction markets on weather outcomes. The Ornstein-Uhlenbeck process used to model temperature dynamics has the form $dX_t = -\kappa X_t \, dt + dW_t$. What does the parameter $\kappa$ represent?

A) The seasonal mean temperature B) The mean-reversion speed of temperature anomalies C) The volatility of temperature shocks D) The correlation between temperature and precipitation

Answer: B Explanation: In the Ornstein-Uhlenbeck temperature model, $\kappa$ is the mean-reversion parameter that captures the tendency of temperature anomalies (deviations from seasonal norms) to decay over time. Higher $\kappa$ means anomalies dissipate faster, while lower $\kappa$ allows anomalies to persist longer.

Question 12

The S-curve model of technology adoption is $N(t) = \frac{K}{1 + e^{-r(t-t_0)}}$. At the inflection point $t = t_0$, what is the adoption level?

A) $0$ B) $K/4$ C) $K/2$ D) $K$

Answer: C Explanation: At $t = t_0$, the exponent is $-r(t_0 - t_0) = 0$, so $N(t_0) = K / (1 + e^0) = K / (1 + 1) = K/2$. The inflection point represents the moment when exactly half the potential adopters have adopted, and the adoption rate begins to slow.

Question 13

For a prediction market using LMSR with cost function $C(\mathbf{q}) = b \ln(\sum e^{q_i/b})$, what is the maximum loss to the market maker in a binary market?

A) $b$ B) $b \ln 2$ C) $2b$ D) $b^2$

Answer: B Explanation: The maximum loss to the LMSR market maker in a market with $n$ outcomes is $b \ln(n)$. For a binary market, $n = 2$, so the maximum loss is $b \ln 2 \approx 0.693b$. This is a key design parameter: choosing $b$ determines how much the market operator is willing to subsidize the market.

Question 14

Which of the following represents the strongest evidence for the value of prediction markets over traditional forecasting?

A) Prediction markets are cheaper to operate than hiring expert analysts B) Prediction markets work in every domain that has been tested C) Meta-analyses show prediction markets outperform polls in approximately 75% of elections D) Prediction markets never produce biased forecasts

Answer: C Explanation: The meta-analytical finding that prediction markets outperform polls in about 75% of elections is strong, concrete evidence. Options A, B, and D are not supported by the evidence: markets are not always cheaper, they do not work in every domain (they struggle in domains with no distributed information), and they can exhibit biases (like the favorite-longshot bias in sports betting).

Question 15

In conditional prediction markets for policy analysis, the estimated policy effect is $\hat{\tau} = E[Y | X=1] - E[Y | X=0]$. A fundamental challenge with this approach is:

A) The math is too complex for policymakers to understand B) Both conditional markets may not attract sufficient liquidity C) The approach only works for binary outcomes D) Prices are always biased toward zero policy effect

Answer: B Explanation: The main practical challenge with conditional prediction markets is that they split an already potentially thin market into two thinner markets. Each conditional market ("outcome given policy A" and "outcome given policy B") must independently attract enough informed participation to produce meaningful prices. This liquidity challenge is the primary reason conditional markets have seen limited practical deployment.

Question 16

The favorite-longshot bias in sports betting markets refers to:

A) Favorites winning more often than the odds suggest B) Longshots being slightly overpriced and favorites slightly underpriced relative to true probabilities C) Markets becoming more efficient as the event start time approaches D) Professional bettors preferring to bet on longshots

Answer: B Explanation: The favorite-longshot bias is the well-documented tendency for longshots (unlikely outcomes) to be slightly overpriced and favorites to be slightly underpriced. This means that betting on favorites is slightly more profitable (or less unprofitable) than betting on longshots. The bias is smaller in exchange markets (like Betfair) than in traditional bookmaker markets.

Question 17

When designing a corporate prediction market pilot, which sequence of phases is recommended?

A) Scale, Design, Evaluate, Pilot, Assess B) Assess, Design, Pilot, Evaluate, Scale C) Design, Pilot, Scale, Assess, Evaluate D) Pilot, Assess, Design, Scale, Evaluate

Answer: B Explanation: The chapter recommends a five-phase implementation framework: Assessment (identifying candidate questions and cultural readiness), Design (selecting mechanism and incentive structure), Pilot (small-scale launch with 50-200 participants), Evaluation (measuring accuracy and organizational impact), and Scaling (expanding participant pool and integrating with decisions).

Question 18

The breakeven inflation rate is calculated as $r_{\text{nominal}} - r_{\text{TIPS}}$. This measure can be distorted by:

A) Changes in the federal funds rate B) Liquidity premia and inflation risk premia C) Corporate bond spreads D) Stock market volatility

Answer: B Explanation: TIPS breakeven inflation rates can be distorted by two main factors: liquidity premia (TIPS are less liquid than nominal Treasuries, so TIPS yields include a liquidity premium that inflates the breakeven) and inflation risk premia (investors demand compensation for bearing inflation risk, which is embedded in nominal yields). During the 2008 crisis, liquidity effects caused breakevens to turn negative despite no expectation of sustained deflation.

Question 19

Which domain has the HIGHEST prediction market trading volume globally?

A) Election forecasting B) Scientific replicability C) Sports betting D) Interest rate futures

Answer: C Explanation: Sports betting is by far the largest prediction market domain by volume, with global revenue exceeding $200 billion annually. This dwarfs all other prediction market domains combined, including election markets and financial futures markets (though interest rate futures volumes are also very large in notional terms).

Question 20

A key lesson from the pandemic forecasting experience was:

A) Prediction markets are always superior to epidemiological models B) Only domain experts should participate in health-related prediction markets C) The best forecasts came from mechanisms combining domain expertise with forecasting skill D) Intrinsic motivation is insufficient to drive participation during a crisis

Answer: C Explanation: The pandemic experience showed that the best forecasts came from mechanisms that combined domain expertise (epidemiologists, virologists) with forecasting skill (experienced superforecasters, quantitative analysts). Pure crowd wisdom without expert input performed poorly on technical questions, but pure expert opinion without aggregation mechanisms also had weaknesses.

Question 21

Catastrophe bonds (cat bonds) function as prediction markets because:

A) They pay out based on the occurrence of natural disasters, with spreads reflecting implied disaster probabilities B) They are traded on prediction market exchanges C) They use LMSR pricing mechanisms D) They aggregate crowd forecasts of disaster timing

Answer: A Explanation: Cat bonds pay out when a specified catastrophic event occurs (e.g., hurricane above a certain intensity). The spread on a cat bond reflects the market's assessment of the probability and severity of the triggering catastrophe. The expected loss component provides an implicit probability forecast, making cat bonds function as a form of prediction market for natural disaster risk.

Question 22

Microsoft's "bug markets" for predicting software quality revealed that:

A) Developers refused to participate because it exposed code quality issues B) Markets were perfect substitutes for quality assurance processes C) Developers had genuine private information about code quality, and markets helped surface it, though incentive tensions existed D) Bug predictions were less accurate than random chance

Answer: C Explanation: Microsoft's bug prediction markets showed that developers who worked on specific components had genuine private information about code quality, and markets helped surface this information. However, there were incentive tensions: developers who knew their code was buggy could profit, but revealing this through trading might reflect poorly on their work. The markets worked best as a complement to, not replacement for, existing QA processes.

Question 23

The maximum loss to an LMSR market maker in a market with 4 outcomes and $b = 200$ is:

A) $200 \ln 2 \approx 138.6$ B) $200 \ln 4 \approx 277.3$ C) $200 \times 4 = 800$ D) $200$

Answer: B Explanation: The maximum loss formula is $b \ln(n)$ where $n$ is the number of outcomes. With $n = 4$ and $b = 200$: maximum loss = $200 \ln 4 = 200 \times 1.386 \approx 277.3$.

Question 24

In the context of prediction market adoption, the "chicken-and-egg problem" refers to:

A) The difficulty of determining whether prediction markets or polls came first B) Markets needing liquidity to be useful, but participants only joining useful markets C) The challenge of choosing between real money and play money incentives D) The conflict between market maker profits and participant welfare

Answer: B Explanation: The chicken-and-egg problem is one of the fundamental barriers to prediction market adoption. Markets need sufficient liquidity (trading volume and participant diversity) to produce meaningful prices, but potential participants are reluctant to join markets that don't yet have enough activity to be useful. This creates a bootstrapping challenge that many prediction markets fail to overcome.

Question 25

A prediction market for clinical trial outcomes is most useful when:

A) The trial outcome is completely unknown to everyone B) Some participants have relevant information (mechanism of action data, biomarker results) that, when aggregated, produces useful forecasts C) The pharmaceutical company already has high-quality internal forecasts D) Only financial analysts participate in the market

Answer: B Explanation: Markets are most valuable when there is genuinely distributed information that can be aggregated. For clinical trials, relevant information includes publication records, mechanism of action data, biomarker results, and insights from researchers familiar with the therapeutic area. When no one truly knows whether a drug will work (pure scientific uncertainty), markets aggregate ignorance rather than knowledge. The sweet spot is when some participants have partial, complementary information.