Quiz: Chapter 26

Instructions: Select the best answer for each question. Some questions may have more than one plausible answer; choose the most complete and accurate one.

Question 1

What is the primary purpose of backtesting a prediction market strategy?

A) To prove that the strategy will be profitable in the future B) To test whether historical evidence is consistent with the strategy having an edge C) To find the optimal parameters that maximize historical returns D) To determine the exact return the strategy will produce

Answer: B Backtesting is hypothesis testing, not profit demonstration. It evaluates whether historical evidence supports the hypothesis that a strategy has a genuine edge, while acknowledging that past performance does not guarantee future results.

Question 2

Which of the following is an example of lookahead bias in a prediction market backtest?

A) Using a 20-day rolling average to generate trading signals B) Filtering the backtest universe to include only markets that resolved YES C) Applying a 2-cent slippage assumption to all trades D) Using walk-forward analysis with 6-month training windows

Answer: B Filtering by resolution outcome uses information (the final resolution) that was not available at the time trading decisions would have been made. This is classic lookahead bias.

Question 3

Survivorship bias in prediction market backtesting most commonly occurs when:

A) You use too many parameters in your strategy B) You exclude cancelled, delisted, or low-liquidity markets from your dataset C) You fail to account for transaction costs D) You test the strategy on too short a time period

Answer: B Survivorship bias occurs when the dataset only includes markets that "survived" (were not cancelled, maintained liquidity, etc.), creating a biased sample that may overstate strategy performance.

Question 4

A strategy has been optimized on historical data with 12 free parameters. Which is the greatest concern?

A) The strategy will trade too frequently B) The strategy is likely overfit and will perform poorly out of sample C) The strategy will have too high a win rate D) The strategy will require too much capital

Answer: B Each free parameter adds a degree of freedom that can fit noise in the historical data. Twelve parameters is likely far more than the underlying economic rationale can support, making overfitting the primary concern.

Question 5

In the event-driven backtesting architecture, which component is responsible for preventing lookahead bias?

A) The Strategy module B) The Portfolio module C) The Data Handler D) The Execution Simulator

Answer: C The Data Handler feeds data one event at a time and only provides access to data that has already been emitted through the get_latest() method. This structural constraint makes lookahead impossible without deliberately bypassing the API.

Question 6

What is the key advantage of event-driven backtesting over vectorized backtesting?

A) It is significantly faster B) It naturally prevents lookahead bias and supports realistic execution modeling C) It requires less code to implement D) It produces more accurate return calculations

Answer: B Event-driven backtesting processes data chronologically, one event at a time, which structurally prevents lookahead and naturally supports realistic execution modeling including partial fills, slippage, and portfolio state tracking.

Question 7

The square-root market impact model is given by $C_{impact} = \sigma \cdot \beta \cdot \sqrt{Q/V}$. What does this model imply about the relationship between order size and impact cost?

A) Impact cost increases linearly with order size B) Impact cost increases with the square root of order size relative to volume C) Impact cost decreases as order size increases D) Impact cost is independent of order size

Answer: B The square-root model implies that doubling your order size increases market impact by a factor of $\sqrt{2} \approx 1.41$, not by a factor of 2. Larger orders have higher total impact but lower marginal impact per contract.

Question 8

A prediction market trade involves buying at $0.55 (ask) when the bid is $0.49. What is the spread cost of this trade relative to the mid-price?

A) $0.03 per contract B) $0.06 per contract C) $0.055 per contract D) $0.01 per contract

Answer: A The mid-price is $(0.55 + 0.49) / 2 = 0.52$. The spread cost for a buy is the difference between the ask and the mid: $0.55 - 0.52 = 0.03$ per contract.

Question 9

What is the opportunity cost of holding a prediction market position worth $0.40 per contract for 180 days at a risk-free rate of 5%?

A) $0.0099 per contract B) $0.0200 per contract C) $0.0400 per contract D) $0.0010 per contract

Answer: A Opportunity cost = $P \times r_f \times T = 0.40 \times 0.05 \times (180/365) = 0.40 \times 0.05 \times 0.493 = 0.00986 \approx 0.0099$ per contract.

Question 10

In walk-forward backtesting, the final performance metric is computed from:

A) The in-sample training periods only B) The out-of-sample test periods only C) The combined in-sample and out-of-sample periods D) The best-performing single window

Answer: B The final performance in walk-forward analysis is the concatenation of all out-of-sample test periods. Every data point in the final track record was genuinely out-of-sample at the time parameters were selected.

Question 11

What does it mean when optimal parameters change dramatically from one walk-forward window to the next?

A) The strategy is robust across different market regimes B) The strategy is likely fitting noise rather than capturing a genuine signal C) The training window is too large D) The test window is too small

Answer: B Unstable parameters across walk-forward windows suggest the optimization is fitting noise that is specific to each training period. A genuine, persistent market inefficiency would be captured by relatively stable parameters.

Question 12

Which of the following Sharpe ratio values is most likely to indicate a backtesting error rather than genuine alpha in a prediction market strategy?

A) 0.8 B) 1.5 C) 2.5 D) 5.0

Answer: D A Sharpe ratio above 3.0 is suspicious for prediction market strategies, and a Sharpe of 5.0 almost certainly indicates a backtesting error such as lookahead bias, unrealistic fill assumptions, or overfitting.

Question 13

What is the Sortino ratio, and how does it differ from the Sharpe ratio?

A) The Sortino ratio uses maximum drawdown instead of volatility in the denominator B) The Sortino ratio uses only downside volatility in the denominator, penalizing only losses C) The Sortino ratio uses the risk-free rate in the numerator instead of excess return D) The Sortino ratio is the Sharpe ratio divided by the number of trades

Answer: B The Sortino ratio replaces total volatility with downside volatility (the standard deviation of negative returns only). This makes it a better metric for strategies with asymmetric return distributions, as it does not penalize upside volatility.

Question 14

A strategy has a win rate of 40%, average win of $0.15, and average loss of $0.06. What is the expectancy per trade?

A) $0.024 B) -$0.024 C) $0.060 D) $0.096

Answer: A $E = W \times \bar{G} - (1-W) \times \bar{L} = 0.40 \times 0.15 - 0.60 \times 0.06 = 0.060 - 0.036 = 0.024$

Question 15

What is the profit factor, and what does a value of 1.0 indicate?

A) The ratio of average win to average loss; 1.0 means the strategy breaks even B) The ratio of gross profits to gross losses; 1.0 means the strategy breaks even C) The ratio of winning trades to losing trades; 1.0 means equal numbers of wins and losses D) The ratio of total return to maximum drawdown; 1.0 means moderate performance

Answer: B The profit factor is the sum of all winning trade profits divided by the absolute sum of all losing trade losses. A value of 1.0 means total profits exactly equal total losses (break-even before costs).

Question 16

A permutation test for strategy significance involves:

A) Testing different parameter permutations and selecting the best B) Randomly shuffling the temporal relationship between signals and returns to create a null distribution C) Permuting the order of trades to check for sequence dependence D) Running the strategy on different subsets of markets

Answer: B A permutation test randomly reassigns signals to returns (breaking the temporal mapping), recalculates the performance metric, and repeats many times. This creates a distribution of performance under the null hypothesis that the signal has no predictive power.

Question 17

You test 20 strategies on the same dataset. Using Bonferroni correction at $\alpha = 0.05$, what p-value threshold must an individual strategy exceed to be considered significant?

A) 0.05 B) 0.025 C) 0.0025 D) 0.01

Answer: C Bonferroni correction divides the significance level by the number of tests: $\alpha_{adjusted} = 0.05 / 20 = 0.0025$.

Question 18

The minimum number of observations needed to detect a Sharpe ratio of 1.0 with 80% power at 5% significance is approximately:

A) 100 B) 500 C) 1,500 D) 5,000

Answer: C Using the formula $n \geq ((z_\alpha + z_\beta) / (S/\sqrt{252}))^2$ with $z_\alpha = 1.645$, $z_\beta = 0.842$, and $S = 1.0$: $n \geq ((1.645 + 0.842) / (1/\sqrt{252}))^2 \approx 1,557$.

Question 19

During the "Shadow Mode" phase of paper trading, what should you primarily check?

A) Whether the strategy is profitable B) Whether signals are generated at the expected frequency and the data feed is reliable C) Whether execution costs match assumptions D) Whether the maximum drawdown is acceptable

Answer: B Shadow Mode runs the strategy without placing orders. The primary purpose is to verify that signals are generated as expected and that the data infrastructure works correctly, before risking any capital.

Question 20

Which of the following is NOT a valid reason to stop a live prediction market strategy?

A) The drawdown exceeds 1.5x the worst backtest drawdown B) The rolling 30-day Sharpe falls below -1.0 C) The strategy has not made a trade in 48 hours D) The platform changes its fee structure

Answer: C A 48-hour period without trades may be completely normal for many prediction market strategies, especially those that wait for specific conditions. The other three options (excessive drawdown, negative Sharpe, and structural changes) are all valid stopping criteria.

Question 21

What is the fundamental difference between VaR and CVaR (Expected Shortfall)?

A) VaR measures the average loss; CVaR measures the worst-case loss B) VaR measures the loss threshold at a confidence level; CVaR measures the expected loss given that the loss exceeds VaR C) VaR is a daily metric; CVaR is a monthly metric D) VaR is computed from the normal distribution; CVaR uses the empirical distribution

Answer: B VaR answers "what is the loss level that is exceeded with probability $\alpha$?" while CVaR answers "given that the loss exceeds VaR, what is the expected loss?" CVaR is more informative about tail risk because it considers the magnitude of extreme losses, not just their probability.

Question 22

A "point-in-time database" is important for backtesting because it:

A) Stores data at regular time intervals rather than at irregular trade times B) Records data exactly as it was known at each historical moment, including revisions C) Indexes data by timestamp for fast retrieval D) Compresses historical data to reduce storage costs

Answer: B A point-in-time database captures data as it existed at each moment, including corrections and revisions that may have been applied later. This prevents the backtest from using "corrected" data that was not available at the time of the trading decision.

Question 23

When comparing anchored vs. rolling walk-forward approaches, which statement is correct?

A) Anchored walk-forward uses a fixed-size training window that slides forward B) Rolling walk-forward always starts training from the beginning of the dataset C) Anchored walk-forward grows the training set over time and adapts more slowly to regime changes D) Rolling walk-forward produces more training data than anchored walk-forward

Answer: C In anchored walk-forward, the training window always starts from the beginning of the dataset, so the training set grows larger with each step. This provides more data for training but makes the process slower to adapt to recent changes in market behavior.

Question 24

A strategy has the following characteristics: Win Rate = 85%, Average Win = $0.02, Average Loss = $0.15. What is the key risk of this strategy?

A) It does not trade frequently enough B) It has a low win rate C) Infrequent but large losses can cause significant drawdowns despite the high win rate D) The average win is too small to cover transaction costs

Answer: C This "picking up pennies in front of a steamroller" profile --- many small wins with occasional catastrophic losses --- is extremely dangerous. The expectancy is $0.85 \times 0.02 - 0.15 \times 0.15 = 0.017 - 0.0225 = -0.0055$, meaning the strategy is actually unprofitable despite the 85% win rate.

Question 25

What is the single most important architectural decision that prevents lookahead bias in a backtesting framework?

A) Using abstract base classes for all components B) Feeding data to the strategy one event at a time in chronological order C) Computing all metrics after the backtest completes D) Using Python rather than a spreadsheet for implementation

Answer: B The event-driven data flow --- where the strategy receives market data one event at a time and can only access data that has already been emitted --- is the structural safeguard against lookahead bias. This makes it impossible for the strategy to "see" future data without explicitly breaking the framework's API.