Chapter 27 Quiz

Instructions: Select the best answer for each question. Answers are provided at the end.

Question 1

What is the primary purpose of a scikit-learn Pipeline?

A) To speed up model training by parallelizing preprocessing steps B) To encapsulate preprocessing and modeling into a single, reproducible object that can be fitted and serialized as a unit C) To automatically select the best hyperparameters for a model D) To visualize the data transformation process

Question 2

In a scikit-learn Pipeline, which of the following is true about the intermediate steps (all steps except the last)?

A) They must implement fit and predict methods B) They must implement fit and transform methods C) They must implement only a transform method D) They can implement any combination of methods

Question 3

What does ColumnTransformer allow you to do?

A) Transform all columns with the same transformation B) Apply different transformations to different subsets of columns C) Automatically detect column types and apply appropriate transformations D) Transform columns from one data type to another

Question 4

What is "training-serving skew"?

A) The difference in training time between two model versions B) The phenomenon where a model is overfit on training data C) A discrepancy between how features are computed during training versus serving, leading to different model behavior D) The delay between training a model and deploying it

Question 5

What does "point-in-time correctness" in a feature store ensure?

A) Features are computed at exactly the right time B) Features used for each training example reflect only information available at the time of that example, preventing data leakage C) All features are timestamped with the current time D) Features are sorted chronologically before being used

Question 6

In the context of feature stores, what is the difference between an "offline store" and an "online store"?

A) The offline store is for backup and the online store is for primary storage B) The offline store holds historical features for batch training; the online store holds latest features for low-latency real-time serving C) The offline store is disconnected from the internet; the online store is cloud-based D) The offline store uses SQL databases; the online store uses NoSQL databases

Question 7

Which pandas function is most efficient for performing point-in-time joins?

A) pd.merge with how='left' B) pd.merge_asof with direction='backward' C) pd.concat followed by groupby D) pd.pivot_table with temporal indices

Question 8

In MLflow, what is the purpose of the "Model Registry"?

A) To store training data versions B) To manage model versions, lifecycle stages (staging, production, archived), and facilitate model deployment C) To register new machine learning algorithms D) To track user permissions for model access

Question 9

Which of the following is NOT a valid model lifecycle stage in the MLflow Model Registry?

A) Staging B) Production C) Testing D) Archived

Question 10

What is concept drift?

A) A change in the distribution of input features B) A change in the relationship between input features and the target variable C) A gradual decrease in model file size D) A shift in the team's understanding of the problem

Question 11

What is the Population Stability Index (PSI) used for?

A) Measuring how stable a model's predictions are over time B) Quantifying how much a distribution has shifted relative to a reference distribution C) Measuring the stability of a population's preferences D) Computing the confidence interval of model predictions

Question 12

According to common interpretation guidelines, a PSI value of 0.3 indicates:

A) No significant drift B) Moderate drift requiring investigation C) Significant drift requiring action D) The distributions are identical

Question 13

What does the Kolmogorov-Smirnov (KS) test measure?

A) The correlation between two distributions B) The maximum difference between the empirical CDFs of two distributions C) The mean difference between two distributions D) The variance ratio of two distributions

Question 14

In the context of model monitoring, what is "data drift" (covariate shift)?

A) The training data being deleted accidentally B) The distribution of input features changing, even if the underlying relationship between features and target stays the same C) The model's parameters changing over time D) The deployment infrastructure drifting away from specifications

Question 15

Which of the following is an advantage of batch prediction serving over real-time serving?

A) Lower latency for individual predictions B) Better ability to respond to market changes instantly C) Simpler infrastructure, ability to precompute predictions, and higher throughput D) More accurate predictions

Question 16

In the FastAPI prediction server described in the chapter, what is the purpose of the /health endpoint?

A) To provide medical advice B) To check if the server is running, report the loaded model version, and provide basic operational metrics C) To trigger model retraining D) To perform model validation

Question 17

What is the recommended total end-to-end latency budget for a prediction market trading model?

A) Less than 1 millisecond B) Less than 100 milliseconds C) Less than 10 seconds D) Latency does not matter for prediction markets

Question 18

Why is ML CI/CD different from traditional software CI/CD?

A) ML code is more complex than traditional code B) ML CI/CD must additionally validate model quality, data schemas, and ensure new models do not regress on key metrics C) Traditional CI/CD does not use automated testing D) ML CI/CD requires more compute resources

Question 19

Which type of test verifies that a trained model meets minimum quality thresholds (e.g., Brier score < 0.25)?

A) Unit test B) Integration test C) Model quality test D) Smoke test

Question 20

What is the purpose of data validation with Pandera in an ML pipeline?

A) To encrypt sensitive data B) To define and enforce expected data schemas, value ranges, and quality constraints C) To compress data for faster processing D) To convert data between formats

Question 21

Which of the following strategies does NOT help with ML reproducibility?

A) Pinning all library versions in requirements.txt B) Setting random seeds for all random number generators C) Using the latest version of every library without version pinning D) Versioning the training data with hashes

Question 22

What is a "validation gate" in the context of automated training pipelines?

A) A physical security barrier in the data center B) An automated check that a candidate model must pass before being promoted to production (e.g., Brier score below a threshold) C) A step that validates the input data format D) A gate that controls user access to the model

Question 23

In the MLOps maturity model described in the chapter, what characterizes Level 3 (Full Automation with Monitoring)?

A) Everything is manual B) Training pipelines are automated but deployment is manual C) The system detects drift, triggers retraining, validates new models, and deploys automatically, with humans overseeing but rarely intervening D) Only code testing is automated

Question 24

Why is graceful degradation important in a prediction serving system?

A) It makes the system look more professional B) It ensures the system continues to provide reasonable predictions even when the primary model fails, preventing cascading failures in downstream trading systems C) It improves model accuracy D) It reduces infrastructure costs

Question 25

What is "model lineage" and why is it important?

A) The family tree of a model's developers B) A complete record of which data, code, features, and configuration produced a specific model version, essential for debugging, auditing, and regulatory compliance C) The evolutionary history of a model architecture D) The list of all users who have accessed a model

Answers

  1. B --- A Pipeline encapsulates the full preprocessing and modeling workflow into a single object that can be fitted, serialized, and deployed as a unit.

  2. B --- All intermediate steps must implement fit and transform methods. Only the last step can optionally implement predict or predict_proba.

  3. B --- ColumnTransformer applies different transformations to different subsets of columns, enabling heterogeneous preprocessing.

  4. C --- Training-serving skew occurs when features are computed differently during training and serving, causing the model to behave differently in production than in training.

  5. B --- Point-in-time correctness ensures that features for each training example use only information available at that example's timestamp, preventing data leakage from the future.

  6. B --- The offline store holds historical features for batch training; the online store holds the latest feature values for low-latency serving.

  7. B --- pd.merge_asof with direction='backward' efficiently performs point-in-time joins by matching each row to the most recent prior feature value.

  8. B --- The Model Registry manages model versions, lifecycle stages (staging, production, archived), and deployment workflows.

  9. C --- "Testing" is not a valid MLflow Model Registry stage. Valid stages are None, Staging, Production, and Archived.

  10. B --- Concept drift is a change in the relationship between input features and the target variable, meaning the model's learned patterns no longer apply.

  11. B --- PSI quantifies how much a distribution has shifted relative to a reference distribution, commonly used for monitoring drift.

  12. C --- A PSI of 0.3 exceeds the 0.25 threshold for significant drift, indicating action is required.

  13. B --- The KS test computes the maximum difference between the empirical cumulative distribution functions of two samples.

  14. B --- Data drift (covariate shift) means the distribution of input features has changed, even if the underlying relationship between features and target remains the same.

  15. C --- Batch serving has simpler infrastructure, can precompute predictions, and achieves higher throughput, though at the cost of higher latency for individual predictions.

  16. B --- The health endpoint verifies the server is operational, reports the loaded model version, and provides basic operational metrics like request count and average latency.

  17. B --- Less than 100 milliseconds end-to-end is the recommended budget for prediction market trading models.

  18. B --- ML CI/CD must additionally validate model quality, test data schemas, and ensure new models do not regress on key performance metrics.

  19. C --- Model quality tests specifically verify that a trained model meets minimum performance thresholds.

  20. B --- Pandera defines and enforces expected data schemas, including column types, value ranges, null rates, and custom validation checks.

  21. C --- Using the latest version of every library without pinning actively harms reproducibility, as different versions may produce different results.

  22. B --- A validation gate is an automated quality check that a candidate model must pass (e.g., Brier score below a threshold) before it can be promoted to production.

  23. C --- Level 3 features full automation: the system detects drift, triggers retraining, validates new models, and deploys them automatically with minimal human intervention.

  24. B --- Graceful degradation ensures the system continues providing reasonable predictions (via fallback models or default values) when the primary model fails, preventing cascading failures in downstream systems.

  25. B --- Model lineage is a complete record of everything that produced a specific model version, critical for debugging, auditing, and meeting regulatory requirements.