Chapter 1 Exercises: The Landscape of AI Engineering

These exercises are organized into five parts of increasing complexity. Work through them sequentially, or jump to the sections most relevant to your learning goals.

Part A: Conceptual Questions

Exercise 1.1 --- Historical Epochs Identify the four major epochs of AI described in this chapter. For each epoch, name one key achievement and one key limitation that motivated the transition to the next era.

Exercise 1.2 --- Symbolic vs. Statistical Explain the fundamental difference between symbolic AI and machine learning in your own words. Give a concrete example of a task where each approach might be preferred, and justify your choices.

Exercise 1.3 --- The Three Ingredients The deep learning revolution required three ingredients: data, compute, and algorithms. Rank these three in order of importance for the initial breakthrough (circa 2012), and separately for the current era of foundation models. Justify your ranking in each case.

Exercise 1.4 --- Foundation Model Paradigm Shift Describe the paradigm shift from traditional ML to the foundation model approach. What are two advantages and two disadvantages of the foundation model paradigm from an AI engineer's perspective?

Exercise 1.5 --- Subfield Identification For each of the following systems, identify which subfields of AI are involved and explain the role of each: a) A self-driving car navigating a city b) A chatbot that answers customer support questions c) A system that generates music in the style of a given artist d) A robotic arm that sorts packages in a warehouse

Exercise 1.6 --- AI Stack Layers Draw (or describe in text) the layers of the modern AI stack. For a company building a fraud detection system for credit card transactions, give a specific example of a tool or technology at each layer.

Exercise 1.7 --- Role Distinctions A startup has a team of five people building an AI-powered medical imaging product. Describe how the responsibilities of an AI engineer, a data scientist, and a software engineer on this team would differ and overlap.

Exercise 1.8 --- AI Winters What factors have historically caused "AI winters"? Do you think the current era of AI is susceptible to another AI winter? Provide at least three arguments for and three arguments against this possibility.

Part B: Calculations and Quantitative Reasoning

Exercise 1.9 --- Speedup Estimation A training job takes 48 hours on a single CPU. Using the rule-of-thumb speedup factor of 10--100x for GPUs, estimate the range of training times on a single GPU. If you need the model trained within 2 hours, what is the minimum speedup factor required, and what might you do to achieve it?

Exercise 1.10 --- Model Parameter Counting A simple neural network has an input layer of size 784, one hidden layer of size 256, and an output layer of size 10. Calculate the total number of trainable parameters (weights and biases) in this network.

$$\text{Parameters} = (n_{\text{in}} \times n_{\text{hidden}} + n_{\text{hidden}}) + (n_{\text{hidden}} \times n_{\text{out}} + n_{\text{out}})$$

Exercise 1.11 --- Foundation Model Scaling GPT-2 (2019) had 1.5 billion parameters. GPT-3 (2020) had 175 billion parameters. Calculate the growth factor. If this growth factor continued for two more generations (released one year apart), estimate the parameter count for each. Discuss whether this growth rate is sustainable.

Exercise 1.12 --- Data Requirements A supervised classification model needs approximately 1,000 labeled examples per class to achieve acceptable performance. If you are building a document classifier with 50 categories and labeling each document takes an average of 2 minutes, estimate: a) The total number of labeled examples needed b) The total labeling time in person-hours c) The cost at $25/hour for labelers

Exercise 1.13 --- Inference Latency Budget A movie recommendation system must return results within 200ms of a user request. The system has four stages: feature retrieval (50ms), candidate generation (Xms), ranking model inference (80ms), and post-processing (20ms). What is the maximum time allowed for candidate generation? If the candidate generation model processes 1,000 items per second, how many candidate movies can it evaluate within this time budget?

Exercise 1.14 --- Cost Comparison A company runs inference for a text classification model. Option A uses a GPU server costing $3.00/hour that can handle 500 requests/second. Option B uses a CPU server costing $0.50/hour that can handle 50 requests/second. If the company processes 1 million requests per day, calculate: a) The number of servers needed for each option b) The daily cost for each option c) Which option is more cost-effective?

Part C: Programming Exercises

Exercise 1.15 --- AI Timeline Extension Extend the example-01-ai-timeline.py code to include at least five additional milestones from the history of AI that were not included in the original timeline. Customize the visualization with different colors for different categories of milestone (theoretical, engineering, commercial).

Exercise 1.16 --- Parameter Counter Write a Python function using NumPy that takes a list of layer sizes (e.g., [784, 256, 128, 10]) and returns the total number of trainable parameters in a fully connected neural network with those layer dimensions.

import numpy as np

def count_parameters(layer_sizes: list[int]) -> dict[str, int]:
    """Count the trainable parameters in a fully connected network.

    Args:
        layer_sizes: List of integers representing the number of
            neurons in each layer, from input to output.

    Returns:
        Dictionary with 'weights', 'biases', and 'total' counts.
    """
    # Your implementation here
    pass

Exercise 1.17 --- Simple Linear Regression from Scratch Using only NumPy, implement a simple linear regression model that fits a line $y = wx + b$ to a dataset. Your implementation should: a) Generate synthetic data: $y = 3x + 7 + \epsilon$ where $\epsilon \sim \mathcal{N}(0, 1)$ b) Implement gradient descent to learn $w$ and $b$ c) Plot the data and the fitted line d) Print the learned values of $w$ and $b$

Exercise 1.18 --- Ecosystem Audit Write a Python script that audits your local AI/ML development environment. The script should: a) Check which of the following packages are installed: numpy, scipy, pandas, scikit-learn, matplotlib, torch, tensorflow, transformers b) Report the version of each installed package c) Check for GPU availability (CUDA for NVIDIA) d) Report system memory and CPU information e) Output a formatted summary table

Exercise 1.19 --- Decision Boundary Visualization Using NumPy and matplotlib, create a visualization that shows the decision boundaries of three different classifiers on a 2D dataset: a) A simple threshold rule (symbolic/rule-based approach) b) A linear decision boundary (logistic regression-style) c) A non-linear decision boundary (polynomial features)

Generate a synthetic 2D dataset with two classes and plot all three decision boundaries side by side, illustrating how different approaches handle the same classification problem.

Exercise 1.20 --- Forward Pass Implementation Implement a complete forward pass for a three-layer neural network using only NumPy. Your implementation should include: a) Weight initialization (random normal, scaled by layer size) b) ReLU activation for hidden layers c) Softmax activation for the output layer d) A function to compute cross-entropy loss

import numpy as np

def initialize_network(
    layer_sizes: list[int],
    seed: int = 42,
) -> list[dict[str, np.ndarray]]:
    """Initialize network weights and biases.

    Args:
        layer_sizes: List of layer dimensions.
        seed: Random seed for reproducibility.

    Returns:
        List of dictionaries, each containing 'W' and 'b'.
    """
    # Your implementation here
    pass

def forward(
    x: np.ndarray,
    parameters: list[dict[str, np.ndarray]],
) -> np.ndarray:
    """Compute forward pass through the network.

    Args:
        x: Input array of shape (n_features,).
        parameters: Network parameters from initialize_network.

    Returns:
        Output probabilities of shape (n_classes,).
    """
    # Your implementation here
    pass

Part D: Analysis and Critical Thinking

Exercise 1.21 --- Stack Trade-offs Consider a startup building an AI-powered customer service chatbot. They have a team of three engineers and a budget of $5,000/month for cloud infrastructure. Recommend a specific technology at each layer of the AI stack, justifying your choices based on the team size and budget constraints. What trade-offs are you making?

Exercise 1.22 --- Build vs. Buy For each of the following scenarios, argue whether the company should build a custom AI solution or use an off-the-shelf service (e.g., a cloud AI API). Consider cost, performance, data privacy, competitive advantage, and maintenance burden. a) A bank needs a fraud detection system for credit card transactions. b) A social media company wants to moderate content for policy violations. c) A law firm wants to extract key clauses from legal contracts. d) A retail chain wants to forecast demand for perishable goods.

Exercise 1.23 --- Failure Mode Analysis Choose one of the following AI systems and conduct a failure mode analysis. For each failure mode, describe the potential impact and propose a mitigation strategy. a) An AI system that screens job applications b) An AI system that recommends medical treatments c) An AI system that prices insurance policies d) An AI system that generates news articles

Exercise 1.24 --- Paradigm Comparison The chapter describes the evolution from symbolic AI to machine learning to deep learning to foundation models. For the task of language translation, describe how each paradigm would approach the problem. What are the strengths and weaknesses of each approach? Which approach would you recommend today, and under what circumstances might an older paradigm still be preferable?

Exercise 1.25 --- Ethical Analysis A healthcare company wants to build an AI system that predicts which patients are at high risk of developing a chronic disease, so that preventive interventions can be offered early. Analyze this scenario from the perspective of: a) Fairness: What biases might be present in the training data, and how could they affect different populations? b) Transparency: Should the system explain its predictions to patients? To doctors? c) Privacy: What data is needed, and how should it be handled? d) Accountability: Who is responsible if the system makes a wrong prediction?

Exercise 1.26 --- Technology Radar Create a "technology radar" for AI engineering. Categorize the following technologies into one of four rings: Adopt (proven, use now), Trial (worth pursuing, not yet proven at scale), Assess (worth exploring, not yet proven), Hold (proceed with caution): - PyTorch - TensorFlow - JAX - LangChain - Hugging Face Transformers - ONNX Runtime - MLflow - DVC - Kubernetes for ML - AutoML platforms

Justify each placement with a brief rationale.

Part E: Research and Exploration

Exercise 1.27 --- Historical Deep Dive Choose one of the following historical AI systems and write a 500-word analysis of its approach, achievements, and limitations. Discuss how the techniques it used relate to modern AI engineering practices. a) MYCIN (medical diagnosis expert system) b) Deep Blue (chess-playing system) c) Watson (Jeopardy!-playing system) d) AlphaGo (Go-playing system)

Exercise 1.28 --- Industry Case Study Select an industry (healthcare, finance, automotive, retail, or another of your choice) and research how AI engineering practices are applied in that sector. Your analysis should cover: a) Three specific AI applications currently deployed in the industry b) The key technical challenges unique to that industry c) Regulatory and ethical considerations d) The typical AI team structure and roles e) Future trends and opportunities

Exercise 1.29 --- Open Source Investigation Explore the GitHub repositories of two major AI/ML frameworks (e.g., PyTorch, scikit-learn, Hugging Face Transformers, LangChain). For each repository: a) How many contributors does it have? b) How frequently is it updated? c) What is its license? d) What does the project structure look like? e) How is the community managed (issues, pull requests, discussions)?

Write a comparison that highlights the different development philosophies and community structures.

Exercise 1.30 --- Future Forecast Based on current trends in AI research and engineering, write a thoughtful forecast of where AI engineering will be in five years. Address: a) Which current techniques will remain dominant? b) What new capabilities might emerge? c) How will the role of the AI engineer change? d) What new ethical and societal challenges will arise? e) What skills should an aspiring AI engineer prioritize developing now?

Support your forecast with references to current research trends, industry analyses, or expert opinions.