Case Study 1: DataRobot vs. Hand-Coded ML — A Head-to-Head Comparison

Introduction

The tension in the Chapter 22 opening scene — NK's two-hour AutoML model versus Tom's two-week hand-coded model — is not a classroom hypothetical. Organizations face this comparison every time they evaluate whether to invest in custom ML development or use an automated platform. The answer, as with most strategic decisions, is "it depends."

This case study presents a structured head-to-head comparison across three different business problems, examining five dimensions: accuracy, speed, cost, interpretability, and governance. The goal is not to declare a winner but to develop judgment about when each approach is appropriate — and when the right answer is to use both.

The Setup

A mid-sized online retailer — 400 employees, $280 million in annual revenue, a small data science team of four — faces three distinct ML problems. The VP of Data Science, Priya Chen, decides to run each problem through both approaches in parallel: a senior data scientist builds a custom model in Python, and a business analyst builds an equivalent model using DataRobot. The results are compared on a standardized rubric.

The three problems are chosen to span a range of complexity:

1. Customer churn prediction — a well-defined binary classification problem with clean tabular data
2. Product demand forecasting — a time series problem with multiple seasonal patterns and external factors
3. Customer support ticket routing — an NLP classification problem requiring text processing and multi-class categorization

Problem 1: Customer Churn Prediction

The Data

The dataset contains 120,000 customer records with 28 features: demographics, purchase history, website behavior, support interactions, and email engagement metrics. The target variable is binary — churned or retained — based on a 90-day inactivity window. The data is clean, well-structured, and stored in a single database table with minimal missing values.

DataRobot Approach

Builder: Maria Santos, Senior Business Analyst (Marketing). No coding experience. Completed Tier 2 certification in the company's citizen data science program.

Process: Maria uploaded the CSV file, selected churned as the target, and clicked "Start." DataRobot completed its automated run in 47 minutes, training 62 models across 8 algorithm families. The platform automatically engineered 340 features from the 28 raw columns, including interaction terms, ratio features, and recency calculations.

Result: Top model (blended ensemble of LightGBM, XGBoost, and elastic net): AUC 0.847, precision at 80% recall: 0.72.

Time: 3 hours total (including data export, upload, configuration review, and results analysis).

Hand-Coded Approach

Builder: James Liu, Senior Data Scientist. Five years of experience, strong Python skills, domain knowledge of the retail business.

Process: James performed detailed EDA, examining distributions, correlations, and class imbalance (the dataset was 8% churned). He engineered 45 features manually, drawing on domain knowledge — for example, creating a "support escalation rate" feature by combining ticket severity and resolution time, and a "purchase velocity change" feature comparing recent 30-day behavior to 90-day averages. He trained and tuned logistic regression, random forest, XGBoost, and a simple neural network. He used Optuna for hyperparameter optimization and performed stratified 5-fold cross-validation.

Result: Top model (tuned XGBoost): AUC 0.862, precision at 80% recall: 0.75.

Time: 8 business days (approximately 50 hours of work, including EDA, feature engineering, model training, tuning, and documentation).

Comparison: Problem 1

Verdict: For this standard, well-defined problem with clean data, DataRobot delivers 98% of the accuracy at 6% of the time and 2% of the cost. For most business contexts, the DataRobot approach is the clear winner. The small accuracy gap does not justify the 17x time investment unless the business case is exceptionally high-stakes.

Business Insight: Churn prediction is the prototypical "AutoML sweet spot" — a well-defined binary classification problem with clean tabular data and established feature patterns. If your problem looks like this, AutoML is likely the right first approach.

Problem 2: Product Demand Forecasting

The Data

The company needs to forecast daily demand for 2,000 SKUs across 12 product categories, accounting for seasonality (weekly, monthly, annual), promotional effects, weather patterns, and economic indicators. The data spans three years and comes from four different systems: the POS system, the promotions calendar, a weather API, and an economic indicators database. Data quality is mixed — the POS data is clean, but the promotions calendar has inconsistent formatting, and weather data has gaps.

DataRobot Approach

Builder: Maria Santos (same business analyst).

Process: Maria attempted to upload the data but encountered immediate challenges. The four data sources needed to be joined and aligned — a task that required data engineering skills beyond the platform's automated data preparation capabilities. She spent two days working with the data engineering team to create a single merged dataset. Once uploaded, DataRobot's time series mode handled the problem competently but with limitations: the platform used standard time series features (lags, rolling averages, calendar features) but could not incorporate the domain-specific promotional interaction effects that the data science team knew were important. The platform also struggled with the multi-SKU forecasting structure — it treated each SKU independently rather than capturing cross-product demand patterns.

Result: Median MAPE (Mean Absolute Percentage Error) across SKUs: 18.4%. Performance varied significantly by category — from 11% MAPE for stable staple products to 34% MAPE for fashion-sensitive items.

Time: 4 days total (2 days for data preparation with engineering support, 2 days for platform work and analysis).

Hand-Coded Approach

Builder: James Liu plus one junior data scientist.

Process: James built a custom forecasting pipeline that incorporated several domain-specific elements the AutoML platform could not replicate:

• Promotional interaction modeling. James created features that captured not just whether a promotion was running but how different promotion types (BOGO, percentage-off, flash sale) affected different product categories differently. He encoded promotion cannibalization effects — when promoting Product A reduces demand for substitute Product B.
• Hierarchical forecasting. Rather than forecasting each SKU independently, James used a hierarchical approach that forecasted at the category level, product group level, and SKU level simultaneously, ensuring that lower-level forecasts were coherent with higher-level totals.
• Custom weather features. James created nonlinear weather features — not just temperature, but "deviation from seasonal norm" and "consecutive days of extreme weather" — based on analysis of historical demand-weather relationships.
• External signal integration. James incorporated Google Trends data for product categories and consumer confidence indices as leading indicators.

Result: Median MAPE: 13.2%. Performance was more consistent across categories — from 9% for stable products to 22% for fashion items.

Time: 3 weeks (approximately 120 hours across two data scientists).

Comparison: Problem 2

Verdict: The accuracy gap matters here. For a retailer managing $280 million in revenue, the 5.2 percentage point improvement in forecast accuracy translates directly to better inventory decisions — fewer stockouts, less overstock, reduced markdowns. A back-of-envelope calculation: if inventory decisions represent 40% of revenue ($112M), and a 5% improvement in forecast accuracy reduces inventory waste by 2%, the annual savings exceed $2 million. The $25,000 investment in custom development pays for itself many times over.

However, DataRobot provided value as a rapid baseline. Maria's 4-day exercise validated that demand is forecastable from the available data and established a performance baseline that the data science team could then work to exceed. The optimal workflow: use AutoML for feasibility validation, then invest in custom development for the production model.

Business Insight: When the problem involves complex data integration, domain-specific feature engineering, and multi-entity forecasting (multiple SKUs, multiple stores, multiple time horizons), AutoML platforms hit their limits. The gap between automated and custom approaches widens as problem complexity increases.

Problem 3: Customer Support Ticket Routing

The Data

The company receives approximately 3,000 support tickets per day. Each ticket needs to be routed to one of 8 specialized teams (billing, returns, product issues, shipping, account management, technical support, complaints, and general inquiries). Currently, a human dispatcher reads each ticket and assigns it manually — a process that takes 2-3 minutes per ticket and results in a 15% misrouting rate that causes delays and customer frustration.

The training data consists of 180,000 historical tickets with human-assigned labels. The text varies from one-sentence descriptions to multi-paragraph complaints with attached images. The language includes informal writing, abbreviations, typos, and occasional non-English content.

DataRobot Approach

Builder: Maria Santos.

Process: DataRobot's NLP capabilities handled the text classification problem competently. The platform automatically applied text tokenization, TF-IDF encoding, and trained several models including regularized logistic regression on text features and gradient-boosted trees on extracted text statistics. However, the platform could not incorporate the ticket metadata (customer tier, product category, purchase history) alongside the text in a unified model — it treated text classification and tabular classification as separate problems.

Result: Accuracy: 76.3% (8-class classification). Misrouting rate: 23.7% — worse than the human dispatcher.

Time: 1 day.

Hand-Coded Approach

Builder: James Liu plus the company's NLP specialist.

Process: James and his colleague built a custom model that combined text features with structured metadata:

• Text encoding. They used a pre-trained sentence transformer (all-MiniLM-L6-v2) to create dense embeddings of ticket text, capturing semantic meaning rather than just keyword frequency.
• Metadata integration. Customer tier, product category, recent purchase history, and previous ticket history were combined with text embeddings in a multi-input neural network.
• Hierarchical classification. They implemented a two-stage classifier: first routing to a high-level category (billing vs. product vs. shipping), then to a specific team within that category. This hierarchical approach reduced error propagation.
• Confidence-based routing. For tickets where the model's confidence was below a threshold (0.75), the system flagged the ticket for human review rather than auto-routing it. This hybrid approach caught the hardest cases.

Result: Accuracy on auto-routed tickets: 91.2%. With the confidence-based human review catching edge cases, effective accuracy: 94.8%. Misrouting rate: 5.2% — one-third of the human dispatcher's rate. Approximately 78% of tickets were auto-routed, with 22% flagged for human review.

Time: 5 weeks (approximately 200 hours across two specialists).

Comparison: Problem 3

Verdict: DataRobot's model is worse than the current human process — it would increase misrouting from 15% to 23.7%. This is a clear case where the AutoML approach fails to meet the minimum performance threshold. The custom model, in contrast, reduces misrouting to 5.2% and automates 78% of the routing workload, delivering substantial operational savings.

The failure is not surprising. The problem requires combining unstructured text with structured metadata, using pre-trained language models (sentence transformers), implementing hierarchical classification logic, and designing a confidence-based human-in-the-loop system. These capabilities are beyond the current state of AutoML platforms for NLP.

Business Insight: When the problem requires combining multiple data types (text + structured), using pre-trained deep learning models, or implementing custom business logic (confidence thresholds, hierarchical routing, human-in-the-loop design), no-code approaches are insufficient. These are the problems where data science expertise creates irreplaceable value.

The Meta-Lesson: A Decision Framework

Priya Chen synthesizes the three experiments into a decision framework that her team now uses for every new ML project:

When AutoML Wins

• Standard problem types (classification, regression on tabular data) with clean, single-source data
• Well-defined target variables with established feature patterns
• Speed is the primary constraint (rapid prototyping, feasibility validation)
• Business impact is moderate (the cost of a 1-2% accuracy gap is manageable)
• Resources are limited (no data science team, or the data science team is fully committed to higher-priority projects)

When Custom ML Wins

• Complex data integration across multiple sources, formats, or modalities
• Domain-specific feature engineering that requires expert knowledge
• Advanced architectures (pre-trained models, custom neural networks, hierarchical models)
• High-stakes decisions where small accuracy improvements translate to significant business value
• Custom business logic (confidence thresholds, multi-stage decisions, human-in-the-loop design)
• Regulatory requirements that demand full model transparency and auditability

When to Use Both

The most sophisticated approach — and the one Priya adopts — uses AutoML and custom development as complements:

1. AutoML first for feasibility. Use AutoML to validate that the problem is solvable with the available data. If AutoML achieves acceptable performance, deploy it. If it falls short, you now have a performance baseline and feature importance analysis that accelerates custom development.

2. AutoML for commoditized problems, custom for differentiated ones. Not every ML problem at the company is strategically important. Use AutoML for the 80% of problems where "good enough" is good enough. Invest custom development time in the 20% that create competitive differentiation.

3. AutoML for domain expert empowerment. Let business analysts build and iterate on models within their domain expertise, with the data science team providing governance oversight and stepping in when problems exceed AutoML capabilities.

Epilogue

Six months after the head-to-head comparison, Priya reports to the executive team:

• The churn prediction model (DataRobot, built by Maria) is in production, informing the retention team's outreach priorities. It has been retrained twice as the platform's monitoring detected performance drift.
• The demand forecasting model (custom, built by James) is in production, driving inventory allocation decisions. It has saved an estimated $1.8 million in the first two quarters through reduced overstock and stockouts.
• The ticket routing model (custom, built by James and the NLP specialist) is in production, auto-routing 78% of tickets with a 94.8% effective accuracy. It has reduced average ticket resolution time by 22%.
• Maria has since built three additional models using DataRobot — all for standard classification and regression problems where AutoML's performance was adequate. Each took less than a week.

The lesson: AutoML and custom ML are not competitors. They are tools in the same toolkit, suited to different problems. The art is knowing which tool to reach for — and that knowledge requires understanding both.

Discussion Questions

1. Priya's experiment used the same performance metrics (AUC, MAPE, accuracy) for both approaches. Are there other evaluation criteria — beyond those five dimensions — that should factor into the AutoML-vs-custom decision? What might they be?

2. Maria's DataRobot churn model is now in production. What governance processes should be in place for ongoing management? How should these differ from the governance processes for James's custom models?

3. The demand forecasting comparison showed that AutoML provided value as a "rapid baseline" even when it was not the final production model. How might organizations formalize this "AutoML first, custom second" workflow? What organizational structures support it?

4. In Problem 3, DataRobot's model performed worse than the human process. Should Maria have been allowed to build and test this model? What does the citizen data science governance framework say about cases where the model underperforms the status quo?

5. If AutoML platforms continue to improve — incorporating pre-trained language models, multi-modal data handling, and custom logic — at what point does the custom ML advantage narrow to the point of being economically unjustifiable? What capabilities would AutoML need to develop to close the gap on Problem 3?

This case study connects to Chapter 7 (classification), Chapter 8 (regression/forecasting), Chapter 14 (NLP), and Chapter 6 (build vs. buy). The governance themes connect forward to Chapter 27 (AI governance frameworks).