Case Study 2: Meridian Financial — Batch vs. Real-Time Credit Decisioning
Context
Meridian Financial is a mid-size consumer lending institution processing 15,000 credit card applications per day. Historically, all applications were scored overnight: applicants submitted their information during the day, a batch job ran at 2am using a gradient-boosted tree ensemble (XGBoost, 500 trees, 200 features), and applicants received their decision by email the following morning.
This process was reliable. The model's AUC on held-out data was 0.83, the reject-rate was 32%, and the 90-day default rate on approved applications was 2.1%. The batch pipeline ran on a single Spark cluster, processed all 15,000 applications in 45 minutes, and had not failed in 14 months.
But the business is under pressure. Three competitors have launched instant-decision credit card products. Meridian's conversion rate (application starts that result in account openings) has dropped from 68% to 54% over 12 months. The product team attributes 60% of the drop to applicants who abandon the process when they learn the decision will come the next day.
The CTO asks the data science team: "Can we give applicants an instant decision without sacrificing model quality or regulatory compliance?"
The Analysis
Understanding the Constraints
The team identifies four constraints that any solution must satisfy:
-
Regulatory. Under ECOA and FCRA, every denial must include an adverse action notice listing the top factors. The model must be interpretable enough to generate these reasons. All feature values, model scores, and decisions must be logged and retained for 7 years.
-
Data availability. The batch model uses 200 features from 5 data sources: - Application data (available instantly): income, employment, housing status, requested credit limit. - Credit bureau (5-15 seconds via API): credit score, number of accounts, utilization ratio, delinquency history, inquiry count. - Bank transaction history (30-60 minutes via bank connection): income verification, spending patterns, cash flow stability. - Employment verification (2-24 hours via third-party service): employer confirmation, tenure, salary verification. - Alternative data (4-8 hours via aggregator): rent payment history, utility payments, telecom payments.
In real-time, only the first two sources are available.
-
Quality. The risk team requires that any real-time model have AUC $\geq 0.78$ (a maximum drop of 0.05 from the batch model's 0.83). The 90-day default rate must not increase by more than 0.3 percentage points.
-
Trust. Applicants who receive an instant pre-approval and are later denied by the batch model will lose trust in the institution. The false positive pre-approval rate (applicants pre-approved in real time but denied in batch) must be below 3%.
Option Analysis
The team evaluates three architectures.
from dataclasses import dataclass
from typing import List, Dict, Tuple
import numpy as np
@dataclass
class ArchitectureOption:
"""A candidate architecture for the credit decisioning system.
Attributes:
name: Architecture name.
description: Brief description.
latency: Expected decision latency for the applicant.
features_available: Number of features available at decision time.
estimated_auc: Estimated model AUC on held-out data.
estimated_false_positive_rate: Rate of real-time approvals
later denied by the full model.
infrastructure_cost_monthly: Estimated infrastructure cost.
engineering_months: Estimated build time in engineer-months.
regulatory_risk: Assessment of regulatory compliance risk.
"""
name: str
description: str
latency: str
features_available: int
estimated_auc: float
estimated_false_positive_rate: float
infrastructure_cost_monthly: float
engineering_months: float
regulatory_risk: str
options = [
ArchitectureOption(
name="Status Quo (batch only)",
description="Keep overnight batch processing. No instant decisions.",
latency="12-18 hours",
features_available=200,
estimated_auc=0.83,
estimated_false_positive_rate=0.0,
infrastructure_cost_monthly=2_500.0,
engineering_months=0.0,
regulatory_risk="Low (proven, audited system)",
),
ArchitectureOption(
name="Full real-time",
description="Score all applications in real time with reduced features.",
latency="< 30 seconds",
features_available=65,
estimated_auc=0.79,
estimated_false_positive_rate=0.0,
infrastructure_cost_monthly=12_000.0,
engineering_months=6.0,
regulatory_risk="Medium (new model needs regulatory validation; "
"fewer features may miss protected-class correlates)",
),
ArchitectureOption(
name="Hybrid (instant pre-approval + batch final)",
description="Real-time model provides instant pre-approval on limited "
"features. Batch model makes final decision with full features.",
latency="< 30 seconds (pre-approval); 12-18 hours (final)",
features_available=65,
estimated_auc=0.79,
estimated_false_positive_rate=0.025,
infrastructure_cost_monthly=15_000.0,
engineering_months=8.0,
regulatory_risk="Low-Medium (pre-approval clearly labeled as conditional; "
"final decision uses proven batch model)",
),
]
print(f"{'Option':<35} {'AUC':>5} {'FP Rate':>8} {'Cost/mo':>10} {'Eng-mo':>7}")
print("-" * 70)
for opt in options:
print(f"{opt.name:<35} {opt.estimated_auc:>5.2f} "
f"{opt.estimated_false_positive_rate:>7.1%} "
f"${opt.infrastructure_cost_monthly:>8,.0f} "
f"{opt.engineering_months:>6.1f}")
Option AUC FP Rate Cost/mo Eng-mo
----------------------------------------------------------------------
Status Quo (batch only) 0.83 0.0% $2,500 0.0
Full real-time 0.79 0.0% $12,000 6.0
Hybrid (instant pre-approval + ... 0.79 2.5% $15,000 8.0
The Hybrid Architecture
The team selects the hybrid architecture. The implementation has two serving paths.
Real-time path (< 30 seconds). When an applicant submits their information:
- Application data is parsed and validated (< 1 second).
- Credit bureau data is fetched via API (5-15 seconds).
- A lightweight logistic regression model scores the application using 65 features (< 100ms).
- The model produces a pre-approval decision: APPROVED, NEEDS_REVIEW, or DECLINED.
- The applicant sees the decision on screen.
Batch path (overnight). All applications from the day — including those that received real-time pre-approval — are scored by the full XGBoost ensemble using all 200 features. The batch model makes the final decision. If the batch model agrees with the real-time model, no action is needed. If the batch model denies a pre-approved applicant, the institution must notify the applicant within 30 days (per ECOA).
from dataclasses import dataclass
from typing import Optional
from enum import Enum
class PreApprovalDecision(Enum):
"""Real-time pre-approval outcomes."""
APPROVED = "approved"
NEEDS_REVIEW = "needs_review"
DECLINED = "declined"
class FinalDecision(Enum):
"""Batch final decision outcomes."""
APPROVED = "approved"
DECLINED = "declined"
@dataclass
class CreditDecisionPipeline:
"""Hybrid credit decision system with real-time and batch paths.
The real-time model uses a conservative threshold: it only pre-approves
applicants with high confidence. Borderline cases receive NEEDS_REVIEW
(the applicant is told the decision requires additional verification).
Attributes:
realtime_approval_threshold: Score above which to pre-approve.
realtime_decline_threshold: Score below which to decline.
batch_approval_threshold: Score above which to approve in batch.
"""
realtime_approval_threshold: float = 0.75
realtime_decline_threshold: float = 0.30
batch_approval_threshold: float = 0.50
def realtime_decision(self, score: float) -> PreApprovalDecision:
"""Make a real-time pre-approval decision.
The threshold is deliberately conservative: only applicants
with high predicted creditworthiness receive instant approval.
This minimizes false positive pre-approvals.
Args:
score: Real-time model predicted probability of repayment.
Returns:
Pre-approval decision.
"""
if score >= self.realtime_approval_threshold:
return PreApprovalDecision.APPROVED
elif score <= self.realtime_decline_threshold:
return PreApprovalDecision.DECLINED
else:
return PreApprovalDecision.NEEDS_REVIEW
def batch_decision(self, score: float) -> FinalDecision:
"""Make a batch final decision.
Args:
score: Batch model predicted probability of repayment.
Returns:
Final decision.
"""
if score >= self.batch_approval_threshold:
return FinalDecision.APPROVED
return FinalDecision.DECLINED
def check_consistency(
self,
realtime_decision: PreApprovalDecision,
batch_decision: FinalDecision,
) -> Optional[str]:
"""Check for inconsistency between real-time and batch decisions.
The critical case is a false positive pre-approval: the applicant
was told they were pre-approved, but the batch model declines them.
Args:
realtime_decision: The real-time pre-approval outcome.
batch_decision: The batch final outcome.
Returns:
Alert message if inconsistent, None if consistent.
"""
if (realtime_decision == PreApprovalDecision.APPROVED
and batch_decision == FinalDecision.DECLINED):
return (
"FALSE POSITIVE PRE-APPROVAL: Applicant was pre-approved "
"in real time but declined by the batch model. "
"Adverse action notice required."
)
return None
# Simulate the hybrid system on a cohort of applicants
pipeline = CreditDecisionPipeline()
rng = np.random.RandomState(42)
n_applicants = 10000
# Simulate correlated real-time and batch scores
# (real-time model is noisier due to fewer features)
true_creditworthiness = rng.beta(2.5, 2.0, size=n_applicants)
realtime_scores = np.clip(
true_creditworthiness + rng.normal(0, 0.12, size=n_applicants), 0, 1
)
batch_scores = np.clip(
true_creditworthiness + rng.normal(0, 0.05, size=n_applicants), 0, 1
)
rt_decisions = [pipeline.realtime_decision(s) for s in realtime_scores]
batch_decisions = [pipeline.batch_decision(s) for s in batch_scores]
# Count outcomes
rt_approved = sum(1 for d in rt_decisions if d == PreApprovalDecision.APPROVED)
rt_declined = sum(1 for d in rt_decisions if d == PreApprovalDecision.DECLINED)
rt_review = sum(1 for d in rt_decisions if d == PreApprovalDecision.NEEDS_REVIEW)
false_positives = sum(
1 for rt, batch in zip(rt_decisions, batch_decisions)
if rt == PreApprovalDecision.APPROVED and batch == FinalDecision.DECLINED
)
false_positive_rate = false_positives / max(rt_approved, 1)
print(f"Cohort: {n_applicants:,} applicants")
print(f"\nReal-time decisions:")
print(f" Pre-approved: {rt_approved:,} ({rt_approved/n_applicants:.1%})")
print(f" Needs review: {rt_review:,} ({rt_review/n_applicants:.1%})")
print(f" Declined: {rt_declined:,} ({rt_declined/n_applicants:.1%})")
print(f"\nFalse positive pre-approvals: {false_positives} "
f"({false_positive_rate:.1%} of pre-approved)")
print(f"Target: < 3.0%")
print(f"Status: {'PASS' if false_positive_rate < 0.03 else 'FAIL'}")
Cohort: 10,000 applicants
Real-time decisions:
Pre-approved: 3,379 (33.8%)
Needs review: 4,207 (42.1%)
Declined: 2,414 (24.1%)
False positive pre-approvals: 49 (1.4% of pre-approved)
Target: < 3.0%
Status: PASS
The conservative real-time threshold (0.75) achieves a 1.4% false positive pre-approval rate — well within the 3.0% target. The trade-off is that only 33.8% of applicants receive instant approval; the remaining 66.2% receive either "needs review" (told to expect a decision within 24 hours) or instant decline.
The ADR
# ADR-003: Credit Decisioning Serving Architecture
## Status
Accepted (2026-02-01)
## Context
Meridian Financial processes 15,000 credit applications/day.
Current batch-only system has 12-18 hour decision latency.
Conversion rate has dropped 14 percentage points (68% → 54%)
over 12 months, attributed primarily to applicant dropout.
Competitors offer instant decisions.
Regulatory constraints: ECOA adverse action notices required
for all denials. FCRA requires 7-year retention of all decision
inputs and outputs. Model must be auditable.
## Decision
Hybrid architecture: real-time pre-approval with conservative
thresholds, batch final decision with full feature set.
## Options Considered
1. Status quo (batch only)
- Pros: Proven, audited, cheap ($2,500/mo), full features
- Cons: Continued conversion loss (~$8M/year estimated)
- Rejected: Business impact too severe
2. Full real-time (replace batch entirely)
- Pros: Simplest architecture, fastest decisions
- Cons: AUC drops from 0.83 to 0.79; fewer features mean
higher default rate; new model requires full regulatory
validation; no batch audit trail
- Rejected: Quality and regulatory risk
3. Hybrid (chosen)
- Pros: Instant feedback for high-confidence applicants;
full-quality batch decision for final terms; batch audit
trail preserved; regulatory risk limited to pre-approval
- Cons: Dual infrastructure; false positive pre-approvals
(managed to < 3%); applicant confusion for "needs review"
## Consequences
- Must maintain both real-time and batch pipelines ($15K/mo).
- Real-time model requires separate regulatory validation.
- False positive pre-approvals require adverse action follow-up.
- "Needs review" experience must be designed to retain applicants.
- Monitoring must track false positive rate daily; alert at 2.5%.
## Review Date
2026-08-01. Reconsider if: (a) real-time model AUC improves
to >= 0.81 with additional data sources, (b) bank connection
API latency drops below 5 seconds (enabling more features in
real time), (c) false positive rate exceeds 2.5% for 7 days.
Outcome
After deploying the hybrid system:
- Conversion rate recovered from 54% to 63% (+9 percentage points). The 33.8% of applicants who received instant pre-approval had a 92% conversion rate. The "needs review" group had a 58% conversion rate (up from 54% due to improved messaging: "We're verifying your information — expect a decision within 24 hours").
- False positive pre-approval rate: 1.6% over the first quarter (within the 3% target).
- Default rate: unchanged at 2.1% on approved applications (the batch model's quality was preserved for final decisions).
- Regulatory audit passed. The hybrid system was audited by the OCC and passed with no findings. The auditors noted that the batch model's audit trail was identical to the previous system's, and the real-time model's conservative thresholds demonstrated appropriate caution.
Key Takeaway
The hybrid architecture is a direct application of the "simplest model that works" principle at the system level. The real-time model does not need to be as good as the batch model — it only needs to be good enough to make high-confidence pre-approval decisions while deferring uncertain cases to the full model. By separating the speed decision (real-time) from the quality decision (batch), Meridian captures the conversion benefit of instant feedback without sacrificing the model quality that regulatory compliance and risk management require.
This pattern — fast approximate decision followed by slow precise decision — applies broadly: real-time fraud pre-screening with batch investigation, instant insurance quotes with batch underwriting, real-time ad relevance filtering with batch billing reconciliation. The architecture is the same; the domain constraints differ.