Case Study 2: Pinnacle Health's Parallel Claims Adjudication

Background

Pinnacle Health Systems processes 1.8 million medical claims per day across their network of 340 hospitals, 12,000 physicians, and 4.2 million active members. The claims adjudication batch — the nightly process that applies benefit rules, calculates provider payments, determines member liability, and posts financial entries — is the single longest job in their batch schedule.

Running serially, claims adjudication takes 3 hours 50 minutes. The batch window is 5 hours, but adjudication is followed by provider payment file generation (40 minutes), member explanation of benefits (30 minutes), general ledger posting (20 minutes), and regulatory reporting extracts (25 minutes). The serial total is 5 hours 45 minutes. Pinnacle Health has been running 45 minutes past the batch window every night, delaying the morning online startup and generating compliance findings from auditors.

Diane, the batch architect, knew that adjudication was the bottleneck. The program (CLMADJUD) was a 12,000-line COBOL program with 200+ benefit rules, cross-reference lookups against 15 DB2 tables, and complex accumulator logic for deductibles, out-of-pocket maximums, and family aggregation. It was the most complex batch program in Pinnacle's portfolio.

Ahmad, the DB2 DBA, had already optimized the database side: indexes were tuned, buffer pools were sized, RUNSTATS were current, and access paths were optimal. There was no more performance to squeeze from DB2 tuning. The only path forward was parallelism.

The Cross-Claim Dependency Problem

Claims adjudication is not a simple per-record process. Claims have dependencies:

Deductible accumulation. Member 4421003 has a $2,000 annual deductible. Claim A is for $800, Claim B is for $1,500, Claim C is for $400. The adjudication of Claim C depends on knowing that Claims A and B have already consumed $2,000 of the deductible. If Claim C is in a different partition than Claims A and B, partition C does not know the deductible has been met.

Family aggregation. Member 4421003 (subscriber), 4421004 (spouse), and 4421005 (dependent) share a family deductible of $4,000 and a family out-of-pocket maximum of $8,000. Claims for all three members must see the same accumulator values. If these members' claims are in different partitions, the accumulators are inconsistent.

Prior authorization. Claim for a surgery references a prior authorization claim. The authorization claim may be in a different partition. The adjudication program must verify the authorization exists and is valid before paying the surgery claim.

Coordination of benefits (COB). When a member has dual coverage, the primary insurer's payment affects the secondary insurer's calculation. If both insurers' claims are in Pinnacle's system (common for employer groups), the primary claim must adjudicate before the secondary.

These dependencies meant that Diane could not simply split claims by member number and run partitions independently. She needed a design that handled cross-partition dependencies.

The Two-Pass Architecture

Diane designed a two-pass parallel architecture:

Pass 1: Independent Adjudication (Parallelized)

The first pass adjudicates all claims that have no cross-partition dependencies. The program checks each claim against three conditions: 1. All deductible/OOP accumulator claims for this member/family are in the same partition 2. Any referenced prior authorizations are in the same partition 3. No COB dependencies cross partition boundaries

Claims that meet all three conditions are adjudicated normally. Claims that fail any condition are flagged as "deferred" and written to a deferred claims file.

The partition strategy: split by member's primary care physician (PCP) group. Pinnacle's network is organized into 48 PCP groups. Members are assigned to PCP groups, and most claims within a family go to providers within the same PCP group. This natural clustering meant that 92% of family-related claims landed in the same partition.

Diane created six partitions, each covering 8 PCP groups:

The imbalance ratio: 312,000/290,000 = 1.076:1. Excellent balance.

Pass 1 ran six partitions in parallel. Each partition processed its assigned claims, adjudicating approximately 92% of them and deferring 8% to the deferred claims file.

Pass 1 elapsed time: 38 minutes (compared to the serial equivalent of 3 hours 32 minutes for 92% of claims).

Pass 2: Deferred Claims Adjudication (Serial)

Pass 2 processed the deferred claims — the 8% with cross-partition dependencies. These claims were merged into a single file sorted by member number, ensuring that all claims for a member/family were processed together.

Because the deferred claims were only 8% of the total (approximately 144,000 claims), serial processing was fast: 22 minutes.

Total adjudication time: 38 + 5 (merge deferred files) + 22 = 65 minutes. Down from 230 minutes. A 3.5x improvement.

The Accumulator Pre-Load

The critical insight in Diane's design was the accumulator pre-load. Before Pass 1, a setup job queried the year-to-date accumulator values for every member from DB2 and loaded them into a flat file sorted by member number. Each partition received a copy of the accumulator file (or, more efficiently, each partition queried its own member range from the accumulator table).

During Pass 1, each partition started with correct year-to-date values. As it adjudicated claims, it updated in-memory accumulators. At the end of Pass 1, each partition wrote its updated accumulators back to DB2.

For deferred claims (Pass 2), the accumulators were refreshed after Pass 1's commit — so Pass 2 saw the most current values including all of Pass 1's updates.

This two-phase accumulator strategy (pre-load before Pass 1, refresh before Pass 2) eliminated 95% of the cross-partition accumulator dependency problem. The remaining 5% were edge cases handled by the deferral mechanism.

Ahmad's DB2 Parallelism Configuration

While Diane handled application-level partitioning, Ahmad configured DB2 for maximum throughput with six concurrent batch threads.

Tablespace alignment. The CLAIMS table had 12 DB2 partitions. Ahmad aligned them with Diane's 6 application partitions: each application partition accessed 2 DB2 partitions. This ensured that DB2's internal partition pruning eliminated 10 of 12 partitions from each application partition's queries.

Buffer pool sizing. Ahmad allocated 120,000 4K pages to BP3 (the CLAIMS tablespace buffer pool). With 6 concurrent threads, each thread had effective access to 20,000 pages — enough for the sequential prefetch and list prefetch patterns in the adjudication program.

DB2 I/O parallelism. With DEGREE(ANY) and 2 DB2 partitions per application partition, each partition's queries used I/O parallelism degree 2. This doubled the I/O throughput for the large tablespace scans in the benefit rule lookups.

Lock management. Ahmad set LOCKSIZE ROW on all batch-accessed tablespaces and increased NUMLKTS (lock escalation threshold) to 5,000. With six partitions committing every 500 updates, the peak lock count per partition was approximately 500 — well below the escalation threshold.

Connection pooling. Each of the six partitions used one DB2 thread. Ahmad reserved 10 threads for batch processing (leaving 4 as headroom). The DSNZPARM MAX_BATCH_CONNECTED was set to 15.

The combined effect: application parallelism (6x theoretical) × DB2 I/O parallelism (1.5x measured) = 9x theoretical improvement. Actual measured improvement: 3.5x (after accounting for the serial Pass 2, merge steps, partition overhead, and I/O contention). Still transformative.

The Pipeline

The complete Pinnacle Health EOD pipeline after redesign:

CLMSETUP        (serial, 5 min)
  - Compute partition boundaries
  - Pre-load accumulators
  - Populate partition control table
  |
  +-- CLMADJUD-P01  (parallel, ~35 min)
  +-- CLMADJUD-P02  (parallel, ~37 min)
  +-- CLMADJUD-P03  (parallel, ~36 min)
  +-- CLMADJUD-P04  (parallel, ~38 min)
  +-- CLMADJUD-P05  (parallel, ~36 min)
  +-- CLMADJUD-P06  (parallel, ~35 min)
  |
CLMDEFMERGE     (serial, 5 min)
  - Merge deferred claim files from all 6 partitions
  - Refresh accumulators from DB2
  |
CLMADJUD-PASS2  (serial, 22 min)
  - Adjudicate deferred claims
  |
CLMRECONCILE    (serial, 3 min)
  - Verify all claims adjudicated
  - Validate accumulator totals
  - Check for gaps/overlaps
  |
  +-- CLMPROVPAY  (parallel with CLMMBREOB, 25 min)
  +-- CLMMBREOB   (parallel with CLMPROVPAY, 20 min)
  |
CLMGLPOST       (serial, 15 min)
CLMREGEXT       (serial, 18 min)

Critical path: 5 + 38 + 5 + 22 + 3 + 25 + 15 + 18 = 131 minutes = 2 hours 11 minutes.

The serial pipeline was 5 hours 45 minutes. The parallel pipeline fit within 2 hours 15 minutes with margin. Pinnacle Health went from 45 minutes past the window to nearly 3 hours of margin.

Production Challenges

Challenge 1: The PCP Group Rebalancing

Six months after go-live, Pinnacle onboarded a large physician group (22 physicians) that was assigned to PCP Group 05 (in Partition 1). This shifted 180,000 members and their claims into Partition 1, increasing its volume by 25%. The imbalance ratio spiked to 1.35:1, and P1's elapsed time grew from 35 minutes to 48 minutes — extending the critical path by 10 minutes.

Diane recomputed partition boundaries, splitting the 48 PCP groups differently: P1 got groups 01-07 (one fewer), P6 got groups 40-48 (one more). The rebalancing took 30 minutes to analyze and implement. The imbalance ratio returned to 1.09:1.

Lesson: rebalancing should be proactive (monthly analysis) not reactive (post-incident).

Challenge 2: The Deadlock Storm

During a high-volume day (insurance open enrollment produced 2.4 million claims instead of the usual 1.8 million), partitions 2 and 4 experienced 47 deadlocks in 10 minutes. The deadlock retry logic handled each one, but the retries added 8 minutes to those partitions' elapsed time.

Ahmad investigated. The deadlocks were all on the MEMBER_ACCUMULATOR table — not the CLAIMS table. Partitions 2 and 4 were both updating accumulators for members in the same family that spanned PCP groups 14 and 28. Despite the PCP-group-based partitioning, some large families (employer groups with 50+ members) had members assigned to multiple PCP groups.

The fix was two-part: (1) change the accumulator update to use individual member rows instead of a family summary row (eliminating the shared row contention), and (2) add a 1-2 second random delay to the deadlock retry (breaking the livelock cycle).

After the fix, deadlocks dropped from 47 per high-volume day to 2-3 — a level that the retry logic handled without perceptible impact.

Challenge 3: The Regulatory Audit

Pinnacle's state regulators audited the claims adjudication process and questioned whether the parallel design could produce different results than serial processing. Specifically: if Claim A and Claim B for the same member arrive on the same day, does the processing order within a partition affect the result?

Diane demonstrated that the adjudication rules were commutative for deductible and OOP calculations — the order in which claims are applied does not change the final accumulator values. $800 + $1,500 = $1,500 + $800 = $2,300, regardless of order.

However, she discovered one rule that was NOT commutative: the "duplicate claim" check. If two identical claims arrived, the first was paid and the second was denied as a duplicate. In serial processing, the "first" claim was always the one with the earlier received timestamp. In parallel processing, if the two claims landed in different partitions, both might be "first" in their respective partitions.

The fix: the duplicate claim check was moved from Pass 1 to Pass 2, where all claims for a member were processed together in timestamp order. This added approximately 30 seconds to Pass 2 elapsed time (negligible) and eliminated the duplicate-claim ordering issue.

The regulators accepted the redesign after reviewing the commutativity analysis and the duplicate claim fix.

Performance Summary

The 6% increase in DB2 CPU per claim was the cost of parallelism overhead — partition control table access, deadlock retry, and additional commit processing. At the claim volumes Pinnacle processes, this was an additional 2 CPU-seconds per day — invisible.

Lessons Learned

1. Domain knowledge drives partition strategy. The PCP group partitioning worked because Diane understood the healthcare domain — families cluster around PCPs, and claims cluster around families. A naive account-number split would have scattered family members across partitions, driving the deferral rate from 8% to over 30% and negating much of the parallelism benefit.

2. The two-pass pattern handles dependencies elegantly. Rather than trying to solve all cross-partition dependencies in the parallel pass (which would require inter-partition communication and synchronization), the deferred/serial pass handles the 8% of complex cases simply and correctly. The 92/8 split makes the serial pass fast.

3. Commutativity analysis is essential. If your processing rules are order-dependent, parallel processing may produce different results than serial processing. You must prove that parallel execution is equivalent to serial execution, or isolate the order-dependent logic into a serial phase.

4. Accumulator pre-loading is the key enabler. Without pre-loaded accumulators, each partition would need real-time visibility into other partitions' accumulator updates — requiring shared memory, message queuing, or database-level synchronization. Pre-loading eliminates this cross-partition communication entirely for the 92% of claims processed in Pass 1.

5. Operational procedures must evolve with the architecture. The DBA's tablespace operations, the scheduler's dependency definitions, the operations team's monitoring procedures, and the auditor's process documentation all changed. Parallel batch is not just a technical change — it is an operational change.

Discussion Questions

1. Diane chose 6 partitions. The system could support 8 or even 12. What factors argue for 6 over a higher count? Under what circumstances would you increase to 8?

2. The two-pass architecture defers 8% of claims to serial processing. If the deferral rate increased to 25% (due to a business change like more family plans), at what point does the two-pass design become ineffective? What alternative design would you consider?

3. Ahmad sized buffer pool BP3 at 120,000 pages for 6 concurrent threads. How would you validate this sizing? What monitoring data would you collect, and what would trigger a buffer pool increase?

4. The duplicate claim fix moved logic from Pass 1 to Pass 2. Are there other adjudication rules that might be order-dependent? How would you systematically identify them?

5. Pinnacle's regulators required proof of equivalence between serial and parallel results. How would you design a periodic verification process that runs both serial and parallel on the same input and compares results?