Case Study 1: Scaling the Banking Application for Multi-Branch Operation
Background
Community Bank has operated as a single-branch institution for twelve years, running the capstone banking application on a single z/OS LPAR with one CICS region, one DB2 subsystem, and a set of VSAM files that handle all customer and account data. The application processes approximately 30,000 transactions per day, runs a nightly batch cycle in under two hours, and serves 50 concurrent CICS users without performance issues.
The bank's board of directors has approved an aggressive expansion plan: three new branches will open over the next eighteen months, and two additional branches are planned for the following year. Each branch will have its own teller workstations connected to the central CICS system, its own ATM network, and its own loan officers who need real-time access to customer account information. The projected transaction volume after all five branches are operational is 200,000 transactions per day, with 200 concurrent CICS users during peak hours.
The CTO has asked the development team to evaluate the capstone application and produce a detailed plan for scaling it to support multi-branch operations. The plan must address three fundamental challenges: the data architecture must accommodate branch-level identity and processing, the batch cycle must handle the increased volume within the existing overnight window, and the online system must support branch-specific views while maintaining a unified customer experience.
Phase 1: Data Architecture Modifications
Branch Code Integration
The first and most pervasive change is the introduction of branch identity throughout the data model. The existing capstone records already contain a branch code field (ACCT-BRANCH-CODE in the account record, CUST-BRANCH-CODE in the customer record), but these fields are populated with a single value ('0001') and are not used in any business logic. The multi-branch design promotes branch code from a passive attribute to an active processing dimension.
The team begins by defining a branch master table in DB2:
EXEC SQL
CREATE TABLE BRANCH_MASTER
( BRANCH_CODE CHAR(4) NOT NULL,
BRANCH_NAME VARCHAR(40) NOT NULL,
BRANCH_ADDRESS VARCHAR(80) NOT NULL,
BRANCH_CITY CHAR(25) NOT NULL,
BRANCH_STATE CHAR(2) NOT NULL,
BRANCH_ZIP CHAR(10) NOT NULL,
REGION_CODE CHAR(4) NOT NULL,
MANAGER_ID CHAR(8) NOT NULL,
OPEN_DATE DATE NOT NULL,
STATUS CHAR(1) NOT NULL
DEFAULT 'A',
DAILY_TXN_LIMIT DECIMAL(11,2)
DEFAULT 500000.00,
PRIMARY KEY (BRANCH_CODE) )
END-EXEC
A corresponding copybook defines the COBOL record layout used by all programs that reference branch data:
*================================================================*
* CAPSTONE-BRANCH.CPY *
* Community Bank Application - Branch Master Record *
*================================================================*
01 BRANCH-RECORD.
05 BRNCH-CODE PIC X(04).
05 BRNCH-NAME PIC X(40).
05 BRNCH-ADDRESS PIC X(80).
05 BRNCH-CITY PIC X(25).
05 BRNCH-STATE PIC X(02).
05 BRNCH-ZIP PIC X(10).
05 BRNCH-REGION PIC X(04).
05 BRNCH-MANAGER-ID PIC X(08).
05 BRNCH-OPEN-DATE PIC X(10).
05 BRNCH-STATUS PIC X(01).
88 BRNCH-ACTIVE VALUE 'A'.
88 BRNCH-INACTIVE VALUE 'I'.
88 BRNCH-PENDING VALUE 'P'.
05 BRNCH-DAILY-TXN-LIMIT PIC S9(9)V99 COMP-3.
05 BRNCH-DAILY-TXN-TOTAL PIC S9(11)V99 COMP-3.
05 BRNCH-DAILY-TXN-COUNT PIC 9(07).
05 BRNCH-FILLER PIC X(30).
Transaction File Partitioning
With multiple branches generating transactions simultaneously, the single daily transaction file becomes a bottleneck. The team redesigns the transaction input architecture to use branch-specific transaction files that are merged before batch processing:
Branch 0001: BANK.BR0001.DAILY.TXN
Branch 0002: BANK.BR0002.DAILY.TXN
Branch 0003: BANK.BR0003.DAILY.TXN
Branch 0004: BANK.BR0004.DAILY.TXN
Each branch's CICS region writes transactions to its own file during the business day. The nightly batch cycle begins with a merge step that combines all branch files into a single sorted transaction file:
//MERGE EXEC PGM=SORT
//SORTIN01 DD DSN=BANK.BR0001.DAILY.TXN,DISP=SHR
//SORTIN02 DD DSN=BANK.BR0002.DAILY.TXN,DISP=SHR
//SORTIN03 DD DSN=BANK.BR0003.DAILY.TXN,DISP=SHR
//SORTIN04 DD DSN=BANK.BR0004.DAILY.TXN,DISP=SHR
//SORTOUT DD DSN=BANK.DAILY.TXN.MERGED,
// DISP=(NEW,CATLG,DELETE),
// SPACE=(CYL,(500,100)),
// DCB=(RECFM=FB,LRECL=300,BLKSIZE=27900)
//SORTWK01 DD UNIT=SYSDA,SPACE=(CYL,(200,50))
//SORTWK02 DD UNIT=SYSDA,SPACE=(CYL,(200,50))
//SORTWK03 DD UNIT=SYSDA,SPACE=(CYL,(200,50))
//SYSIN DD *
SORT FIELDS=(1,10,CH,A,23,10,CH,A,25,8,CH,A)
SUM FIELDS=NONE
/*
The SORT FIELDS specification sorts by account number (positions 1-10), then transaction date (positions 23-32), then transaction time (positions 25-32), ensuring that transactions are processed in chronological order within each account.
Customer Relationship Across Branches
A critical design decision is how to handle customers who bank at multiple branches. The team establishes a rule: a customer has one master record with a "home branch" designation, but can transact at any branch. The customer record's CUST-BRANCH-CODE indicates the home branch for reporting and relationship management purposes, but no branch-level restriction is placed on transaction processing.
This decision has implications for the CICS inquiry programs. When a teller at Branch 0003 looks up a customer whose home branch is 0001, the system must display the full customer relationship while noting the home branch. The team adds a branch-awareness indicator to the CICS screen:
01 WS-BRANCH-INDICATOR.
05 WS-BRNCH-IND-TEXT PIC X(30).
88 BRNCH-HOME VALUE
'HOME BRANCH '.
88 BRNCH-VISITING VALUE
'VISITING - HOME BR: '.
05 WS-BRNCH-IND-CODE PIC X(04).
Phase 2: Batch Processing Scalability
Volume Analysis
The increase from 30,000 to 200,000 daily transactions fundamentally changes the batch processing challenge. The team performs a detailed timing analysis of the current batch cycle:
| Step | Program | Current Time | Projected Time (200K) |
|---|---|---|---|
| Validation | CAPVALID | 8 min | 53 min |
| Sort | DFSORT | 3 min | 20 min |
| Posting | CAPTRANS | 45 min | 300 min |
| Interest | CAPINTRN | 30 min | 30 min |
| Statements | CAPSTMNT | 25 min | 167 min |
| GL Balance | CAPRPORT | 10 min | 10 min |
| Total | 121 min | 580 min |
The projected batch window of 580 minutes (nearly 10 hours) far exceeds the 4-hour overnight window. The team must reduce this to under 240 minutes.
Parallel Processing Strategy
The team designs a parallel processing approach that exploits the branch partitioning:
//*=============================================================*
//* PHASE 1: PARALLEL BRANCH VALIDATION AND SORT *
//*=============================================================*
//BR01VALD EXEC PGM=CAPVALID
//TXNIN DD DSN=BANK.BR0001.DAILY.TXN,DISP=SHR
//TXNOUT DD DSN=BANK.BR0001.TXN.VALID,
// DISP=(NEW,CATLG,DELETE),
// SPACE=(CYL,(50,10))
//*
//BR02VALD EXEC PGM=CAPVALID
//TXNIN DD DSN=BANK.BR0002.DAILY.TXN,DISP=SHR
//TXNOUT DD DSN=BANK.BR0002.TXN.VALID,
// DISP=(NEW,CATLG,DELETE),
// SPACE=(CYL,(50,10))
//*
//BR03VALD EXEC PGM=CAPVALID
//TXNIN DD DSN=BANK.BR0003.DAILY.TXN,DISP=SHR
//TXNOUT DD DSN=BANK.BR0003.TXN.VALID,
// DISP=(NEW,CATLG,DELETE),
// SPACE=(CYL,(50,10))
//*
//BR04VALD EXEC PGM=CAPVALID
//TXNIN DD DSN=BANK.BR0004.DAILY.TXN,DISP=SHR
//TXNOUT DD DSN=BANK.BR0004.TXN.VALID,
// DISP=(NEW,CATLG,DELETE),
// SPACE=(CYL,(50,10))
By running four validation steps in parallel (using JES2 job scheduling or a workload automation tool), the validation phase completes in approximately 53 minutes instead of 212 minutes.
The transaction posting program is more challenging to parallelize because multiple branches can have transactions for the same account (a customer deposits at Branch 0001 and their spouse withdraws at Branch 0003). The team solves this by partitioning the account file by account number ranges rather than by branch:
*================================================================*
* Partition-aware transaction posting *
* Each instance processes a range of account numbers *
*================================================================*
01 WS-PARTITION-RANGE.
05 WS-ACCT-LOW PIC X(10).
05 WS-ACCT-HIGH PIC X(10).
PROCEDURE DIVISION.
0000-MAIN.
ACCEPT WS-ACCT-LOW FROM SYSIN-LOW
ACCEPT WS-ACCT-HIGH FROM SYSIN-HIGH
PERFORM 1000-INITIALIZE
PERFORM 2000-PROCESS-TRANSACTIONS
UNTIL END-OF-FILE
PERFORM 9000-TERMINATE
STOP RUN.
2000-PROCESS-TRANSACTIONS.
IF TXN-ACCT-NUMBER >= WS-ACCT-LOW
AND TXN-ACCT-NUMBER <= WS-ACCT-HIGH
PERFORM 3000-POST-TRANSACTION
END-IF
PERFORM 2900-READ-NEXT-TRANSACTION
.
Four posting instances run in parallel, each handling a quarter of the account range. Because each instance has exclusive access to its account range, there is no contention on the VSAM file. The account ranges are defined so each partition contains approximately the same number of accounts.
Optimized Interest Calculation
The interest calculation program is already I/O-bound rather than CPU-bound: it reads every active account sequentially and performs a simple computation. The team optimizes it by using multi-row FETCH from DB2 and buffering VSAM updates:
01 WS-ACCT-BLOCK.
05 WS-BLOCK-SIZE PIC S9(4) COMP VALUE 100.
05 WS-BLOCK-COUNT PIC S9(4) COMP VALUE ZERO.
05 WS-ACCT-ROWS OCCURS 100 TIMES.
10 WS-ROW-ACCT-NUM PIC X(10).
10 WS-ROW-BALANCE PIC S9(11)V99 COMP-3.
10 WS-ROW-RATE PIC 9V9(6).
10 WS-ROW-ACCRUED PIC S9(9)V99 COMP-3.
10 WS-ROW-STATUS PIC X(01).
4000-FETCH-ACCOUNT-BLOCK.
EXEC SQL
FETCH ACCT-INT-CSR
FOR :WS-BLOCK-SIZE ROWS
INTO :WS-ACCT-ROWS
END-EXEC
MOVE SQLERRD(3) TO WS-BLOCK-COUNT
.
4100-PROCESS-BLOCK.
PERFORM VARYING WS-BLK-IDX FROM 1 BY 1
UNTIL WS-BLK-IDX > WS-BLOCK-COUNT
IF WS-ROW-STATUS(WS-BLK-IDX) = 'A'
AND WS-ROW-BALANCE(WS-BLK-IDX) > ZERO
COMPUTE WS-ROW-ACCRUED(WS-BLK-IDX)
ROUNDED =
WS-ROW-ACCRUED(WS-BLK-IDX) +
(WS-ROW-BALANCE(WS-BLK-IDX) *
WS-ROW-RATE(WS-BLK-IDX) / 365)
END-IF
END-PERFORM
.
This block-fetch approach reduces the number of DB2 calls by a factor of 100, cutting the interest calculation time from 30 minutes to approximately 5 minutes.
Phase 3: Consolidated Reporting
Branch-Level Reports
Each branch manager needs daily, weekly, and monthly reports that show their branch's performance. The team designs a reporting hierarchy:
- Branch Daily Summary: Transaction counts and volumes by type, new accounts opened, accounts closed, exception items
- Branch Monthly P&L: Interest income, fee income, operating expenses allocated to the branch
- Regional Rollup: Aggregation of branch reports within each region
- Bank-Wide Consolidation: Total across all branches with inter-branch elimination
The report program uses a three-level control break structure keyed on region, branch, and account type:
01 WS-RPT-CONTROL-FIELDS.
05 WS-PREV-REGION PIC X(04).
05 WS-PREV-BRANCH PIC X(04).
05 WS-PREV-ACCT-TYPE PIC X(01).
05 WS-BRANCH-TOTALS.
10 WS-BR-TXN-COUNT PIC 9(07) VALUE ZERO.
10 WS-BR-DEPOSIT-AMT PIC S9(13)V99 COMP-3
VALUE ZERO.
10 WS-BR-WDRAW-AMT PIC S9(13)V99 COMP-3
VALUE ZERO.
10 WS-BR-XFER-AMT PIC S9(13)V99 COMP-3
VALUE ZERO.
10 WS-BR-FEE-AMT PIC S9(9)V99 COMP-3
VALUE ZERO.
10 WS-BR-INT-AMT PIC S9(9)V99 COMP-3
VALUE ZERO.
10 WS-BR-NEW-ACCTS PIC 9(05) VALUE ZERO.
10 WS-BR-CLOSED-ACCTS PIC 9(05) VALUE ZERO.
05 WS-REGION-TOTALS.
10 WS-RG-TXN-COUNT PIC 9(09) VALUE ZERO.
10 WS-RG-DEPOSIT-AMT PIC S9(15)V99 COMP-3
VALUE ZERO.
10 WS-RG-WDRAW-AMT PIC S9(15)V99 COMP-3
VALUE ZERO.
05 WS-BANK-TOTALS.
10 WS-BK-TXN-COUNT PIC 9(09) VALUE ZERO.
10 WS-BK-DEPOSIT-AMT PIC S9(15)V99 COMP-3
VALUE ZERO.
10 WS-BK-WDRAW-AMT PIC S9(15)V99 COMP-3
VALUE ZERO.
Inter-Branch Transfer Reconciliation
When a customer transfers money between accounts at different branches, the general ledger must record an inter-branch settlement entry. The team creates a new GL account class for inter-branch transfers and a reconciliation program that verifies all inter-branch entries net to zero at the bank level:
5000-CHECK-INTER-BRANCH.
* For each branch pair, verify that debits
* from Branch A to Branch B equal credits
* from Branch B to Branch A
EXEC SQL
SELECT SENDING_BRANCH,
RECEIVING_BRANCH,
SUM(DEBIT_AMOUNT) AS TOTAL_DEBIT,
SUM(CREDIT_AMOUNT) AS TOTAL_CREDIT
FROM GL_INTER_BRANCH
WHERE POST_DATE = :HV-PROCESS-DATE
GROUP BY SENDING_BRANCH, RECEIVING_BRANCH
ORDER BY SENDING_BRANCH, RECEIVING_BRANCH
END-EXEC
PERFORM UNTIL SQLCODE NOT = 0
IF HV-TOTAL-DEBIT NOT = HV-TOTAL-CREDIT
PERFORM 5100-REPORT-DISCREPANCY
END-IF
EXEC SQL FETCH INTERBR-CSR
INTO :HV-SEND-BR, :HV-RECV-BR,
:HV-TOTAL-DEBIT, :HV-TOTAL-CREDIT
END-EXEC
END-PERFORM
.
Phase 4: Operational Considerations
CICS Routing for Multi-Branch
The team configures CICS transaction routing so that each branch's terminal traffic is handled by a local CICS Application Owning Region (AOR), while all branches share a common set of VSAM files and DB2 tables through a CICS Terminal Owning Region (TOR) and interconnected AORs. This architecture provides:
- Load distribution: Each AOR handles its branch's transaction volume independently
- Isolation: A problem in one branch's AOR does not affect other branches
- Centralized data: All AORs access the same VSAM files and DB2 subsystem, ensuring data consistency
The CICS routing is configured using CICS CSD definitions that map transaction codes to the appropriate AOR based on the terminal's branch identifier.
Disaster Recovery Implications
Multi-branch operation changes the disaster recovery calculus. With a single branch, a system outage meant one location was down. With four branches, an outage affects the entire bank. The team designs a recovery strategy with two key elements:
First, the VSAM files are journaled using CICS journaling and DFHSM automatic backup. The Recovery Point Objective is zero lost transactions: every committed CICS transaction is recorded in the CICS system log and can be forward-recovered from the last VSAM backup.
Second, the batch cycle includes explicit checkpoint records that allow restart from any failed step. The checkpoint file records the last successfully processed transaction ID, the running totals for all accumulators, and the DB2 commit point. If the batch cycle fails at 3:00 AM with 150,000 of 200,000 transactions posted, the restart logic reads the checkpoint, repositions the input file, and resumes from transaction 150,001.
Performance Monitoring
The team implements a performance monitoring framework that tracks key metrics in real-time:
01 WS-PERF-METRICS.
05 PERF-TXN-COUNT PIC 9(09) VALUE ZERO.
05 PERF-TXN-TOTAL-MS PIC 9(09) VALUE ZERO.
05 PERF-TXN-MAX-MS PIC 9(05) VALUE ZERO.
05 PERF-DB2-CALLS PIC 9(09) VALUE ZERO.
05 PERF-DB2-TOTAL-MS PIC 9(09) VALUE ZERO.
05 PERF-VSAM-READS PIC 9(09) VALUE ZERO.
05 PERF-VSAM-WRITES PIC 9(09) VALUE ZERO.
05 PERF-LAST-REPORT-TIME PIC 9(08) VALUE ZERO.
Every 1,000 transactions, the batch program writes a performance record showing throughput (transactions per second), average DB2 call time, and VSAM I/O rates. Operations staff monitor these metrics to detect degradation before it becomes a batch window overrun.
Results and Lessons Learned
The multi-branch scaling project was completed in fourteen months. The key results were:
Batch window reduction: The parallel processing design reduced the projected 580-minute batch cycle to 185 minutes for 200,000 transactions, well within the 240-minute target. The account-range partitioning of the posting step provided the largest improvement, reducing posting time from 300 minutes to 80 minutes across four parallel instances.
Online performance: CICS response times remained under 0.5 seconds for all transaction types with 200 concurrent users. The multi-AOR architecture distributed the load effectively, and the branch-level transaction file writes eliminated contention on the single daily file.
Data integrity: The inter-branch reconciliation program caught three discrepancies during the first month of multi-branch operation, all caused by timing differences between CICS commit and DB2 commit. The team resolved this by implementing two-phase commit using CICS's Resource Recovery Services (RRS).
Lessons learned: The most valuable architectural decision was partitioning the batch processing by account range rather than by branch. Branch-based partitioning would have been simpler to implement but would have failed the first time a customer had transactions from multiple branches in the same day. Account-range partitioning guaranteed that all transactions for a given account were processed by the same instance, eliminating the need for cross-partition coordination.
The most underestimated challenge was report modification. Every existing report needed a branch-code column and branch-level subtotals. What appeared to be a simple field addition cascaded into changes to 12 report programs, 8 copybooks, and 6 JCL procedures. The lesson: even a single new data dimension touches every layer of a banking application.
Discussion Questions
-
The case study chose account-range partitioning over branch-based partitioning for the posting step. Under what circumstances would branch-based partitioning be the better choice?
-
The inter-branch reconciliation program runs at the end of the batch cycle. What would happen if it were run in real-time as each transfer is posted? What are the trade-offs?
-
As the bank grows to 10 or 20 branches, what additional scalability challenges would emerge that this four-branch design does not address?
-
The performance monitoring framework reports metrics every 1,000 transactions. How would you design an alerting system that automatically notifies operations if throughput drops below a threshold?
-
How would the data architecture change if the bank decided to allow customers to have different interest rates at different branches (e.g., a promotional rate for the new branch)?