> "Any developer can process one file. The real skill is coordinating six files that have to agree with each other at the end of the run."
In This Chapter
- 14.1 Why Multi-File Processing Matters
- 14.2 Multi-File Processing Fundamentals
- 14.3 The Balanced-Line Algorithm
- 14.4 Multiple Input File Merge
- 14.5 File Comparison and Reconciliation
- 14.6 Control Break Processing
- 14.7 MedClaim Case: Matching Claims Against Provider and Member Files
- 14.8 Checkpoint/Restart for Large File Processing
- 14.9 Handling Variable-Length Records in Multi-File Processing
- 14.10 The Human Factor: Maintaining File Processing Code
- 14.11 Error Recovery Patterns in Multi-File Processing
- 14.12 Try It Yourself: Hands-On Exercises
- 14.13 Common Mistakes in Multi-File Processing
- 14.14 Putting It All Together: The Daily Batch Cycle
- 14.15 Performance Considerations for Multi-File Processing
- 14.16 Chapter Summary
Chapter 14: Advanced File Techniques
"Any developer can process one file. The real skill is coordinating six files that have to agree with each other at the end of the run." — James Okafor, during a post-mortem on a reconciliation failure
Chapters 11, 12, and 13 taught you how to process individual files — sequential, indexed, and relative. But enterprise batch systems rarely work with a single file in isolation. A typical production job reads transactions from one file, updates a master file, writes rejected records to another file, produces a report, logs errors, and generates a control file for the next job in the chain. When multiple files must be processed together — matched, merged, reconciled, or synchronized — the patterns become more complex and the consequences of errors become severe.
This chapter covers the essential multi-file processing techniques that form the backbone of enterprise COBOL batch systems. You will learn the balanced-line algorithm for master-transaction updates, multi-file merge patterns, control break processing for report generation, and checkpoint/restart for long-running jobs. These are not academic exercises — they are the patterns running inside the banking, insurance, healthcare, and government systems that process trillions of dollars every day.
14.1 Why Multi-File Processing Matters
Consider GlobalBank's daily batch cycle. The TXN-DAILY file contains the day's transactions — deposits, withdrawals, transfers, fee assessments, interest postings. The ACCT-MASTER file contains the current state of every account. Every night, these two files must be brought into agreement: each transaction applied to the correct account, the master file updated, exception records written for any mismatches, and a summary report produced for the operations team.
This is the master-transaction update pattern — arguably the most important pattern in all of COBOL batch processing. Getting it right means accounts balance to the penny. Getting it wrong means incorrect balances, regulatory violations, and very unhappy customers.
💡 Why This Still Matters: Even in organizations that have moved much of their processing to databases (DB2, Oracle), batch file processing remains essential for high-volume data feeds, ETL (Extract-Transform-Load) pipelines, end-of-day reconciliation, and inter-system interfaces. The patterns in this chapter apply regardless of whether the files are VSAM, sequential, or database extracts.
14.2 Multi-File Processing Fundamentals
14.2.1 The Basic Pattern
Most multi-file processing follows this structure:
Input File A (sorted by key) ──┐
├── Processing Logic ──> Output File(s)
Input File B (sorted by key) ──┘
The critical prerequisite: all input files must be sorted on the same key. If the transaction file is sorted by account number and the master file is sorted by account number, we can process both files in a single pass through each, comparing keys as we go. This is enormously efficient — each file is read exactly once.
14.2.2 Key Comparison Logic
The heart of multi-file processing is comparing keys from different files:
EVALUATE TRUE
WHEN MASTER-KEY < TXN-KEY
* Master record with no matching transaction
PERFORM PROCESS-MASTER-ONLY
WHEN MASTER-KEY = TXN-KEY
* Match — apply transaction to master
PERFORM PROCESS-MATCH
WHEN MASTER-KEY > TXN-KEY
* Transaction with no matching master
PERFORM PROCESS-TXN-ONLY
END-EVALUATE
This three-way comparison drives all of the patterns in this chapter. Understand it deeply, and the rest follows naturally.
14.2.3 High-Values Sentinel Technique
When one file reaches end-of-file before the other, we need a way to ensure the remaining records from the other file are processed. The standard technique is to set the key from the exhausted file to HIGH-VALUES (the highest possible value), which forces all subsequent comparisons to process from the other file:
READ-MASTER.
READ MASTER-FILE
AT END
MOVE HIGH-VALUES TO MASTER-KEY
SET WS-MASTER-EOF TO TRUE
NOT AT END
ADD 1 TO WS-MASTER-READS
END-READ.
READ-TXN.
READ TXN-FILE
AT END
MOVE HIGH-VALUES TO TXN-KEY
SET WS-TXN-EOF TO TRUE
NOT AT END
ADD 1 TO WS-TXN-READS
END-READ.
When both keys equal HIGH-VALUES, all records have been processed:
PERFORM UNTIL MASTER-KEY = HIGH-VALUES
AND TXN-KEY = HIGH-VALUES
PERFORM PROCESS-PAIR
END-PERFORM
⚠️ Critical Detail: HIGH-VALUES works because it is the highest possible collating value (X'FF' in EBCDIC, X'FF' in ASCII). Any real key value will always be less than HIGH-VALUES. This sentinel technique is so fundamental that you will see it in virtually every multi-file COBOL program.
14.3 The Balanced-Line Algorithm
The balanced-line algorithm is the gold standard for master-transaction file updates. It handles all possible scenarios: master-only records (no transactions), matched records (one or more transactions for a master), and transaction-only records (no matching master — typically new accounts or errors).
14.3.1 Algorithm Overview
1. Read first record from master file
2. Read first record from transaction file
3. Loop until both files exhausted:
a. If master key < txn key:
- Write master record unchanged to new master
- Read next master record
b. If master key = txn key:
- Apply all transactions for this key to the master record
- Write updated master to new master
- Read next master record
- (txn key advances during the apply step)
c. If master key > txn key:
- Transaction has no master — handle as add or error
- Read next transaction record
4. Close all files
The "balanced-line" name comes from the fact that the algorithm keeps the two file positions "in balance" — advancing whichever file has the lower key, so neither gets too far ahead of the other.
14.3.2 Why a New Master File?
Notice that the algorithm writes to a new master file, not updating the old one in place. This is a deliberate design choice:
- Recovery: If the job fails midway, the old master is intact. Restart from scratch.
- Audit trail: The old master can be compared with the new master to verify changes.
- Simplicity: Sequential writing is simpler and faster than random updates.
- Consistency: The old master represents "yesterday's truth" and the new master represents "today's truth."
📊 Industry Practice: This "old master + transactions = new master" pattern is sometimes called the grandfather-father-son backup scheme. The "grandfather" is the master from two days ago, the "father" is yesterday's master, and the "son" is today's. If today's run fails, you can recreate from the father. If the father is corrupted, you can recreate from the grandfather. This scheme has kept banks running reliably since the 1960s.
14.3.3 Complete Balanced-Line Implementation
Here is the complete GlobalBank daily master file update — the program that applies TXN-DAILY to ACCT-MASTER:
IDENTIFICATION DIVISION.
PROGRAM-ID. DAILY-UPDATE.
*=============================================================*
* DAILY-UPDATE: Apply daily transactions to account master *
* GlobalBank Core Banking System *
* *
* Balanced-line algorithm: *
* OLD-MASTER + TXN-DAILY = NEW-MASTER + EXCEPTION-FILE *
* *
* Prerequisites: *
* - OLD-MASTER sorted by ACCT-NUMBER *
* - TXN-DAILY sorted by TXN-ACCT-NUMBER *
* *
* Transaction types: *
* 'A' = Add new account *
* 'D' = Deposit *
* 'W' = Withdrawal *
* 'C' = Close account *
*=============================================================*
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.
SELECT OLD-MASTER-FILE
ASSIGN TO OLDMAST
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-OM-STATUS.
SELECT TXN-DAILY-FILE
ASSIGN TO TXNDAILY
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-TD-STATUS.
SELECT NEW-MASTER-FILE
ASSIGN TO NEWMAST
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-NM-STATUS.
SELECT EXCEPTION-FILE
ASSIGN TO EXCEPTF
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-EX-STATUS.
SELECT REPORT-FILE
ASSIGN TO RPTFILE
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-RP-STATUS.
DATA DIVISION.
FILE SECTION.
FD OLD-MASTER-FILE.
01 OLD-MASTER-RECORD.
05 OM-ACCT-NUMBER PIC X(10).
05 OM-HOLDER-NAME PIC X(40).
05 OM-ACCT-TYPE PIC X(02).
05 OM-BALANCE PIC S9(11)V99 COMP-3.
05 OM-STATUS-CODE PIC X(02).
05 OM-LAST-TXN-DATE PIC 9(08).
05 FILLER PIC X(87).
FD TXN-DAILY-FILE.
01 TXN-DAILY-RECORD.
05 TD-ACCT-NUMBER PIC X(10).
05 TD-TXN-TYPE PIC X(01).
88 TD-ADD VALUE 'A'.
88 TD-DEPOSIT VALUE 'D'.
88 TD-WITHDRAWAL VALUE 'W'.
88 TD-CLOSE VALUE 'C'.
05 TD-TXN-AMOUNT PIC S9(11)V99 COMP-3.
05 TD-TXN-DATE PIC 9(08).
05 TD-DESCRIPTION PIC X(30).
05 TD-NEW-NAME PIC X(40).
05 TD-NEW-TYPE PIC X(02).
05 FILLER PIC X(56).
FD NEW-MASTER-FILE.
01 NEW-MASTER-RECORD PIC X(150).
FD EXCEPTION-FILE.
01 EXCEPTION-RECORD.
05 EX-ACCT-NUMBER PIC X(10).
05 EX-TXN-TYPE PIC X(01).
05 EX-REASON-CODE PIC X(04).
05 EX-DESCRIPTION PIC X(60).
05 FILLER PIC X(75).
FD REPORT-FILE.
01 REPORT-RECORD PIC X(132).
WORKING-STORAGE SECTION.
01 WS-OM-STATUS PIC XX.
01 WS-TD-STATUS PIC XX.
01 WS-NM-STATUS PIC XX.
01 WS-EX-STATUS PIC XX.
01 WS-RP-STATUS PIC XX.
01 WS-COUNTERS.
05 WS-MASTER-READS PIC 9(09) VALUE ZERO.
05 WS-TXN-READS PIC 9(09) VALUE ZERO.
05 WS-MASTERS-WRITTEN PIC 9(09) VALUE ZERO.
05 WS-MASTERS-UNCHANGED PIC 9(09) VALUE ZERO.
05 WS-MASTERS-UPDATED PIC 9(09) VALUE ZERO.
05 WS-ACCOUNTS-ADDED PIC 9(09) VALUE ZERO.
05 WS-ACCOUNTS-CLOSED PIC 9(09) VALUE ZERO.
05 WS-DEPOSITS PIC 9(09) VALUE ZERO.
05 WS-WITHDRAWALS PIC 9(09) VALUE ZERO.
05 WS-EXCEPTIONS PIC 9(09) VALUE ZERO.
05 WS-DEPOSIT-TOTAL PIC S9(13)V99 COMP-3
VALUE ZERO.
05 WS-WITHDRAWAL-TOTAL PIC S9(13)V99 COMP-3
VALUE ZERO.
01 WS-WORK-RECORD PIC X(150).
01 WS-WORK-FIELDS REDEFINES WS-WORK-RECORD.
05 WK-ACCT-NUMBER PIC X(10).
05 WK-HOLDER-NAME PIC X(40).
05 WK-ACCT-TYPE PIC X(02).
05 WK-BALANCE PIC S9(11)V99 COMP-3.
05 WK-STATUS-CODE PIC X(02).
05 WK-LAST-TXN-DATE PIC 9(08).
05 FILLER PIC X(87).
01 WS-MASTER-EOF-FLAG PIC X VALUE 'N'.
88 WS-MASTER-EOF VALUE 'Y'.
01 WS-TXN-EOF-FLAG PIC X VALUE 'N'.
88 WS-TXN-EOF VALUE 'Y'.
01 WS-BOTH-DONE-FLAG PIC X VALUE 'N'.
88 WS-BOTH-DONE VALUE 'Y'.
01 WS-TODAYS-DATE PIC 9(08).
01 WS-RPT-LINE PIC X(132).
01 WS-DISPLAY-BAL PIC Z(10)9.99-.
01 WS-DISPLAY-AMT PIC Z(12)9.99-.
PROCEDURE DIVISION.
0000-MAIN.
PERFORM 1000-INITIALIZE
PERFORM 2000-PROCESS-FILES
UNTIL WS-BOTH-DONE
PERFORM 8000-WRITE-SUMMARY
PERFORM 9000-TERMINATE
STOP RUN.
1000-INITIALIZE.
ACCEPT WS-TODAYS-DATE FROM DATE YYYYMMDD
OPEN INPUT OLD-MASTER-FILE
OPEN INPUT TXN-DAILY-FILE
OPEN OUTPUT NEW-MASTER-FILE
OPEN OUTPUT EXCEPTION-FILE
OPEN OUTPUT REPORT-FILE
* Prime both files
PERFORM 2100-READ-OLD-MASTER
PERFORM 2200-READ-TXN.
2000-PROCESS-FILES.
* The core balanced-line comparison
EVALUATE TRUE
WHEN OM-ACCT-NUMBER < TD-ACCT-NUMBER
* Master only — no transactions
PERFORM 3000-MASTER-ONLY
WHEN OM-ACCT-NUMBER = TD-ACCT-NUMBER
* Match — apply transactions
PERFORM 4000-MATCHED
WHEN OM-ACCT-NUMBER > TD-ACCT-NUMBER
* Transaction only — add or error
PERFORM 5000-TXN-ONLY
END-EVALUATE
* Check if both files exhausted
IF OM-ACCT-NUMBER = HIGH-VALUES
AND TD-ACCT-NUMBER = HIGH-VALUES
SET WS-BOTH-DONE TO TRUE
END-IF.
2100-READ-OLD-MASTER.
READ OLD-MASTER-FILE
AT END
MOVE HIGH-VALUES TO OM-ACCT-NUMBER
SET WS-MASTER-EOF TO TRUE
NOT AT END
ADD 1 TO WS-MASTER-READS
END-READ.
2200-READ-TXN.
READ TXN-DAILY-FILE
AT END
MOVE HIGH-VALUES TO TD-ACCT-NUMBER
SET WS-TXN-EOF TO TRUE
NOT AT END
ADD 1 TO WS-TXN-READS
END-READ.
3000-MASTER-ONLY.
* Master record with no transactions — copy unchanged
WRITE NEW-MASTER-RECORD FROM OLD-MASTER-RECORD
ADD 1 TO WS-MASTERS-WRITTEN
ADD 1 TO WS-MASTERS-UNCHANGED
PERFORM 2100-READ-OLD-MASTER.
4000-MATCHED.
* Copy master to work area for modification
MOVE OLD-MASTER-RECORD TO WS-WORK-RECORD
* Apply all transactions for this account
PERFORM 4100-APPLY-TRANSACTIONS
UNTIL TD-ACCT-NUMBER NOT = WK-ACCT-NUMBER
* Write updated master
WRITE NEW-MASTER-RECORD FROM WS-WORK-RECORD
ADD 1 TO WS-MASTERS-WRITTEN
ADD 1 TO WS-MASTERS-UPDATED
* Advance master
PERFORM 2100-READ-OLD-MASTER.
4100-APPLY-TRANSACTIONS.
EVALUATE TRUE
WHEN TD-DEPOSIT
ADD TD-TXN-AMOUNT TO WK-BALANCE
MOVE TD-TXN-DATE TO WK-LAST-TXN-DATE
ADD 1 TO WS-DEPOSITS
ADD TD-TXN-AMOUNT TO WS-DEPOSIT-TOTAL
WHEN TD-WITHDRAWAL
IF TD-TXN-AMOUNT > WK-BALANCE
* Insufficient funds — exception
MOVE TD-ACCT-NUMBER TO EX-ACCT-NUMBER
MOVE TD-TXN-TYPE TO EX-TXN-TYPE
MOVE 'INSF' TO EX-REASON-CODE
STRING 'Insufficient funds. Balance: '
WK-BALANCE
' Requested: ' TD-TXN-AMOUNT
DELIMITED BY SIZE
INTO EX-DESCRIPTION
WRITE EXCEPTION-RECORD
ADD 1 TO WS-EXCEPTIONS
ELSE
SUBTRACT TD-TXN-AMOUNT FROM WK-BALANCE
MOVE TD-TXN-DATE TO WK-LAST-TXN-DATE
ADD 1 TO WS-WITHDRAWALS
ADD TD-TXN-AMOUNT
TO WS-WITHDRAWAL-TOTAL
END-IF
WHEN TD-CLOSE
IF WK-BALANCE NOT = ZERO
MOVE TD-ACCT-NUMBER TO EX-ACCT-NUMBER
MOVE TD-TXN-TYPE TO EX-TXN-TYPE
MOVE 'CBAL' TO EX-REASON-CODE
STRING 'Close with balance: '
WK-BALANCE
DELIMITED BY SIZE
INTO EX-DESCRIPTION
WRITE EXCEPTION-RECORD
ADD 1 TO WS-EXCEPTIONS
ELSE
MOVE 'CL' TO WK-STATUS-CODE
MOVE TD-TXN-DATE TO WK-LAST-TXN-DATE
ADD 1 TO WS-ACCOUNTS-CLOSED
END-IF
WHEN TD-ADD
* ADD transaction for existing account — error
MOVE TD-ACCT-NUMBER TO EX-ACCT-NUMBER
MOVE TD-TXN-TYPE TO EX-TXN-TYPE
MOVE 'DUPE' TO EX-REASON-CODE
STRING 'Add for existing account'
DELIMITED BY SIZE
INTO EX-DESCRIPTION
WRITE EXCEPTION-RECORD
ADD 1 TO WS-EXCEPTIONS
WHEN OTHER
MOVE TD-ACCT-NUMBER TO EX-ACCT-NUMBER
MOVE TD-TXN-TYPE TO EX-TXN-TYPE
MOVE 'ITYP' TO EX-REASON-CODE
STRING 'Invalid txn type: ' TD-TXN-TYPE
DELIMITED BY SIZE
INTO EX-DESCRIPTION
WRITE EXCEPTION-RECORD
ADD 1 TO WS-EXCEPTIONS
END-EVALUATE
* Read next transaction
PERFORM 2200-READ-TXN.
5000-TXN-ONLY.
* Transaction with no matching master
IF TD-ADD
* New account — create master record
INITIALIZE WS-WORK-RECORD
MOVE TD-ACCT-NUMBER TO WK-ACCT-NUMBER
MOVE TD-NEW-NAME TO WK-HOLDER-NAME
MOVE TD-NEW-TYPE TO WK-ACCT-TYPE
MOVE ZERO TO WK-BALANCE
MOVE 'AC' TO WK-STATUS-CODE
MOVE TD-TXN-DATE TO WK-LAST-TXN-DATE
WRITE NEW-MASTER-RECORD FROM WS-WORK-RECORD
ADD 1 TO WS-MASTERS-WRITTEN
ADD 1 TO WS-ACCOUNTS-ADDED
ELSE
* Non-add transaction for non-existent account
MOVE TD-ACCT-NUMBER TO EX-ACCT-NUMBER
MOVE TD-TXN-TYPE TO EX-TXN-TYPE
MOVE 'NMST' TO EX-REASON-CODE
STRING 'No master record for account'
DELIMITED BY SIZE INTO EX-DESCRIPTION
WRITE EXCEPTION-RECORD
ADD 1 TO WS-EXCEPTIONS
END-IF
PERFORM 2200-READ-TXN.
8000-WRITE-SUMMARY.
MOVE SPACES TO WS-RPT-LINE
STRING '=== DAILY UPDATE SUMMARY ==='
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING 'Date: ' WS-TODAYS-DATE
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
MOVE SPACES TO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING 'Master records read: '
WS-MASTER-READS
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING 'Transactions read: '
WS-TXN-READS
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING 'Masters written: '
WS-MASTERS-WRITTEN
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING ' Unchanged: '
WS-MASTERS-UNCHANGED
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING ' Updated: '
WS-MASTERS-UPDATED
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING ' New accounts added: '
WS-ACCOUNTS-ADDED
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING ' Accounts closed: '
WS-ACCOUNTS-CLOSED
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING 'Deposits: '
WS-DEPOSITS
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
MOVE WS-DEPOSIT-TOTAL TO WS-DISPLAY-AMT
STRING 'Total deposited: $'
WS-DISPLAY-AMT
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING 'Withdrawals: '
WS-WITHDRAWALS
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
MOVE WS-WITHDRAWAL-TOTAL TO WS-DISPLAY-AMT
STRING 'Total withdrawn: $'
WS-DISPLAY-AMT
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING 'Exceptions: '
WS-EXCEPTIONS
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
* Reconciliation check
MOVE SPACES TO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING '--- RECONCILIATION ---'
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE
STRING 'New master count should equal: '
'Old master + Adds - Closes'
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-RECORD FROM WS-RPT-LINE.
9000-TERMINATE.
CLOSE OLD-MASTER-FILE
TXN-DAILY-FILE
NEW-MASTER-FILE
EXCEPTION-FILE
REPORT-FILE.
🔴 Production War Story: Derek Washington once asked Maria Chen why the program writes unchanged master records to the new file. "Why not just copy the old master and update only the changed records?" Maria's answer: "Because then you need to handle random updates on a sequential file, which means either an indexed file or a complex in-place update. The sequential copy-everything approach is simpler, faster for batch, and gives you a clean new master every day. I've seen 'optimize by updating in place' approaches fail in ways that corrupt the entire master file. Keep it simple."
14.3.4 Handling Multiple Transactions per Master Record
One of the most critical aspects of the balanced-line algorithm is correctly handling multiple transactions for the same master record. Consider an account that receives three transactions in one day: a deposit, a withdrawal, and a fee assessment. All three have the same account number and must be applied to the same master record.
The key insight: the transaction read loop (4100-APPLY-TRANSACTIONS) continues reading transactions as long as the transaction key matches the current master key. It modifies the work record in memory — not on disk — accumulating all changes. Only after all matching transactions have been applied does the program write the updated master.
* This is the inner loop of section 4000-MATCHED:
PERFORM 4100-APPLY-TRANSACTIONS
UNTIL TD-ACCT-NUMBER NOT = WK-ACCT-NUMBER
* At this point, ALL transactions for this account
* have been applied. The work record reflects the
* cumulative effect of all changes.
What if the transactions are contradictory? For example, a deposit of $1,000 followed by a close? The algorithm applies them in order: the deposit increases the balance to $1,000, then the close attempt fails because the balance is not zero (generating an exception). This sequential application preserves the business meaning of the transaction stream.
⚠️ Sort Order Within Key: When multiple transactions share the same key, their relative order within the sorted file matters. Most shops use a secondary sort key — typically a timestamp or sequence number — to ensure transactions are applied in chronological order. The JCL SORT statement handles this:
//SORT EXEC PGM=SORT
//SORTIN DD DSN=GLOBANK.TXN.DAILY.RAW,DISP=SHR
//SORTOUT DD DSN=GLOBANK.TXN.DAILY.SORTED,DISP=(NEW,CATLG),...
//SYSIN DD *
SORT FIELDS=(1,10,CH,A, Account number ascending
20,8,CH,A) Transaction date ascending
/*
14.3.5 Error Recovery in the Balanced-Line Algorithm
What happens when an error occurs midway through applying transactions to a master record? Consider this scenario: account ACC0001234 has 5 transactions. The first 3 apply successfully (two deposits and a withdrawal). The 4th transaction (another withdrawal) fails due to insufficient funds. What should happen to the 5th transaction?
Different shops handle this differently:
Option A — Skip the failing transaction, continue: Apply transactions 1-3, write the exception for transaction 4, then process transaction 5 normally. This is the most common approach and is what our DAILY-UPDATE program does.
Option B — Skip all remaining transactions for this key: After the first error, write exceptions for transactions 4 and 5. Rationale: the balance after transaction 3 may not be what the business expected (since transaction 4 was supposed to happen), so transaction 5's context is wrong.
Option C — Rollback all transactions for this key: Discard all changes and write the master record unchanged. Write all 5 transactions to an exception file for manual review. This is the safest approach but the most operationally expensive.
Maria Chen uses Option A at GlobalBank: "Each transaction is independent. If a withdrawal fails due to insufficient funds, that doesn't mean the next deposit should be rejected. We log the exception and move on."
James Okafor uses Option B at MedClaim: "In claims processing, transactions for the same claim are sequentially dependent — a price adjustment depends on the base price being correct. If the base price fails, the adjustment is meaningless."
The choice depends on the business domain. Understand the business rules before coding the error handling.
14.3.6 The Balanced-Line Algorithm Step by Step
Let us trace through a small example to make the algorithm concrete:
Old Master (sorted by key):
ACC001 Alice $1,000
ACC003 Charlie $3,000
ACC005 Eve $5,000
Transactions (sorted by key):
ACC001 D $500 (Deposit $500)
ACC002 A (Add new account for Bob)
ACC003 W $1,000 (Withdraw $1,000)
ACC003 W $500 (Withdraw $500)
ACC004 D $100 (Deposit — no master!)
Iteration 1: Master ACC001 = Txn ACC001 → MATCHED - Apply deposit $500: Alice's balance becomes $1,500 - Read next txn: ACC002 - ACC002 ≠ ACC001, so write updated master for Alice - Read next master: ACC003
Iteration 2: Master ACC003 > Txn ACC002 → TXN-ONLY - ACC002 is type 'A' (Add) → create new master for Bob, balance $0 - Read next txn: ACC003
Iteration 3: Master ACC003 = Txn ACC003 → MATCHED - Apply withdrawal $1,000: Charlie's balance becomes $2,000 - Read next txn: ACC003 (another txn for same account) - Apply withdrawal $500: Charlie's balance becomes $1,500 - Read next txn: ACC004 - ACC004 ≠ ACC003, so write updated master for Charlie - Read next master: ACC005
Iteration 4: Master ACC005 > Txn ACC004 → TXN-ONLY - ACC004 is type 'D' (Deposit), not 'A' → no master exists → EXCEPTION - Read next txn: HIGH-VALUES (EOF)
Iteration 5: Master ACC005, Txn HIGH-VALUES → MASTER-ONLY - Copy Eve's record unchanged to new master - Read next master: HIGH-VALUES (EOF)
Both at HIGH-VALUES: Loop ends.
New Master: ACC001 Alice $1,500 (updated) ACC002 Bob $0 (new) ACC003 Charlie $1,500 (updated) ACC005 Eve $5,000 (unchanged)
Exception File: ACC004 D $100 NMST (No master record)
Summary: Masters read: 3, Txns read: 5, Masters written: 4 Unchanged: 1, Updated: 2, Added: 1, Exceptions: 1
This trace confirms: 3 (old master) + 1 (add) = 4 (new master). The reconciliation arithmetic checks out.
14.4 Multiple Input File Merge
Sometimes you need to merge records from multiple sorted files into a single sorted output. This extends the balanced-line concept to three or more files.
14.4.1 Two-File Merge
A simple merge of two sorted files:
MERGE-TWO-FILES.
* Both files sorted by the same key
PERFORM READ-FILE-A
PERFORM READ-FILE-B
PERFORM UNTIL FILE-A-KEY = HIGH-VALUES
AND FILE-B-KEY = HIGH-VALUES
IF FILE-A-KEY <= FILE-B-KEY
WRITE OUTPUT-RECORD FROM FILE-A-RECORD
PERFORM READ-FILE-A
ELSE
WRITE OUTPUT-RECORD FROM FILE-B-RECORD
PERFORM READ-FILE-B
END-IF
END-PERFORM.
Note the <= comparison: when keys are equal, File A's record comes first. This is a design choice — in a merge, you need to decide which input takes priority for matching keys.
14.4.2 Three-File Merge
MedClaim processes claims from three sources: electronic claims (EDI), paper claims (keyed), and resubmissions. All three must be merged into a single claim intake file:
MERGE-THREE-SOURCES.
PERFORM READ-EDI-FILE
PERFORM READ-PAPER-FILE
PERFORM READ-RESUB-FILE
PERFORM UNTIL EDI-KEY = HIGH-VALUES
AND PAPER-KEY = HIGH-VALUES
AND RESUB-KEY = HIGH-VALUES
* Find the lowest key among three files
IF EDI-KEY <= PAPER-KEY
AND EDI-KEY <= RESUB-KEY
WRITE MERGED-RECORD FROM EDI-RECORD
PERFORM READ-EDI-FILE
ELSE IF PAPER-KEY <= EDI-KEY
AND PAPER-KEY <= RESUB-KEY
WRITE MERGED-RECORD FROM PAPER-RECORD
PERFORM READ-PAPER-FILE
ELSE
WRITE MERGED-RECORD FROM RESUB-RECORD
PERFORM READ-RESUB-FILE
END-IF
END-PERFORM.
14.4.3 Generalizing to N-Way Merge
For more than three files, the comparison logic becomes unwieldy. A cleaner approach uses a merge key table:
01 WS-MERGE-TABLE.
05 WS-MERGE-ENTRY OCCURS 5 TIMES.
10 WS-MERGE-KEY PIC X(10).
10 WS-MERGE-SOURCE PIC 9(01).
10 WS-MERGE-ACTIVE PIC X VALUE 'Y'.
88 WS-ENTRY-ACTIVE VALUE 'Y'.
88 WS-ENTRY-EOF VALUE 'N'.
N-WAY-MERGE.
* Prime all files
PERFORM VARYING WS-IDX FROM 1 BY 1
UNTIL WS-IDX > WS-FILE-COUNT
PERFORM READ-FILE-N
END-PERFORM
* Main merge loop
PERFORM UNTIL WS-ALL-EOF
* Find entry with lowest key
MOVE HIGH-VALUES TO WS-LOW-KEY
PERFORM VARYING WS-IDX FROM 1 BY 1
UNTIL WS-IDX > WS-FILE-COUNT
IF WS-ENTRY-ACTIVE(WS-IDX)
AND WS-MERGE-KEY(WS-IDX) < WS-LOW-KEY
MOVE WS-MERGE-KEY(WS-IDX)
TO WS-LOW-KEY
MOVE WS-IDX TO WS-LOW-SOURCE
END-IF
END-PERFORM
* Write the record from the lowest source
PERFORM WRITE-FROM-SOURCE
* Read next record from that source
PERFORM READ-FILE-N
END-PERFORM.
14.5 File Comparison and Reconciliation
Reconciliation programs compare two files that should contain the same data (or related data) and identify discrepancies. This is critical for auditing and data integrity.
14.5.1 Basic File Comparison
IDENTIFICATION DIVISION.
PROGRAM-ID. FILE-COMPARE.
*=============================================================*
* FILE-COMPARE: Compare two sorted files, report differences *
*=============================================================*
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.
SELECT FILE-A
ASSIGN TO FILEA
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-FA-STATUS.
SELECT FILE-B
ASSIGN TO FILEB
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-FB-STATUS.
SELECT DIFF-FILE
ASSIGN TO DIFFILE
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-DF-STATUS.
DATA DIVISION.
FILE SECTION.
FD FILE-A.
01 FILE-A-RECORD.
05 FA-KEY PIC X(10).
05 FA-DATA PIC X(140).
FD FILE-B.
01 FILE-B-RECORD.
05 FB-KEY PIC X(10).
05 FB-DATA PIC X(140).
FD DIFF-FILE.
01 DIFF-RECORD PIC X(160).
WORKING-STORAGE SECTION.
01 WS-FA-STATUS PIC XX.
01 WS-FB-STATUS PIC XX.
01 WS-DF-STATUS PIC XX.
01 WS-COUNTERS.
05 WS-MATCHES PIC 9(09) VALUE ZERO.
05 WS-MISMATCHES PIC 9(09) VALUE ZERO.
05 WS-A-ONLY PIC 9(09) VALUE ZERO.
05 WS-B-ONLY PIC 9(09) VALUE ZERO.
01 WS-DIFF-LINE PIC X(160).
PROCEDURE DIVISION.
0000-MAIN.
OPEN INPUT FILE-A
OPEN INPUT FILE-B
OPEN OUTPUT DIFF-FILE
PERFORM READ-A
PERFORM READ-B
PERFORM UNTIL FA-KEY = HIGH-VALUES
AND FB-KEY = HIGH-VALUES
EVALUATE TRUE
WHEN FA-KEY < FB-KEY
STRING 'A-ONLY: ' FA-KEY ' '
FA-DATA(1:40)
DELIMITED BY SIZE
INTO WS-DIFF-LINE
WRITE DIFF-RECORD FROM WS-DIFF-LINE
ADD 1 TO WS-A-ONLY
PERFORM READ-A
WHEN FA-KEY = FB-KEY
IF FA-DATA = FB-DATA
ADD 1 TO WS-MATCHES
ELSE
STRING 'MISMATCH: ' FA-KEY
DELIMITED BY SIZE
INTO WS-DIFF-LINE
WRITE DIFF-RECORD FROM WS-DIFF-LINE
STRING ' FILE-A: '
FA-DATA(1:60)
DELIMITED BY SIZE
INTO WS-DIFF-LINE
WRITE DIFF-RECORD FROM WS-DIFF-LINE
STRING ' FILE-B: '
FB-DATA(1:60)
DELIMITED BY SIZE
INTO WS-DIFF-LINE
WRITE DIFF-RECORD FROM WS-DIFF-LINE
ADD 1 TO WS-MISMATCHES
END-IF
PERFORM READ-A
PERFORM READ-B
WHEN FA-KEY > FB-KEY
STRING 'B-ONLY: ' FB-KEY ' '
FB-DATA(1:40)
DELIMITED BY SIZE
INTO WS-DIFF-LINE
WRITE DIFF-RECORD FROM WS-DIFF-LINE
ADD 1 TO WS-B-ONLY
PERFORM READ-B
END-EVALUATE
END-PERFORM
DISPLAY 'Matches: ' WS-MATCHES
DISPLAY 'Mismatches: ' WS-MISMATCHES
DISPLAY 'A-only: ' WS-A-ONLY
DISPLAY 'B-only: ' WS-B-ONLY
CLOSE FILE-A FILE-B DIFF-FILE
STOP RUN.
READ-A.
READ FILE-A
AT END
MOVE HIGH-VALUES TO FA-KEY
END-READ.
READ-B.
READ FILE-B
AT END
MOVE HIGH-VALUES TO FB-KEY
END-READ.
14.6 Control Break Processing
Control break processing produces reports with subtotals at each change in a grouping field. When the branch code changes, print branch subtotals. When the region changes, print region subtotals. When the file ends, print grand totals.
14.6.1 Single-Level Control Break
IDENTIFICATION DIVISION.
PROGRAM-ID. BRANCH-RPT.
*=============================================================*
* BRANCH-RPT: Account summary by branch *
* Demonstrates single-level control break processing *
*=============================================================*
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.
SELECT ACCOUNT-FILE
ASSIGN TO ACCTFILE
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-AF-STATUS.
SELECT REPORT-FILE
ASSIGN TO RPTFILE
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-RP-STATUS.
DATA DIVISION.
FILE SECTION.
FD ACCOUNT-FILE.
01 ACCOUNT-RECORD.
05 AR-BRANCH-CODE PIC X(05).
05 AR-ACCT-NUMBER PIC X(10).
05 AR-HOLDER-NAME PIC X(40).
05 AR-BALANCE PIC S9(11)V99 COMP-3.
05 FILLER PIC X(88).
FD REPORT-FILE.
01 REPORT-LINE PIC X(132).
WORKING-STORAGE SECTION.
01 WS-AF-STATUS PIC XX.
01 WS-RP-STATUS PIC XX.
01 WS-PREV-BRANCH PIC X(05).
01 WS-BRANCH-COUNT PIC 9(07) VALUE ZERO.
01 WS-BRANCH-TOTAL PIC S9(13)V99 COMP-3
VALUE ZERO.
01 WS-GRAND-COUNT PIC 9(09) VALUE ZERO.
01 WS-GRAND-TOTAL PIC S9(15)V99 COMP-3
VALUE ZERO.
01 WS-DISPLAY-TOTAL PIC Z(12)9.99-.
01 WS-DISPLAY-COUNT PIC Z(06)9.
01 WS-RPT-LINE PIC X(132).
01 WS-EOF-FLAG PIC X VALUE 'N'.
88 WS-EOF VALUE 'Y'.
PROCEDURE DIVISION.
0000-MAIN.
PERFORM 1000-INITIALIZE
PERFORM 2000-PROCESS-RECORDS
UNTIL WS-EOF
PERFORM 3000-FINAL-BREAK
PERFORM 9000-TERMINATE
STOP RUN.
1000-INITIALIZE.
OPEN INPUT ACCOUNT-FILE
OPEN OUTPUT REPORT-FILE
* Write report header
STRING 'GLOBALBANK ACCOUNT SUMMARY BY BRANCH'
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-LINE FROM WS-RPT-LINE
MOVE SPACES TO WS-RPT-LINE
STRING 'Branch Account Name'
' Balance'
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-LINE FROM WS-RPT-LINE
MOVE ALL '-' TO WS-RPT-LINE
WRITE REPORT-LINE FROM WS-RPT-LINE
* Read first record and save branch as control field
READ ACCOUNT-FILE
AT END
SET WS-EOF TO TRUE
NOT AT END
MOVE AR-BRANCH-CODE TO WS-PREV-BRANCH
END-READ.
2000-PROCESS-RECORDS.
* Check for control break
IF AR-BRANCH-CODE NOT = WS-PREV-BRANCH
PERFORM 2500-BRANCH-BREAK
END-IF
* Process current record
ADD 1 TO WS-BRANCH-COUNT
ADD AR-BALANCE TO WS-BRANCH-TOTAL
MOVE AR-BALANCE TO WS-DISPLAY-TOTAL
STRING AR-BRANCH-CODE ' '
AR-ACCT-NUMBER ' '
AR-HOLDER-NAME ' '
WS-DISPLAY-TOTAL
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-LINE FROM WS-RPT-LINE
* Read next record
READ ACCOUNT-FILE
AT END
SET WS-EOF TO TRUE
END-READ.
2500-BRANCH-BREAK.
* Print subtotals for previous branch
MOVE WS-BRANCH-COUNT TO WS-DISPLAY-COUNT
MOVE WS-BRANCH-TOTAL TO WS-DISPLAY-TOTAL
MOVE SPACES TO WS-RPT-LINE
STRING ' Branch ' WS-PREV-BRANCH
' Total: ' WS-DISPLAY-COUNT ' accounts, $'
WS-DISPLAY-TOTAL
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-LINE FROM WS-RPT-LINE
MOVE SPACES TO WS-RPT-LINE
WRITE REPORT-LINE FROM WS-RPT-LINE
* Roll up to grand totals
ADD WS-BRANCH-COUNT TO WS-GRAND-COUNT
ADD WS-BRANCH-TOTAL TO WS-GRAND-TOTAL
* Reset branch accumulators
MOVE ZERO TO WS-BRANCH-COUNT
MOVE ZERO TO WS-BRANCH-TOTAL
MOVE AR-BRANCH-CODE TO WS-PREV-BRANCH.
3000-FINAL-BREAK.
* Process last branch
PERFORM 2500-BRANCH-BREAK
* Print grand totals
MOVE ALL '=' TO WS-RPT-LINE
WRITE REPORT-LINE FROM WS-RPT-LINE
MOVE WS-GRAND-COUNT TO WS-DISPLAY-COUNT
MOVE WS-GRAND-TOTAL TO WS-DISPLAY-TOTAL
STRING 'GRAND TOTAL: '
WS-DISPLAY-COUNT ' accounts, $'
WS-DISPLAY-TOTAL
DELIMITED BY SIZE INTO WS-RPT-LINE
WRITE REPORT-LINE FROM WS-RPT-LINE.
9000-TERMINATE.
CLOSE ACCOUNT-FILE
REPORT-FILE.
14.6.2 Multi-Level Control Break
Production reports often need multiple break levels. For example: subtotals by branch within region within division. The key rule: break levels must be nested, and a higher-level break triggers all lower-level breaks.
* Multi-level control break hierarchy:
* Division → Region → Branch
*
* When Division changes: print Branch total,
* Region total, Division total
* When Region changes: print Branch total,
* Region total
* When Branch changes: print Branch total
2000-CHECK-BREAKS.
IF AR-DIVISION-CODE NOT = WS-PREV-DIVISION
* Division break triggers all levels
PERFORM 2510-BRANCH-BREAK
PERFORM 2520-REGION-BREAK
PERFORM 2530-DIVISION-BREAK
ELSE IF AR-REGION-CODE NOT = WS-PREV-REGION
* Region break triggers branch break too
PERFORM 2510-BRANCH-BREAK
PERFORM 2520-REGION-BREAK
ELSE IF AR-BRANCH-CODE NOT = WS-PREV-BRANCH
* Branch break only
PERFORM 2510-BRANCH-BREAK
END-IF.
⚠️ Common Pitfall: A frequent bug in multi-level control breaks is forgetting that a higher-level break must trigger all lower-level breaks. If the division changes, you must first close out the current branch, then the current region, then the current division — in that order. Skipping a level produces incorrect subtotals.
14.6.3 The First-Record and Last-Record Problem
Control break processing has two tricky edge cases that cause bugs in many programs:
The first record: When the program reads the first record, there is no "previous" branch to compare against. If you forget to initialize the previous-branch field, the first record always triggers a break — printing a subtotal for an empty group before any records have been processed.
The correct approach: initialize the previous-branch from the first record, BEFORE entering the processing loop:
1000-INITIALIZE.
...
* Read first record and capture initial break value
READ ACCOUNT-FILE
AT END
SET WS-EOF TO TRUE
NOT AT END
MOVE AR-BRANCH-CODE TO WS-PREV-BRANCH
END-READ.
The last group: When the file reaches EOF, the last group's subtotals have not been printed (because there was no change in the break field to trigger them). You must explicitly call the break routine after the main loop:
0000-MAIN.
PERFORM 1000-INITIALIZE
PERFORM 2000-PROCESS-RECORDS
UNTIL WS-EOF
PERFORM 3000-FINAL-BREAK *> Don't forget this!
PERFORM 9000-TERMINATE
STOP RUN.
The 3000-FINAL-BREAK paragraph calls the same break routine(s) used during processing, then prints grand totals. This ensures the last group is always accounted for.
🔴 Production War Story: James Okafor recalls a claims report that was missing the last provider's subtotal for three months before anyone noticed. "The report had 847 providers. Nobody checked that provider 847 was missing from the bottom of the report. The fix was one line — adding the final break call. But the embarrassment lasted much longer."
14.6.4 Page Overflow and Headers
Production reports must handle page breaks properly. A detail line that would print past the bottom of the page must instead trigger a new page with headers:
CHECK-PAGE-OVERFLOW.
ADD 1 TO WS-LINE-COUNT
IF WS-LINE-COUNT > WS-LINES-PER-PAGE
PERFORM WRITE-PAGE-HEADER
END-IF.
WRITE-PAGE-HEADER.
ADD 1 TO WS-PAGE-COUNT
WRITE REPORT-LINE FROM WS-PAGE-HEADER-1
AFTER ADVANCING PAGE
WRITE REPORT-LINE FROM WS-PAGE-HEADER-2
WRITE REPORT-LINE FROM WS-COLUMN-HEADERS
WRITE REPORT-LINE FROM WS-DASH-LINE
MOVE 4 TO WS-LINE-COUNT.
A subtlety: when a control break coincides with a page boundary, should the subtotal print at the bottom of the current page or the top of the next? Most shops prefer subtotals at the bottom of the current page (closing out the group), with the next group starting on a new page. This means checking for page overflow before printing detail lines, but NOT before printing subtotal lines.
14.7 MedClaim Case: Matching Claims Against Provider and Member Files
The MedClaim claims adjudication process matches each claim against two reference files: the PROVIDER-FILE and the MEMBER-FILE. Both the claim and the reference files are sorted differently, so the process uses a combination of sequential and random access.
14.7.1 The Matching Pattern
ADJUDICATE-CLAIM.
* For each claim (read sequentially):
* 1. Look up provider (random access by provider ID)
* 2. Look up member (random access by member ID)
* 3. Cross-validate claim against both
* 4. Write result (approved, denied, or pend)
MOVE CLM-PROVIDER-ID TO PRV-PROVIDER-ID
READ PROVIDER-FILE
KEY IS PRV-PROVIDER-ID
INVALID KEY
MOVE 'PRVNF' TO CLM-DENY-REASON
PERFORM WRITE-DENIAL
GO TO ADJUDICATE-EXIT
END-READ
MOVE CLM-MEMBER-ID TO MBR-MEMBER-ID
READ MEMBER-FILE
KEY IS MBR-MEMBER-ID
INVALID KEY
MOVE 'MBRNF' TO CLM-DENY-REASON
PERFORM WRITE-DENIAL
GO TO ADJUDICATE-EXIT
END-READ
* Both found — cross-validate
PERFORM VALIDATE-PROVIDER-MEMBER-MATCH
PERFORM VALIDATE-COVERAGE-DATES
PERFORM VALIDATE-PROCEDURE-CODES
PERFORM CALCULATE-ALLOWED-AMOUNT
IF CLM-VALID
PERFORM WRITE-APPROVAL
ELSE
PERFORM WRITE-DENIAL
END-IF.
ADJUDICATE-EXIT.
EXIT.
This pattern — sequential input with random lookups against reference files — is one of the most common in enterprise processing. It combines the efficiency of sequential reading for the high-volume claims with the precision of random access for the reference data.
14.8 Checkpoint/Restart for Large File Processing
When a batch job processes millions of records and takes hours to run, a failure midway through means restarting from the beginning — unless the program implements checkpoint/restart.
14.8.1 The Problem
Consider GlobalBank's month-end interest calculation, which processes 4.7 million accounts. If the job fails after processing 3 million records (after 4 hours of runtime), restarting from the beginning wastes 4 hours. Checkpoint/restart solves this by periodically saving the program's state, so it can resume from the last checkpoint rather than the beginning.
14.8.2 Basic Checkpoint Pattern
WORKING-STORAGE SECTION.
01 WS-CHECKPOINT-INTERVAL PIC 9(07) VALUE 10000.
01 WS-RECORDS-SINCE-CKPT PIC 9(07) VALUE ZERO.
01 WS-CHECKPOINT-COUNT PIC 9(05) VALUE ZERO.
01 WS-LAST-KEY-PROCESSED PIC X(10).
01 WS-RESTART-KEY PIC X(10).
01 WS-RESTART-MODE PIC X VALUE 'N'.
88 WS-IS-RESTART VALUE 'Y'.
88 WS-IS-FRESH-START VALUE 'N'.
01 WS-CHECKPOINT-RECORD.
05 CK-PROGRAM-ID PIC X(08) VALUE
'INTCALC'.
05 CK-LAST-KEY PIC X(10).
05 CK-RECORDS-PROCESSED PIC 9(09).
05 CK-TOTAL-INTEREST PIC S9(15)V99 COMP-3.
05 CK-CHECKPOINT-NUMBER PIC 9(05).
05 CK-TIMESTAMP PIC X(26).
PROCEDURE DIVISION.
1000-INITIALIZE.
* Check if this is a restart
PERFORM CHECK-FOR-RESTART
IF WS-IS-RESTART
PERFORM RESTORE-FROM-CHECKPOINT
PERFORM POSITION-TO-RESTART-KEY
ELSE
PERFORM NORMAL-INITIALIZATION
END-IF.
2000-PROCESS-ACCOUNTS.
PERFORM CALCULATE-INTEREST
* Checkpoint logic
ADD 1 TO WS-RECORDS-SINCE-CKPT
MOVE AM-ACCT-NUMBER TO WS-LAST-KEY-PROCESSED
IF WS-RECORDS-SINCE-CKPT >= WS-CHECKPOINT-INTERVAL
PERFORM WRITE-CHECKPOINT
MOVE ZERO TO WS-RECORDS-SINCE-CKPT
END-IF.
WRITE-CHECKPOINT.
ADD 1 TO WS-CHECKPOINT-COUNT
MOVE WS-LAST-KEY-PROCESSED
TO CK-LAST-KEY
MOVE WS-TOTAL-RECORDS-PROCESSED
TO CK-RECORDS-PROCESSED
MOVE WS-TOTAL-INTEREST
TO CK-TOTAL-INTEREST
MOVE WS-CHECKPOINT-COUNT
TO CK-CHECKPOINT-NUMBER
ACCEPT CK-TIMESTAMP FROM DATE YYYYMMDD
WRITE CHECKPOINT-RECORD
FROM WS-CHECKPOINT-RECORD
DISPLAY 'CHECKPOINT ' WS-CHECKPOINT-COUNT
' at key ' WS-LAST-KEY-PROCESSED
' (' WS-TOTAL-RECORDS-PROCESSED
' records)'.
RESTORE-FROM-CHECKPOINT.
READ CHECKPOINT-FILE
AT END
DISPLAY 'No checkpoint found - '
'starting fresh'
SET WS-IS-FRESH-START TO TRUE
NOT AT END
MOVE CK-LAST-KEY TO WS-RESTART-KEY
MOVE CK-RECORDS-PROCESSED
TO WS-TOTAL-RECORDS-PROCESSED
MOVE CK-TOTAL-INTEREST
TO WS-TOTAL-INTEREST
MOVE CK-CHECKPOINT-NUMBER
TO WS-CHECKPOINT-COUNT
DISPLAY 'RESTARTING from checkpoint '
WS-CHECKPOINT-COUNT
' at key ' WS-RESTART-KEY
END-READ.
POSITION-TO-RESTART-KEY.
* Skip records already processed
MOVE WS-RESTART-KEY TO AM-ACCT-NUMBER
START ACCT-MASTER-FILE
KEY IS GREATER THAN AM-ACCT-NUMBER
INVALID KEY
DISPLAY 'Cannot position to restart key'
STOP RUN
END-START
READ ACCT-MASTER-FILE NEXT
AT END
SET WS-ACCT-EOF TO TRUE
END-READ
DISPLAY 'Positioned to: ' AM-ACCT-NUMBER.
14.8.3 Production Checkpoint Considerations
In a real production environment, checkpoint/restart is more complex:
-
Commit scope: If the program writes to multiple output files, all outputs must be consistent at checkpoint time. You cannot checkpoint the master update without also checkpointing the report position and the exception file position.
-
JCL support: IBM z/OS provides system-level checkpoint/restart through the CHKPT macro and the RESTART parameter in JCL. This handles file repositioning automatically.
-
Database coordination: If the program updates DB2, checkpoints must coordinate with DB2 commit points. We cover this in Chapter 32 (Transaction Design Patterns).
-
Frequency trade-off: More frequent checkpoints mean less rework on restart but add overhead during normal processing. A common guideline: checkpoint every 5,000-50,000 records, or every 5-15 minutes of processing time.
💡 Defensive Programming Note: Even if you do not implement formal checkpoint/restart, always design your programs so they can be safely restarted. This means: write output files that can be recreated from scratch (rather than appending), use the old-master/new-master pattern instead of in-place updates, and log enough information to verify correct completion.
14.9 Handling Variable-Length Records in Multi-File Processing
So far, our examples have used fixed-length records. Production systems often work with variable-length records, which introduce additional considerations for multi-file processing.
14.9.1 Variable-Length Record Layouts
COBOL supports variable-length records through the RECORD CONTAINS clause:
FD CLAIM-FILE
RECORD CONTAINS 100 TO 500 CHARACTERS.
01 CLAIM-RECORD.
05 CLM-HEADER.
10 CLM-ID PIC X(10).
10 CLM-TYPE PIC X(02).
10 CLM-LINE-COUNT PIC 9(02).
05 CLM-LINE OCCURS 1 TO 10 TIMES
DEPENDING ON CLM-LINE-COUNT.
10 CLM-LINE-PROC-CODE PIC X(05).
10 CLM-LINE-AMOUNT PIC S9(07)V99 COMP-3.
10 CLM-LINE-DATE PIC 9(08).
10 CLM-LINE-DIAG PIC X(07).
10 CLM-LINE-MODIFIER PIC X(02).
When processing variable-length records in the balanced-line algorithm, the key comparison logic is the same, but you must be careful about:
-
Record length on WRITE: Ensure the RECORD CONTAINS clause and the OCCURS DEPENDING ON value are correct before each WRITE. A wrong length truncates data or writes garbage.
-
Work record sizing: The work record in WORKING-STORAGE must accommodate the maximum record size, not just the current record size.
-
Key position: The key must be at a fixed offset within the record, regardless of the variable portion. This is why keys are typically in the header area, before any OCCURS DEPENDING ON fields.
14.9.2 Handling Different Record Types in the Same File
Some files contain multiple record types — a common pattern in mainframe data processing. For example, a transaction file might contain header records, detail records, and trailer records:
01 TXN-RECORD.
05 TXN-RECORD-TYPE PIC X(01).
88 TXN-IS-HEADER VALUE 'H'.
88 TXN-IS-DETAIL VALUE 'D'.
88 TXN-IS-TRAILER VALUE 'T'.
05 TXN-RECORD-DATA PIC X(149).
01 TXN-HEADER REDEFINES TXN-RECORD.
05 FILLER PIC X(01).
05 TH-FILE-DATE PIC 9(08).
05 TH-SOURCE-SYSTEM PIC X(08).
05 TH-RECORD-COUNT PIC 9(09).
05 FILLER PIC X(124).
01 TXN-DETAIL REDEFINES TXN-RECORD.
05 FILLER PIC X(01).
05 TD-ACCT-NUMBER PIC X(10).
05 TD-TXN-TYPE PIC X(01).
05 TD-AMOUNT PIC S9(11)V99 COMP-3.
05 FILLER PIC X(131).
01 TXN-TRAILER REDEFINES TXN-RECORD.
05 FILLER PIC X(01).
05 TT-TOTAL-AMOUNT PIC S9(15)V99 COMP-3.
05 TT-RECORD-COUNT PIC 9(09).
05 TT-HASH-TOTAL PIC 9(18).
05 FILLER PIC X(113).
In the balanced-line algorithm, you must distinguish between record types during processing:
2000-PROCESS-FILES.
* Skip non-detail transaction records
IF TXN-IS-HEADER
PERFORM VALIDATE-HEADER
PERFORM 2200-READ-TXN
ELSE IF TXN-IS-TRAILER
PERFORM VALIDATE-TRAILER
PERFORM 2200-READ-TXN
ELSE
* Normal balanced-line comparison for detail records
EVALUATE TRUE
WHEN OM-ACCT-NUMBER < TD-ACCT-NUMBER
PERFORM 3000-MASTER-ONLY
WHEN OM-ACCT-NUMBER = TD-ACCT-NUMBER
PERFORM 4000-MATCHED
WHEN OM-ACCT-NUMBER > TD-ACCT-NUMBER
PERFORM 5000-TXN-ONLY
END-EVALUATE
END-IF.
The header record provides metadata (file date, source system, expected record count) that should be validated before processing begins. The trailer record provides control totals (total amount, record count, hash total) that should be verified after processing ends. This header-detail-trailer pattern is a standard control mechanism in production batch processing.
✅ Best Practice — Control Totals: Always validate the trailer record's control totals against your program's accumulated totals. If the expected record count is 150,000 but your program processed 149,997, three records were lost somewhere — possibly in the sort, possibly in the extract, possibly in the transfer. Catching this discrepancy prevents downstream errors.
14.10 The Human Factor: Maintaining File Processing Code
File processing programs are often the longest-lived code in an organization. GlobalBank's DAILY-UPDATE program (Section 14.3) was originally written in 1992. Over 30 years, it has been modified by dozens of developers, each adding new transaction types, new validation rules, and new exception conditions.
14.10.1 Maintainability Principles
Maria Chen's team follows these principles for file processing code:
-
Paragraph names describe the action:
3000-MASTER-ONLY,4000-MATCHED,5000-TXN-ONLY— anyone reading the code knows immediately what each section handles. -
Counters for everything: Every path through the program increments a counter. The summary report serves as both documentation and a diagnostic tool. If the numbers do not add up, something went wrong.
-
Exception file, not silent skipping: Never silently skip a record you cannot process. Write it to an exception file with a reason code. James Okafor's rule: "If a record enters the program, it must exit through one of the defined paths — updated master, exception file, or report. No record disappears."
-
Reconciliation arithmetic: The summary should include reconciliation checks. For the daily update:
New master count = Old master count + Adds - Physical deletes. If the numbers do not match, investigate. -
Comments at decision points: Every EVALUATE WHEN and every IF should have a comment explaining the business reason, not just the technical condition.
14.10.2 Common Maintenance Pitfalls
⚠️ Adding a new transaction type: When adding a new WHEN clause to the EVALUATE in 4100-APPLY-TRANSACTIONS, developers must also add a counter, add the new type to the summary report, and test the new type in both matched and unmatched scenarios.
⚠️ Changing the record layout: If the master file layout changes, every program that reads the master must be recompiled — and the COPYBOOK must be updated first. Use COPY statements and maintain a single authoritative layout.
⚠️ Sort order assumptions: The balanced-line algorithm assumes both files are sorted on the same key in the same order. If a new file source is added upstream without proper sorting, the algorithm produces incorrect results with no error message. Always verify sort order at the beginning of the program (check that each key is >= the previous key).
14.11 Error Recovery Patterns in Multi-File Processing
Production batch programs must handle errors gracefully. When processing millions of records across multiple files, a single bad record should not terminate the entire job. This section presents patterns that GlobalBank and MedClaim use to build resilience into their file processing programs.
14.11.1 The Exception File Pattern
Every multi-file processing program should include an exception file that captures records the program cannot process normally:
FD EXCEPTION-FILE.
01 EXCEPTION-RECORD.
05 EX-TIMESTAMP PIC X(26).
05 EX-SOURCE-FILE PIC X(08).
05 EX-RECORD-KEY PIC X(10).
05 EX-REASON-CODE PIC X(04).
05 EX-REASON-TEXT PIC X(50).
05 EX-ORIGINAL-DATA PIC X(150).
01 WS-EXCEPTION-COUNTS.
05 WS-EX-INVALID-KEY PIC 9(07) VALUE ZERO.
05 WS-EX-INVALID-AMT PIC 9(07) VALUE ZERO.
05 WS-EX-DUP-TXN PIC 9(07) VALUE ZERO.
05 WS-EX-BAD-TYPE PIC 9(07) VALUE ZERO.
05 WS-EX-TOTAL PIC 9(07) VALUE ZERO.
The exception file serves three purposes: it prevents job abends from individual bad records, it preserves the bad records for investigation, and it provides a clear count of how many records failed each processing rule.
14.11.2 Threshold-Based Abort
While exception files allow individual records to fail gracefully, there must be a limit. If too many records are failing, something is fundamentally wrong — the wrong file was presented, the sort was corrupted, or the data was not validated upstream. Derek Washington's rule at GlobalBank: "If more than 2% of input records go to exceptions, abort the job and call operations."
CHECK-EXCEPTION-THRESHOLD.
IF WS-RECORDS-READ > 1000
COMPUTE WS-EXCEPTION-RATE =
(WS-EX-TOTAL / WS-RECORDS-READ) * 100
IF WS-EXCEPTION-RATE > 2
DISPLAY '*** ABORT: Exception rate '
WS-EXCEPTION-RATE '% exceeds 2% ***'
DISPLAY 'Records read: ' WS-RECORDS-READ
DISPLAY 'Exceptions: ' WS-EX-TOTAL
PERFORM 9500-EMERGENCY-CLOSE
MOVE 16 TO RETURN-CODE
STOP RUN
END-IF
END-IF.
The threshold check waits until at least 1,000 records have been processed (to avoid false triggers on small initial batches where one or two exceptions would exceed the percentage). The RETURN-CODE of 16 tells the JCL scheduler to skip subsequent steps and alert operations.
14.11.3 Graceful Degradation with Reference File Failures
In multi-file processing, the main input file is typically mandatory, but reference files may tolerate partial failures. Consider MedClaim's claims adjudication: if the provider reference file cannot be opened, the program cannot adjudicate claims at all — that is a hard failure. But if the fee schedule reference file is temporarily unavailable, the program might still process claims using a default fee schedule and flag them for manual review:
OPEN-REFERENCE-FILES.
OPEN INPUT PROVIDER-FILE
IF WS-PF-STATUS NOT = '00'
DISPLAY '*** CRITICAL: Cannot open PROVIDER-FILE'
MOVE 16 TO RETURN-CODE
STOP RUN
END-IF
OPEN INPUT FEE-SCHEDULE-FILE
IF WS-FS-STATUS NOT = '00'
DISPLAY '*** WARNING: Cannot open FEE-SCHEDULE '
'- using default fee table'
SET WS-USE-DEFAULT-FEES TO TRUE
ELSE
SET WS-USE-FEE-FILE TO TRUE
END-IF.
This distinction between critical and non-critical reference files is a design decision that must involve the business stakeholders. At MedClaim, James Okafor maintains a matrix showing which file failures require an abort and which allow degraded processing, reviewed quarterly with the claims operations team.
14.11.4 Audit Trail and Reconciliation
Every production multi-file program must produce a summary that allows operations to verify correct processing. The summary should include:
- Input record counts (one per input file)
- Output record counts (one per output file)
- Exception counts by category
- Control totals (total dollar amounts, hash totals)
- Reconciliation arithmetic that proves completeness
9100-PRINT-RECONCILIATION.
DISPLAY '=== DAILY UPDATE RECONCILIATION ==='
DISPLAY 'OLD MASTER RECORDS: ' WS-OLD-MASTER-COUNT
DISPLAY ' + ADDS: ' WS-ADD-COUNT
DISPLAY ' - DELETES: ' WS-DELETE-COUNT
DISPLAY ' = EXPECTED NEW: ' WS-EXPECTED-NEW
DISPLAY 'ACTUAL NEW MASTER: ' WS-NEW-MASTER-COUNT
IF WS-EXPECTED-NEW NOT = WS-NEW-MASTER-COUNT
DISPLAY '*** RECONCILIATION FAILURE ***'
MOVE 8 TO RETURN-CODE
END-IF
DISPLAY 'EXCEPTIONS: ' WS-EX-TOTAL
DISPLAY ' Invalid key: ' WS-EX-INVALID-KEY
DISPLAY ' Invalid amount: ' WS-EX-INVALID-AMT
DISPLAY ' Duplicate txn: ' WS-EX-DUP-TXN
DISPLAY ' Bad type code: ' WS-EX-BAD-TYPE
COMPUTE WS-INPUT-TOTAL =
WS-OLD-MASTER-COUNT + WS-TXN-COUNT
COMPUTE WS-OUTPUT-TOTAL =
WS-NEW-MASTER-COUNT + WS-EX-TOTAL
+ WS-TXN-ONLY-COUNT
DISPLAY 'RECORDS IN: ' WS-INPUT-TOTAL
DISPLAY 'RECORDS OUT: ' WS-OUTPUT-TOTAL.
The reconciliation arithmetic is the last line of defense. If records entered the program but did not exit through any defined path, the numbers will not balance — and operations will know something went wrong before the bad data propagates to downstream systems.
🔴 Production War Story: In 2019, a GlobalBank developer modified the daily update to support a new "account freeze" transaction type. The new code path processed the freeze correctly but did not increment any counter. For two months, freeze transactions were invisible in the summary report. When an auditor noticed that the summary counts no longer balanced, the investigation revealed 847 freeze transactions that had been applied but never reported. Maria Chen added a post-processing check: the program now verifies that the sum of all transaction-type counters equals the total transaction count. "If they don't match, we missed a code path," she says.
14.12 Try It Yourself: Hands-On Exercises
Exercise 14.1 — Simple Balanced-Line
Create a simplified version of the DAILY-UPDATE program:
- Master file: Student ID, Name, GPA, Credits
- Transaction file: Student ID, Type ('U' update GPA, 'A' add, 'D' delete), Data
- Implement the balanced-line algorithm
- Write a new master file, exception file, and summary report
- Test with: 10 master records, 15 transactions (including adds, updates, deletes, and errors)
Exercise 14.2 — Control Break Report
Build a multi-level control break report:
- Input: Account file sorted by Region, Branch, Account Number
- Report levels: Detail line, Branch subtotal, Region subtotal, Grand total
- Include: account count and balance total at each level
Exercise 14.3 — File Comparison
Write a comparison program that:
- Reads two sorted files with the same layout
- Identifies records in File A only, File B only, and matches
- For matches, compares all data fields and flags differences
- Produces a reconciliation report
🧪 GnuCOBOL Compatibility: All programs in this chapter work with GnuCOBOL. The multi-file patterns (balanced-line, merge, control break) are pure COBOL logic with no VSAM-specific features. Use LINE SEQUENTIAL files for all input and output.
14.13 Common Mistakes in Multi-File Processing
Mistake 1: Forgetting HIGH-VALUES Sentinel
*--- WRONG: No HIGH-VALUES — loop terminates too early
PERFORM UNTIL WS-BOTH-EOF
IF MASTER-KEY < TXN-KEY
...
END-IF
END-PERFORM
*--- RIGHT: HIGH-VALUES ensures all records processed
PERFORM UNTIL MASTER-KEY = HIGH-VALUES
AND TXN-KEY = HIGH-VALUES
...
END-PERFORM
Mistake 2: Not Handling Multiple Transactions per Master
*--- WRONG: Reads only one transaction per match
WHEN MASTER-KEY = TXN-KEY
PERFORM APPLY-TRANSACTION
PERFORM READ-MASTER
PERFORM READ-TXN
*--- RIGHT: Loop to apply all transactions for same key
WHEN MASTER-KEY = TXN-KEY
PERFORM APPLY-ALL-TXNS-FOR-KEY
PERFORM READ-MASTER
* (transactions already advanced past this key)
Mistake 3: Control Break With Unsorted Data
The control break algorithm assumes data is sorted by the break field. If the input is not sorted, you get incorrect subtotals — and no error message. Always verify sort order:
VERIFY-SORT-ORDER.
IF AR-BRANCH-CODE < WS-PREV-BRANCH
DISPLAY '*** FATAL: Input not sorted! ***'
DISPLAY 'Previous branch: ' WS-PREV-BRANCH
DISPLAY 'Current branch: ' AR-BRANCH-CODE
STOP RUN
END-IF.
Mistake 4: Checkpoint Without All State
If your checkpoint saves the last key processed but not the running totals, the restarted program will produce incorrect summary numbers. Save ALL state that affects the final output.
14.14 Putting It All Together: The Daily Batch Cycle
Let us step back and see how all the patterns from this chapter fit together in a real production batch cycle. GlobalBank's nightly processing illustrates the flow:
6:00 PM — Online systems close
6:15 PM — STEP 1: SORT-TXN
Sort daily transactions by account number
Input: TXN-DAILY (unsorted, 2.3M records)
Output: TXN-SORTED (sorted by account number)
Pattern: External sort (DFSORT utility or COBOL SORT)
6:25 PM — STEP 2: DAILY-UPDATE (Balanced-Line Algorithm)
Apply sorted transactions to old master
Input: ACCT-MASTER-OLD + TXN-SORTED
Output: ACCT-MASTER-NEW + EXCEPTION-FILE + SUMMARY
Pattern: Balanced-line (Chapter 14.3)
7:10 PM — STEP 3: VERIFY-UPDATE (File Comparison)
Compare old master + txns against new master
Input: ACCT-MASTER-OLD + TXN-SORTED + ACCT-MASTER-NEW
Output: VERIFICATION-REPORT
Pattern: File comparison/reconciliation (Chapter 14.5)
7:45 PM — STEP 4: BRANCH-RPT (Control Break Report)
Generate account summary by branch
Input: ACCT-MASTER-NEW
Output: BRANCH-SUMMARY-REPORT
Pattern: Control break (Chapter 14.6)
8:30 PM — STEP 5: EXCEPTION-PROC
Process exception records from DAILY-UPDATE
Input: EXCEPTION-FILE
Output: EXCEPTION-REPORT + CORRECTION-FILE
Pattern: Sequential processing with classification
9:00 PM — STEP 6: INTEREST-CALC (with Checkpoint/Restart)
Calculate daily interest for all accounts
Input: ACCT-MASTER-NEW
Output: ACCT-MASTER-WITH-INTEREST + INTEREST-REPORT
Pattern: Sequential update with checkpoint (Chapter 14.8)
10:30 PM — STEP 7: RENAME
Rename NEW master to become tomorrow's OLD master
Pattern: JCL utility
10:35 PM — STEP 8: BACKUP
Backup all files to tape
Pattern: JCL utility
11:00 PM — Batch cycle complete (5 hours)
Each step depends on the successful completion of the previous step. If any step fails, the JCL conditional execution (COND parameter) stops subsequent steps, the operations team is paged, and the resolution process begins. The old master is always available for recovery.
This architecture has been running at GlobalBank — with modifications and enhancements — since 1992. The patterns are the same ones you have learned in this chapter. The scale is larger, the error handling more comprehensive, and the monitoring more sophisticated, but the core algorithms are identical.
14.15 Performance Considerations for Multi-File Processing
14.15.1 I/O Optimization
| Technique | Benefit | Trade-off |
|---|---|---|
| Larger block sizes | Fewer I/Os per file | More buffer memory |
| Multiple buffers (BUFNO) | Read-ahead for sequential | Memory consumption |
| Minimize file opens/closes | Reduced overhead | Code complexity |
| Sort before processing | Enables single-pass algorithms | Sort time and space |
14.15.2 Memory vs. I/O Trade-offs
For reference files accessed randomly (like the MedClaim provider lookup), loading the entire file into a COBOL table in WORKING-STORAGE eliminates I/O but consumes memory. The break-even point depends on file size and access frequency:
- File < 10,000 records with high access rate → load into memory
- File > 100,000 records → use VSAM with good buffering
- File 10,000-100,000 → analyze access pattern and test both approaches
14.15.3 Parallel Processing Considerations
Modern mainframe systems can process multiple files simultaneously using:
- Multi-threaded batch: z/OS Language Environment supports multi-threading, though COBOL programs typically use single-threaded designs
- JCL step parallelism: Independent processing steps can run concurrently
- Sort optimization: IBM DFSORT can merge while sorting, combining steps
These topics are covered in detail in Chapter 38 (Batch Processing Patterns).
14.15.4 Buffer Management
When processing multiple files simultaneously, buffer allocation becomes critical. Each OPEN file consumes buffer space, and the operating system allocates buffers based on the BUFFERSPACE clause or system defaults. For a balanced-line program with old master, transaction, and new master files, plus an exception file and a report file, that is five open files competing for buffer resources.
On z/OS, the JCL BUFNO parameter on the DD statement controls the number of I/O buffers per file. The default is typically 2, but increasing to 5-10 for sequentially processed files can significantly improve throughput by allowing the system to read ahead while the program processes the current record:
//OLDMAST DD DSN=PROD.ACCT.MASTER,DISP=SHR,BUFNO=8
//TXNFILE DD DSN=PROD.TXN.DAILY,DISP=SHR,BUFNO=8
//NEWMAST DD DSN=PROD.ACCT.NEWMSTR,DISP=(NEW,CATLG),BUFNO=5
//EXCEPTF DD DSN=PROD.EXCEPT.DAILY,DISP=(NEW,CATLG),BUFNO=2
//RPTFILE DD SYSOUT=*,BUFNO=2
Notice the asymmetry: high-volume sequential input files get more buffers than output files or low-volume exception files. This is intentional — read-ahead benefits are greatest for sequentially processed input, where the system can prefetch the next block while the program processes the current one.
In GnuCOBOL, buffer management is handled by the operating system's file cache. While you cannot control it directly, processing files sequentially (as all patterns in this chapter do) naturally benefits from the OS page cache's read-ahead behavior.
💡 Rule of Thumb for Multi-File Programs: Total buffer allocation across all open files should not exceed available memory. On modern mainframes this is rarely a constraint, but on older systems or in regions with tight memory limits, over-allocating buffers can cause paging — which is far worse than having too few buffers. When in doubt, consult your systems programmer.
14.16 Chapter Summary
This chapter has covered the essential multi-file processing patterns that form the backbone of enterprise COBOL batch systems:
-
The balanced-line algorithm: The gold standard for master-transaction updates. Two sorted files processed in a single pass, with three-way key comparison driving all logic. The HIGH-VALUES sentinel technique ensures all records from both files are processed.
-
Multi-file merge: Combining records from multiple sorted sources into a single sorted output. Scales from two files to N-way merge using a merge key table.
-
File comparison and reconciliation: Identifying discrepancies between two files that should agree. Essential for auditing and data integrity verification.
-
Control break processing: Producing reports with subtotals at each change in grouping fields. Multi-level breaks must be properly nested — a higher-level break triggers all lower-level breaks.
-
Checkpoint/restart: Saving program state periodically so long-running jobs can resume from the last checkpoint instead of restarting from the beginning. Critical for jobs that process millions of records.
-
The human factor: File processing programs are long-lived and frequently modified. Maintainability requires clear paragraph naming, comprehensive counters, exception files (never silent skipping), reconciliation arithmetic, and careful handling of record layout changes.
These patterns are defensive programming in action. Every record is accounted for. Every error is captured. Every run produces reconciliation numbers that can be verified. This discipline is what makes COBOL systems reliable enough to process trillions of dollars daily — and it is the standard you should aspire to in your own programs.
🔗 Connections: This chapter connects to Chapter 11 (Sequential files — the basis for most multi-file patterns), Chapter 12 (Indexed files — random lookup within sequential processing), Chapter 15 (Sort and Merge — COBOL SORT verb as an alternative to manual merge), Chapter 38 (Batch Processing Patterns — production-scale implementations), and Chapter 43 (Capstone: Complete Banking Transaction System).