Case Study 2: Federal Benefits Administration's DFSORT Revolution
"Thirty-one thousand lines of COBOL, maintained by three different people over twenty years, doing what DFSORT can do in twelve hundred lines of JCL. And doing it four times slower."
— Sandra Chen, Modernization Lead, Federal Benefits Administration
The Setup
Federal Benefits Administration runs one of the largest civilian COBOL systems in the federal government — 15 million lines of code, some dating to the early 1980s, processing benefit calculations, eligibility determinations, and payment processing for 65 million beneficiaries. The batch window runs 12 hours (6 PM to 6 AM), and on month-end runs, it uses nearly every minute.
Sandra Chen arrived at FBA three years ago with a mandate to modernize. Her PhD in computer science and her experience at two Silicon Valley firms gave her a sharp eye for inefficiency. But she also had the pragmatism to recognize that modernization doesn't always mean replacing old code with new code. Sometimes it means replacing bad old code with better old tools.
Marcus Whitfield, the senior SME with two years until retirement, was her guide to the codebase. Marcus had written or maintained most of the programs Sandra would eventually target. He knew their histories, their quirks, and their undocumented dependencies.
The Discovery
Sandra's modernization assessment started with a code inventory. She categorized every batch program by function:
Category Programs Lines of Code % of Batch
──────────────────────────────────────────────────────────────────────────
Business logic (calculations) 87 410,000 58%
Data access (DB2/IMS CRUD) 34 185,000 23%
Sort/merge/reformat 23 31,400 4%
Report generation 18 62,000 8%
Utility/housekeeping 12 18,000 3%
Interface (file transfer prep) 9 28,000 4%
The sort/merge/reformat category caught her attention. Twenty-three programs. 31,400 lines of COBOL. Every single one of them did something that DFSORT or ICETOOL could do natively — sorting records, merging sorted files, reformatting field layouts, deduplicating, or producing summary counts.
She pulled the SMF data for these 23 programs over the previous month:
Total elapsed time (per night): 147 minutes average
Total CPU time (per night): 89 minutes
Total EXCP (per night): 14.2 million
Maintenance incidents (past year): 12
- 4 abends due to record layout changes (copybook not updated)
- 3 abends due to SORT work dataset space (B37)
- 2 abends due to REGION too small after volume growth
- 2 incorrect output (sort key position errors after field additions)
- 1 production outage (MERGE program deadlocked with concurrent job)
Twelve maintenance incidents in one year for programs that did nothing a utility could do better. The production outage — a MERGE program that held DISP=OLD on a dataset another job needed — had caused a 45-minute delay in the batch window during a month-end run.
Sandra showed the data to Marcus. "These 23 programs are 4% of the code but they generate 15% of the maintenance incidents. Every one of them could be replaced by DFSORT control statements."
Marcus's first reaction was defensive. "I wrote seven of those programs. They work." Sandra didn't push. She asked instead: "Can you show me why each one needs to be a COBOL program rather than a DFSORT job?"
The Analysis
Marcus and Sandra reviewed all 23 programs together. For each one, Marcus explained what it did and Sandra determined whether DFSORT/ICETOOL could replace it.
The analysis took three days. The results were unambiguous:
Program Function Lines DFSORT Feasible?
────────────────────────────────────────────────────────────────────────
FBASRT01 Sort eligibility by SSN 180 Yes - simple SORT
FBASRT02 Sort payments by date 210 Yes - simple SORT
FBASRT03 Sort claims by provider 190 Yes - simple SORT
FBASRT04 Sort adjustments by type 165 Yes - SORT + INCLUDE
FBASRT05 Sort exceptions by priority 240 Yes - SORT + OUTREC
FBAMRG01 Merge 3 payment files 520 Yes - MERGE
FBAMRG02 Merge eligibility updates 380 Yes - MERGE
FBAMRG03 Merge exception reports 290 Yes - MERGE
FBARFM01 Reformat for Treasury 890 Yes - OUTREC
FBARFM02 Reformat for audit extract 720 Yes - OUTREC
FBARFM03 Reformat provider records 540 Yes - OUTREC
FBARFM04 Reformat for EDI 834 1,100 Mostly - complex padding
FBADUP01 Dedup SSN extract 280 Yes - SUM FIELDS=NONE
FBADUP02 Dedup provider list 250 Yes - SUM FIELDS=NONE
FBASUM01 Summary by state 680 Yes - SUM
FBASUM02 Summary by program code 490 Yes - SUM
FBASUM03 Summary by payment type 520 Yes - SUM + OUTREC
FBASPL01 Split by region code 840 Yes - OUTFIL INCLUDE
FBASPL02 Split debits/credits 320 Yes - OUTFIL INCLUDE
FBASEL01 Select active records 290 Yes - INCLUDE/OMIT
FBASEL02 Select high-value claims 310 Yes - INCLUDE
FBASEL03 Select flagged exceptions 220 Yes - INCLUDE
FBACNV01 Convert packed to display 780 Yes - OUTREC + EDIT
Twenty-three out of twenty-three. Every program was fully replaceable by DFSORT/ICETOOL.
Marcus stared at the spreadsheet. "I spent two weeks writing FBARFM01," he said. "The Treasury format has 47 fields with specific padding and justification rules. You're telling me DFSORT can do that?"
Sandra showed him the OUTREC statement:
SORT FIELDS=COPY
OUTREC FIELDS=(1,9, SSN
C' ', Filler
15,30, Name
C' ', Filler
50,2, State
60,8,PD,EDIT=(STTTTTTTTTT.TT), Amount
C' ', Filler
75,8, Date
... continues for 47 fields ...)
"OUTREC handles field extraction, padding, packed-to-display conversion, date reformatting, and literal insertion," Sandra explained. "Everything FBARFM01 does in 890 lines of COBOL, OUTREC does in 35 lines of control statements."
Marcus spent the afternoon verifying. He ran FBARFM01 against its test dataset and then ran the DFSORT equivalent. He compared the output files byte-by-byte.
Identical.
He did the same for FBADUP01, FBASUM01, and FBASPL01. All identical.
"Should have done this fifteen years ago," Marcus said.
The Implementation
Sandra designed a phased rollout over six weeks:
Phase 1 (Weeks 1-2): Simple Sorts and Selects
Eight programs replaced: FBASRT01-04, FBASEL01-03, FBADUP01.
These were the simplest conversions — direct SORT, INCLUDE/OMIT, and SUM FIELDS=NONE operations. Each DFSORT replacement was validated by running both the COBOL program and the DFSORT job against the same input and comparing output byte-by-byte.
Sample replacement — FBASRT04 (Sort adjustments, filter by type):
Before (COBOL - 165 lines):
SORT SORT-WORK ON ASCENDING KEY SW-ADJ-TYPE
ASCENDING KEY SW-ADJ-DATE
INPUT PROCEDURE IS 1000-FILTER
GIVING SORTED-ADJ-FILE.
1000-FILTER SECTION.
OPEN INPUT ADJUSTMENT-FILE.
PERFORM UNTIL EOF
READ ADJUSTMENT-FILE
AT END SET EOF-FLAG TO TRUE
END-READ
IF NOT EOF-FLAG
IF ADJ-TYPE = 'A' OR 'B' OR 'C'
IF ADJ-AMOUNT > ZERO
RELEASE SW-RECORD FROM ADJ-RECORD
END-IF
END-IF
END-IF
END-PERFORM.
CLOSE ADJUSTMENT-FILE.
After (DFSORT - 5 lines):
SORT FIELDS=(85,1,CH,A,90,8,CH,A)
INCLUDE COND=(85,1,CH,EQ,C'A',OR,
85,1,CH,EQ,C'B',OR,
85,1,CH,EQ,C'C')
INCLUDE COND=(100,8,PD,GT,+0)
OPTION MAINSIZE=MAX
Results after Phase 1:
Programs replaced: 8
Lines eliminated: 2,180
Elapsed saved: 28 min/night
CPU saved: 19 min/night
EXCP saved: 3.4M/night
Phase 2 (Weeks 3-4): Merges, Summaries, and Splits
Nine programs replaced: FBAMRG01-03, FBASUM01-03, FBASPL01-02, FBADUP02.
The merge and split programs were more complex — ICETOOL was required for multi-operation jobs:
Sample replacement — FBASPL01 (Split by region code):
Before (COBOL - 840 lines):
Four OUTPUT files opened, records read sequentially,
EVALUATE on region code to route to appropriate file,
trailer records with counts written to each output.
After (DFSORT OUTFIL - 22 lines):
SORT FIELDS=COPY
OUTFIL FNAMES=REG1,INCLUDE=(45,2,CH,EQ,C'NE'),
TRAILER1=(COUNT=(M10,LENGTH=8),
C' NE REGION COUNT: ',COUNT=(M10,LENGTH=10))
OUTFIL FNAMES=REG2,INCLUDE=(45,2,CH,EQ,C'SE'),
TRAILER1=(COUNT=(M10,LENGTH=8),
C' SE REGION COUNT: ',COUNT=(M10,LENGTH=10))
OUTFIL FNAMES=REG3,INCLUDE=(45,2,CH,EQ,C'MW'),
TRAILER1=(COUNT=(M10,LENGTH=8),
C' MW REGION COUNT: ',COUNT=(M10,LENGTH=10))
OUTFIL FNAMES=REG4,INCLUDE=(45,2,CH,EQ,C'WE'),
TRAILER1=(COUNT=(M10,LENGTH=8),
C' WE REGION COUNT: ',COUNT=(M10,LENGTH=10))
One SORT pass. Four simultaneous outputs. Automatic record counts. The COBOL program read the file four times (once per region) because the original programmer didn't know how to open four output files simultaneously in a single pass. DFSORT reads the file once.
Results after Phase 2:
Programs replaced: 9 (17 cumulative)
Lines eliminated: 14,820 (17,000 cumulative)
Elapsed saved: 72 min/night (100 cumulative)
CPU saved: 48 min/night (67 cumulative)
EXCP saved: 6.8M/night (10.2M cumulative)
Phase 3 (Weeks 5-6): Reformats and the EDI Challenge
Six programs replaced: FBARFM01-04, FBACNV01, FBASEL02 (moved from Phase 1 due to complexity).
FBARFM04 — the EDI 834 reformatter — was the most complex conversion. The EDI 834 format requires variable-length segments with specific delimiters, qualifier codes, and padding rules. Sandra spent two days writing the DFSORT control statements and another day validating output against the EDI specification.
FBARFM04 DFSORT replacement: 68 lines of control statements
(The original COBOL program: 1,100 lines)
Key DFSORT features used:
OUTREC FIELDS — Field extraction and rearrangement
OUTREC EDIT — Packed decimal to display conversion
OUTREC TRAN=LTOU — Lowercase to uppercase translation
IFTHEN — Conditional field processing
BUILD — Complex field construction
SEQNUM — Automatic sequence numbering
The IFTHEN capability was critical for the EDI conversion — different record types required different field layouts:
OUTREC IFTHEN=(WHEN=(1,3,CH,EQ,C'INS'),
BUILD=(1,3,C'*',10,9,C'*',25,30,C'~'))
OUTREC IFTHEN=(WHEN=(1,3,CH,EQ,C'REF'),
BUILD=(1,3,C'*',10,2,C'*',15,20,C'~'))
OUTREC IFTHEN=(WHEN=(1,3,CH,EQ,C'DTP'),
BUILD=(1,3,C'*',10,3,C'*',15,8,C'~'))
Marcus reviewed the EDI output carefully. "If this is wrong, CMS rejects the entire file and we miss the submission deadline," he warned. Sandra ran the DFSORT version in parallel with the COBOL version for three consecutive month-end runs. All three produced identical output.
Final results after Phase 3:
Programs replaced: 6 (23 cumulative — ALL)
Lines eliminated: 14,400 (31,400 cumulative — ALL)
Elapsed saved: 47 min/night (147 cumulative)
CPU saved: 22 min/night (89 cumulative)
EXCP saved: 4.0M/night (14.2M cumulative)
The Numbers
FEDERAL BENEFITS ADMINISTRATION — DFSORT MIGRATION RESULTS
═══════════════════════════════════════════════════════════
Before (COBOL) After (DFSORT) Change
────────────────────────────────────────────────────────────────────
Programs maintained: 23 0 -100%
Lines of code: 31,400 1,180 (JCL/ctrl) -96.2%
Compile/link jobs: 23 0 -100%
Copybook dependencies: 47 0 -100%
Total elapsed/night: 147 min 38 min -74.1%
Total CPU/night: 89 min 14 min -84.3%
Total EXCP/night: 14.2M 3.8M -73.2%
Maintenance incidents: 12/year 1/year (est.) -91.7%
The maintenance reduction was the metric that mattered most to Sandra's modernization charter. Twelve incidents per year times an average of 4 hours per incident (diagnosis, fix, test, deploy) equals 48 person-hours of reactive maintenance — for code that was doing nothing sophisticated.
The single remaining estimated incident per year accounts for the possibility of a record layout change requiring DFSORT field position updates. Even that is simpler than a COBOL recompile: change the positions in the control statement, no compilation required.
The Controversy
Not everyone celebrated.
The COBOL purists argued that replacing COBOL with JCL-based utilities was "moving backward." One senior programmer wrote in the change review: "We are a COBOL shop. Our developers maintain COBOL. If we replace COBOL with DFSORT, who maintains the DFSORT?"
Sandra's response: "Everyone. DFSORT control statements are simpler than COBOL. Any developer who can read a sort key specification can maintain these jobs. That's the point — we're reducing the skill barrier, not raising it."
The risk-averse contingent worried about byte-for-byte equivalence. "How do we know the DFSORT output is exactly the same?" they asked.
Sandra's team ran three months of parallel execution — COBOL and DFSORT processing the same input, output compared programmatically. Zero discrepancies across 90 production runs.
Marcus Whitfield had the most nuanced perspective. "The programs I wrote worked. They weren't optimal, but they worked. Sandra made them faster and easier to maintain. That's the right kind of modernization — you don't throw away the system, you replace the parts that have better alternatives."
He paused. "But I'll tell you what bothers me. We should have known about DFSORT's capabilities twenty years ago. The tool was there. We just never learned it. How many millions of CPU minutes did we waste because nobody took a DFSORT class?"
Ahmad's Compliance Perspective
Ahmad Rashidi, FBA's compliance officer turned technical lead, had a different concern: auditability.
"Every one of those COBOL programs has been through compliance review," Ahmad said. "They have test evidence, change history, and audit documentation. If we replace them with DFSORT JCL, we need equivalent documentation."
Sandra agreed. Each DFSORT replacement included: - A mapping document showing the COBOL logic and the equivalent DFSORT control statements - Test evidence from the parallel runs (three months of byte-for-byte output comparison) - A change history noting the replacement date, the original program name, and the rationale - Updated disaster recovery procedures (DFSORT failures recover differently than COBOL abends)
Ahmad reviewed the documentation and approved. "Actually, this is easier to audit," he admitted. "I can read a DFSORT SORT FIELDS statement and understand what it does. Some of those COBOL programs — the ones with PERFORM THRU and GO TO DEPENDING ON — took me days to trace through."
Long-Term Impact
Six months after the migration completed, Sandra compiled the long-term metrics:
Metric 6-Month Average Prior Year Average
───────────────────────────────────────────────────────────────────────
Batch window margin: +109 min/night +38 min/night
Maintenance incidents (sort): 0 12
Developer hours (sort maint): 0 48
MSU consumption (sort jobs): 42 MSU/month 280 MSU/month
The MSU reduction of 238 MSU/month at FBA's government pricing translated to measurable cost avoidance. But the real value was developer time. The three people who had been maintaining those 23 COBOL sort programs were reassigned to Sandra's modernization project — working on business logic migration that actually required COBOL expertise.
Marcus Whitfield, approaching his final year, was philosophical. "I spent twenty years maintaining sort programs that a utility could have done all along. The young people coming in should learn DFSORT before they learn COBOL SORT. The utility is always going to be faster than the program, because the utility team has fifty years of optimization behind them."
Sandra added that quote to the FBA modernization case study she presented at SHARE. It was the most-downloaded session paper that year.
Lessons Learned
-
Not all modernization requires new technology. DFSORT has been available since 1965. The "modern" approach was using an existing tool to its full capability, not introducing a new one.
-
Measure the maintenance cost, not just the runtime cost. Twelve incidents per year at 4 hours each was the real pain point. The 74% elapsed time reduction was a bonus.
-
Parallel execution proves equivalence. Three months of side-by-side runs eliminated all doubt about output correctness. The investment in parallel validation was worth every minute.
-
Complexity is the enemy of reliability. 31,400 lines of COBOL are 31,400 opportunities for bugs. 1,180 lines of DFSORT control statements are 1,180 opportunities. Simpler is more reliable.
-
Tool knowledge is architecture knowledge. The batch window was 109 minutes shorter not because of better hardware or cleverer algorithms, but because the team learned what their tools could already do. The capability was always there.
-
The skeptics become advocates when they see the data. Marcus went from "they work" to "should have done this fifteen years ago" after examining the evidence. Don't argue philosophy; show byte-for-byte output comparison and elapsed time metrics.