Case Study 01: First National Bank's COBOL Legacy

Background

First National Bank (FNB) is a mid-sized regional bank headquartered in the American Midwest. Founded in 1952, FNB serves 2.3 million customers across 180 branch locations in six states. The bank offers a full range of consumer and commercial banking products: checking and savings accounts, mortgages, auto loans, credit cards, and small business lending.

FNB's core banking system -- the software that manages customer accounts, processes transactions, calculates interest, generates statements, and handles regulatory reporting -- runs on an IBM z15 mainframe at the bank's primary data center. The system consists of approximately 14 million lines of COBOL code, organized into roughly 4,200 individual programs. These programs interact with a DB2 relational database containing 3.8 billion records and approximately 600 VSAM datasets that store historical transaction data and configuration files.

The oldest COBOL programs in the system date back to 1983, when FNB first migrated from a Burroughs mainframe to IBM. Many of the core transaction processing programs have been in continuous production use for over 30 years, receiving incremental modifications to accommodate new products, regulatory changes, and business rule updates, but never being fundamentally rewritten.

The system processes an average of 8.2 million transactions per day, with peaks reaching 15 million during month-end and tax season. It operates with 99.998% uptime -- the bank has experienced fewer than four hours of unplanned downtime in the past five years.

The Current Technology Stack

FNB's mainframe environment includes:

  • Hardware: IBM z15 with 190 configurable processors and 40 TB of memory
  • Operating System: z/OS 2.5
  • Transaction Processing: CICS Transaction Server 5.6
  • Database: DB2 for z/OS version 13
  • Batch Scheduling: CA-7 Workload Automation
  • Programming Languages: COBOL (85% of code), PL/I (10%), Assembler (5%)
  • File Storage: VSAM, sequential datasets, and GDGs (Generation Data Groups)
  • Security: RACF (Resource Access Control Facility)

The nightly batch processing window runs from 11:00 PM to 5:30 AM and executes approximately 2,800 JCL job steps in a carefully orchestrated sequence. The batch window processes end-of-day postings, interest calculations, fee assessments, regulatory data extracts, and statement generation.

During business hours, CICS handles online transactions from 180 branch terminals, 412 ATMs, the online banking portal, and the mobile banking application. The online and mobile channels communicate with the mainframe through a middleware layer that translates web requests into CICS transactions.

The People

FNB's mainframe development team consists of 28 developers:

  • Senior developers (20+ years experience): 8 team members, ages 58-67
  • Mid-level developers (10-20 years experience): 11 team members, ages 42-57
  • Junior developers (less than 10 years experience): 9 team members, ages 28-41

The team is led by Margaret Chen, a 62-year-old Senior Vice President of Core Systems Engineering who has been with FNB for 34 years. Margaret personally wrote or supervised the writing of many of the bank's most critical COBOL programs and carries an enormous amount of institutional knowledge about the system's architecture, business rules, and historical decisions.

Three of the eight senior developers have announced their intention to retire within the next 18 months. Two more are expected to retire within three years. Margaret herself plans to retire in two years.

The bank's human resources department reports that their last three job postings for COBOL developers received a total of 14 applications. Of those, only four candidates met the minimum qualifications. The bank hired two. For comparison, a posting for a Java developer received 127 applications.

The Challenges

Challenge 1: Knowledge Loss

When senior developer Robert Huang retired last year, the team discovered that several critical batch programs he maintained had minimal documentation. Robert had carried detailed knowledge of the programs' business logic, edge cases, and interdependencies in his head. It took two mid-level developers six months of code analysis to reconstruct enough understanding to safely maintain these programs.

The bank estimates that approximately 40% of its COBOL codebase has inadequate documentation. Business rules are embedded directly in the code, often in complex conditional structures that reflect decades of regulatory changes and product modifications.

Challenge 2: Modernization Pressure

FNB's Chief Technology Officer, David Park, has been under pressure from the board of directors to "modernize" the bank's technology. A consulting firm engaged by the board recommended replacing the mainframe COBOL system with a cloud-based platform within five years. The consultants estimated the cost at $180 million.

David is skeptical. He has seen industry reports documenting failed core banking replacements at other institutions -- projects that went years over schedule and billions over budget. Commonwealth Bank of Australia's core system replacement took five years. TSB's migration from Lloyds' systems in 2018 caused a catastrophic failure that locked 1.9 million customers out of their accounts. Closer to home, a competing regional bank spent $240 million on a core system replacement that was ultimately abandoned after three years.

Challenge 3: Integration with Modern Channels

FNB's customers increasingly expect real-time digital experiences -- instant balance updates, immediate fund transfers, and personalized financial insights delivered through mobile apps. The bank's mobile banking application, built three years ago using React Native, communicates with the mainframe through a middleware layer that sometimes introduces latency.

The digital banking team, composed of developers who work in JavaScript, Python, and Java, finds it difficult to collaborate with the mainframe team. The two groups use different development tools, different version control systems (the mainframe team recently migrated from ENDEVOR to Git), different deployment processes, and different terminology. Cultural friction between the teams is a growing concern.

Challenge 4: Regulatory Compliance

Banking regulators conduct periodic technology risk assessments. In FNB's most recent examination, regulators flagged the concentration of institutional knowledge in a small number of senior developers as a "key person risk." They recommended that the bank develop a formal knowledge transfer and succession plan.

Additionally, new regulatory requirements around real-time fraud detection and faster payment processing are straining the existing system's batch-oriented architecture. While the core system is reliable, its design reflects an era when end-of-day batch processing was the norm. The shift toward real-time processing requires architectural changes.

Challenge 5: Cost Perceptions

The bank's CFO has noted that mainframe licensing costs represent a significant line item in the technology budget. IBM's pricing model, which is based on MIPS (Millions of Instructions Per Second) consumed, results in monthly bills that can be difficult to predict. Some board members question whether moving to the cloud would reduce costs.

However, detailed analysis has shown that the mainframe's per-transaction cost is actually quite low when measured against the volume of transactions processed. The challenge is communicating this effectively to stakeholders who are more familiar with cloud pricing models.

The Decision

FNB's leadership team is debating three strategic options:

Option A: Full Replacement

Replace the entire mainframe COBOL system with a modern cloud-based core banking platform (such as Temenos, Finacle, or Thought Machine). Estimated cost: $180-250 million. Estimated timeline: 4-7 years. Risk: very high.

Option B: Strategic Modernization

Keep the mainframe and COBOL systems but invest in modernization: API enablement, DevOps practices, improved documentation, and selective rewriting of specific components. Estimated cost: $25-40 million over 3 years. Risk: moderate.

Option C: Hybrid Approach

Migrate selected non-critical workloads to the cloud while keeping core transaction processing on the mainframe. Invest in API layers to enable seamless integration. Build new capabilities in modern languages that communicate with COBOL systems through well-defined interfaces. Estimated cost: $50-80 million over 4 years. Risk: moderate.

What They Decided

After extensive deliberation, FNB chose Option C: the Hybrid Approach, with elements of Option B. The key components of their strategy include:

  1. API Enablement: Implementing IBM z/OS Connect to expose critical COBOL programs as RESTful APIs, allowing the mobile app and web portal to communicate directly with mainframe services without custom middleware.

  2. Knowledge Capture Program: Launching a formal initiative to document business rules, system architecture, and operational procedures. Senior developers are paired with junior team members in a structured mentoring program. Each critical COBOL program is being documented using a standardized template.

  3. Recruitment and Training: Partnering with two local universities to create a COBOL/mainframe training program. FNB offers paid internships and tuition assistance for students who complete the program. The bank also raised COBOL developer salaries by 15% to be more competitive in the market.

  4. Selective Cloud Migration: Moving non-critical workloads -- marketing analytics, customer survey processing, and development/test environments -- to AWS. Core transaction processing remains on the mainframe.

  5. DevOps Adoption: Implementing Git for source control, Jenkins for automated builds, and Zowe for mainframe DevOps. The goal is to bring the mainframe development workflow closer to the practices used by the digital banking team, reducing cultural friction.

  6. Real-Time Capabilities: Implementing IBM Event Streaming (Kafka on z/OS) to enable near-real-time event processing alongside the existing batch architecture, addressing regulatory requirements for faster fraud detection.

Outcomes (18 Months Later)

Eighteen months into the hybrid strategy:

  • The API layer is operational, reducing mobile app response times by 40%
  • Eight university interns have completed the COBOL training program; four have accepted full-time positions
  • Documentation coverage has improved from 60% to 82% of critical programs
  • The DevOps pipeline has reduced deployment cycles from two weeks to three days
  • Margaret Chen has postponed her retirement by one year to oversee the knowledge transfer program
  • Mainframe operating costs have decreased by 8% due to workload optimization
  • The digital banking team reports significantly improved collaboration with the mainframe team

The bank estimates that the full hybrid transformation will take another two to three years to complete, but early results have validated the approach.

Discussion Questions

  1. Risk Assessment: Why is a full system replacement (Option A) considered the highest-risk option, despite being the most "modern" approach? What specific risks could cause such a project to fail at a bank like FNB?

  2. Knowledge Management: FNB discovered that 40% of its codebase had inadequate documentation. What practices could have prevented this situation? How should the bank prioritize which programs to document first?

  3. The Human Factor: Margaret Chen carries decades of institutional knowledge. What specific steps should FNB take to capture and transfer her knowledge before she retires? How can the bank avoid creating similar "key person" dependencies in the future?

  4. Cost Analysis: The board initially focused on mainframe licensing costs as a reason to migrate to the cloud. How would you present a more complete cost comparison that accounts for per-transaction costs, migration costs, risk costs, and the total cost of ownership?

  5. Cultural Bridge: The chapter describes friction between the mainframe team and the digital banking team. What organizational and technical strategies can reduce this friction? How important is a shared toolchain (e.g., both teams using Git) in bridging the cultural divide?

  6. Regulatory Perspective: Banking regulators flagged "key person risk." If you were advising FNB on their response to this regulatory finding, what specific, measurable actions would you recommend?

  7. Recruitment Strategy: FNB raised salaries and partnered with universities. What other strategies could a bank use to attract new COBOL developers? Consider both traditional and creative approaches.

  8. COBOL Program Exploration: The accompanying code file (code/case-study-code.cob) implements a simplified version of FNB's bank welcome screen. Review the code and answer: - How does the program validate an account number? - What COBOL features are demonstrated in the program? - How would you modify the program to add a "View Loan Payment Schedule" menu option?

Key Lessons

  • COBOL systems in banking are not "legacy" in the pejorative sense -- they are mission-critical infrastructure that processes billions of dollars daily with exceptional reliability.
  • The greatest risk to COBOL systems is not the technology itself but the loss of human expertise.
  • Successful modernization is incremental, not revolutionary. Organizations that try to replace everything at once frequently fail.
  • Hybrid architectures that combine mainframe COBOL systems with modern cloud technologies offer the best balance of stability and innovation.
  • Investing in people -- through training, mentoring, competitive compensation, and knowledge documentation -- is at least as important as investing in technology.