Case Study 36.1: SecureFirst's DevOps Transformation — Yuki Leads the Pipeline Revolution

Background

When Yuki Tanaka was promoted to Director of Mainframe Engineering at SecureFirst Insurance in early 2025, she inherited a development process that hadn't fundamentally changed since 1998. The company's 380 COBOL programs, processing 2.3 million insurance claims annually, were managed through Endevor, compiled via manually submitted JCL, tested by hand in a shared CICS region, and deployed quarterly through a two-week promotion cycle that involved six people, forty-three manual steps, and a 300-page change control document.

The quarterly release cycle was the bottleneck that made everything else harder. Regulatory changes from the state insurance commissions needed to be implemented within 90 days — and with a quarterly cycle, that left only one release window to get it right. Bug fixes that could have been deployed in a day waited weeks for the next deployment window. And every quarterly release was an all-hands, all-weekend affair that left the team exhausted and anxious.

Yuki's CTO gave her a clear mandate: "Get us to monthly releases within twelve months. If you can get to bi-weekly, even better."

The Assessment

Yuki spent her first month conducting a thorough assessment. She interviewed every developer, every tester, every operations staff member. She mapped every process, measured every lead time, and documented every handoff.

The findings were stark:

The bottleneck analysis revealed that 60% of the lead time was waiting — waiting for a test region, waiting for a reviewer, waiting for the deployment window. Only 15% was actual development. The remaining 25% was manual testing and manual deployment steps.

The Strategy

Yuki presented a phased strategy to her CTO and the development team:

Phase 1: Foundation (Months 1-3)

• Migrate source from Endevor to Git
• Set up Jenkins on a Linux server connected to z/OS
• Install and configure Zowe CLI
• Install and configure IBM DBB
• Train the team on Git and Jenkins basics

Phase 2: Build Automation (Months 3-5)

• Create the Jenkinsfile for automated builds
• Configure DBB for dependency-based compilation
• Automate the compile-link-bind cycle
• Establish the first automated pipeline (commit -> build -> report)

Phase 3: Test Automation (Months 5-8)

• Implement unit testing framework (IBM zUnit)
• Write unit tests for the 50 most critical paragraphs
• Implement integration testing harness
• Write integration tests for the 10 most critical transaction flows
• Integrate tests into the Jenkins pipeline

Phase 4: Deployment Automation (Months 8-10)

• Automate library promotion (dev -> QA -> staging -> production)
• Implement CICS NEWCOPY automation
• Implement DB2 bind automation
• Add deployment verification (smoke tests)
• Add automated rollback capability

Phase 5: Optimization (Months 10-12)

• Performance testing in the pipeline
• Security scanning in the pipeline
• Monitoring integration (post-deployment metrics)
• Process refinement based on metrics

Implementation

The Git Migration

Carlos Mendez, SecureFirst's automation lead, managed the Git migration. The migration from Endevor to Git was the most politically sensitive phase — Endevor had been the source of truth for 25 years, and changing it felt to many like pulling the foundation from under the building.

Carlos followed the three-phase approach from Section 36.2. During Phase 1 (parallel operation), he set up a nightly synchronization job that exported all Endevor source to the Git repository. Developers could browse the code in Git, look at the history, and experiment with branching — all without any risk to production.

The first real test came when two developers, Maria and James, needed to modify the same copybook for different projects. Under Endevor, one of them would have had to wait until the other's change was promoted through to production — a delay of potentially months. With Git branches, they each created a feature branch, made their changes, and submitted pull requests. When James's change was merged first, Maria pulled the updated develop branch into her feature branch, resolved one minor conflict (they'd both changed adjacent fields), and merged successfully.

"That was the moment the light went on for the team," Carlos recalled. "Branching wasn't just a theoretical advantage — it saved Maria three weeks of waiting."

The Jenkins Pipeline

Building the Jenkins pipeline required solving a problem that isn't well-documented: connecting Jenkins running on Linux to the z/OS build environment. Carlos used a three-layer architecture:

Jenkins Master (Linux)
    |
    | SSH
    v
Jenkins Agent (z/OS USS)
    |
    | Zowe CLI + DBB
    v
z/OS Build (MVS datasets, DB2, CICS)

A Jenkins agent running in z/OS Unix System Services received build requests from the Jenkins master. The agent used Zowe CLI for data transfer and job submission, and DBB for dependency analysis and compilation.

The first pipeline build took 23 minutes — unacceptably long. Carlos profiled the pipeline and found that 80% of the time was in data transfer between USS and MVS datasets. He optimized by keeping source in USS-resident directories (where DBB could read them directly) and only transferring load modules to MVS datasets at the end. This reduced the build time to 7 minutes for a full build and under 2 minutes for incremental builds (the typical case).

The Testing Revolution

Implementing automated testing was the hardest phase — not technically, but culturally. SecureFirst had never had automated tests for COBOL. Testing meant a developer logged into a test CICS region, entered transactions, and looked at the screen. If the screen looked right, the code was "tested."

Yuki assigned Carlos to build the unit testing framework and recruited Priya Sharma, a mid-career developer with testing experience from her previous role at a Java shop, to design the test strategy.

Priya's approach was pragmatic:

1. Don't try to test everything. Start with the 50 most critical paragraphs — the ones that calculate premiums, process payments, determine eligibility, and update account balances.

2. Use AI to generate initial test cases. Following the techniques from Chapter 35, Priya used AI tools to generate test cases for each paragraph, then manually reviewed and corrected them.

3. Build the test harness once, use it everywhere. Carlos built a generic COBOL test harness that could load any program, set up working storage, PERFORM any paragraph, and compare results to expected values. Every test case was a data file, not a program.

4. Make tests run fast. Unit tests that mock external resources (DB2, VSAM, CICS) ran in under a second each. The entire unit test suite of 1,847 tests ran in 3 minutes.

The first time the unit test suite ran in the pipeline and caught a bug — a decimal precision error in the premium calculation that would have been invisible in manual testing — the team's attitude shifted. "When the pipeline caught that bug, José came to my desk and said, 'OK, I'm a believer,'" Priya recalled. "José had been the most vocal skeptic on the team."

The Deployment Automation

The final phase automated the 43-manual-step deployment process. Carlos wrote a deployment automation script that:

1. Created a deployment package with all load modules, DB2 changes, and CICS updates
2. Backed up current production modules
3. Promoted load modules to the production library
4. Executed DB2 BIND for any changed plans or packages
5. Performed CICS NEWCOPY for each changed program
6. Ran a suite of smoke tests against the production environment
7. Verified that key transaction response times were within acceptable ranges
8. Sent a deployment report to the change management system

The first automated deployment took 12 minutes. The previous manual process took 6 hours and two people. The first time the team deployed on a Wednesday morning (instead of the traditional Saturday night deployment window), Yuki got an email from the CTO: "Did something just deploy? I didn't even notice."

"That," Yuki told her team, "is exactly the point."

Results at Twelve Months

Cost Analysis

Payback period: 7 months.

Lessons Learned

1. Start with the build, not the tests. Automated builds are lower risk and faster to implement than automated tests. Getting the build automated first creates momentum and infrastructure that testing builds upon.

2. The first automated deployment is transformative. Nothing converts skeptics faster than a deployment that takes 12 minutes instead of 6 hours. Make that first deployment happen as early as possible, even if it's for a low-risk change.

3. Recruit champions from within, don't import them. Carlos and Priya were insiders who saw the value of DevOps. Their advocacy was far more credible than an outside consultant's would have been.

4. Measure and share everything. Yuki published pipeline metrics weekly. When the team could see lead time dropping and deployment success rate rising, the abstract benefits of DevOps became concrete.

5. The manual process is your fallback, not your enemy. Knowing that the manual process still worked if the pipeline failed reduced anxiety during the transition. "We didn't burn the boats," Yuki said. "We just built a better boat."

Discussion Questions

1. Yuki's CTO gave her a clear mandate with specific targets (monthly releases). How critical was this executive sponsorship to the project's success? What might have happened without it?

2. Carlos optimized the build pipeline from 23 minutes to 7 minutes by keeping source in USS. What are the security implications of storing COBOL source in USS rather than in MVS datasets behind RACF protection?

3. The team reached 62% automated test coverage in twelve months. Is that sufficient for a system processing 2.3 million insurance claims? What coverage level would you target, and how long would it take to reach?

4. The payback period was 7 months. How would you present this business case to a CTO who is skeptical of mainframe investment? What if the CTO asks, "Why not just replace the mainframe instead?"

5. Priya used AI to generate initial test cases (Chapter 35 techniques) within the DevOps pipeline (Chapter 36 infrastructure). How do Chapters 35 and 36 reinforce each other? Could you do mainframe DevOps without AI assistance? Could you do AI-assisted COBOL without DevOps infrastructure?