Chapter 36: DevOps for the Mainframe — Git, Jenkins/Zowe, CI/CD Pipelines, and Automated Testing on z/OS

"The day we stopped treating z/OS as a sacred shrine and started treating it as a deployment target was the day everything changed. We went from four releases a year to four releases a month, and our defect rate went down, not up." — Yuki Tanaka, SecureFirst Insurance

Spaced Review: Before we begin, revisit two concepts that form the foundation for mainframe DevOps. From Chapter 27, recall how mainframe monitoring works — the metrics and observability patterns you learned there become the feedback loop in our DevOps pipeline. From Chapter 32, recall the modernization strategy framework — DevOps is not a modernization strategy; it's the enabling infrastructure that makes any modernization strategy executable.

Threshold Concept: The Mainframe Is Just Another Platform in the Pipeline

This is the idea that, once internalized, transforms how you think about z/OS development. For decades, the mainframe existed in a parallel universe from the rest of IT. It had its own source control (SCLM, Endevor, ChangeMan), its own build process (manual JCL submission), its own testing approach (manual testing in dedicated regions), and its own deployment process (promotion through libraries). None of these integrated with the tools used by the rest of the enterprise.

The threshold concept is simple to state and difficult to internalize: the mainframe is just another platform in the pipeline. Just as your CI/CD pipeline builds Java for Linux, Node.js for Kubernetes, and Swift for iOS, it can build COBOL for z/OS. The same Git repository holds the source. The same Jenkins (or GitHub Actions, or GitLab CI) orchestrates the build. The same testing framework validates the result. The same deployment automation promotes the code.

Once you cross this conceptual threshold, the question changes from "How do we do DevOps on the mainframe?" to "How do we configure our existing DevOps pipeline for z/OS as a target?" That second question is dramatically easier to answer.

Diane at Pinnacle described her moment of crossing: "For years, I thought we needed special mainframe DevOps tools. Then I realized we just needed z/OS to participate in the same pipeline as everything else. The tooling exists — Zowe, IBM DBB, Jenkins plugins. The real barrier was in our heads."

36.1 Why DevOps for Mainframe: The Cultural Shift

The Problem with Traditional Mainframe Development

Traditional mainframe development follows a process that has been essentially unchanged since the 1980s:

1. Source control: Programs stored in PDS (Partitioned Data Set) members, managed by SCLM, Endevor, or ChangeMan. No branching. No merging. Locking-based concurrency (one developer at a time per member).

2. Build: Developer submits JCL to compile and link-edit. Build results checked by looking at SYSOUT. No automated build verification. If it compiled, it's "built."

3. Test: Developer tests in a personal or shared test CICS region. Test data is a copy of production (often stale). No automated tests. No regression suites. "I tested it and it works" is the testing report.

4. Deployment: Completed changes are promoted through library levels (DEV -> QA -> STAGING -> PROD) by a change management tool or by manual copy. Promotion is a manual, audited process with scheduled deployment windows (often monthly or quarterly).

5. Feedback: Production issues are discovered by operations, users, or batch abends. There is no automated feedback loop from production to development.

This process was adequate when change was slow. It is catastrophically inadequate in an era where business requirements change weekly, security patches must be deployed immediately, and regulatory compliance demands traceable, auditable, repeatable processes.

What DevOps Brings to the Mainframe

DevOps for the mainframe brings the same benefits it brings to any platform:

Speed: Changes move from development to production in hours or days rather than weeks or months. CNB went from quarterly releases to bi-weekly releases within six months of implementing their pipeline.

Quality: Automated testing catches defects before deployment. SecureFirst's defect escape rate dropped 67% in the first year of automated testing on z/OS.

Reliability: Automated, repeatable deployments eliminate the human errors inherent in manual promotion processes. At Federal Benefits, Sandra's team tracked zero deployment-related incidents in the nine months after implementing automated deployment, compared to four in the nine months before.

Traceability: Every change is tracked from requirement to deployment — who changed what, when, why, and what tests verified it. This is not just good practice; it's a regulatory requirement in financial services, healthcare, and government.

Collaboration: When mainframe developers and distributed developers use the same tools (Git, Jenkins, Jira), they can collaborate on cross-platform features without tool-boundary friction.

The Cultural Resistance

Let us be honest about the resistance you will encounter. Veteran mainframe developers have legitimate concerns:

"We've been doing this for 30 years and it works." True — but "works" means "we haven't had a catastrophic failure recently." It doesn't mean the process is efficient, scalable, or sustainable as the workforce shrinks.

"Our processes are proven and audited." Also true — but auditors are increasingly asking for automated traceability, not binder-based documentation. The audit landscape is evolving.

"Git doesn't understand EBCDIC." Technically true in the naive case — but Zowe and IBM's tools handle the encoding transparently. This is a solved problem.

"Jenkins can't submit JCL." It can, through Zowe CLI and IBM's z/OS plugins. This too is a solved problem.

"Automated testing on z/OS is impossible." It's not impossible; it's unfamiliar. Frameworks exist (IBM zUnit, Compuware Topaz for Total Test, Micro Focus Unit Testing). The challenge is building the test suites, not running them.

Rob Chen at CNB captured the cultural dynamic perfectly: "Every objection I heard was really the same objection: 'I don't want to learn new tools.' And I get that — I was the same way. But the alternative is watching our platform become unmaintainable. I'd rather learn Git at 57 than watch the mainframe die because we were too stubborn to change."

The Business Case in Numbers

The business case for mainframe DevOps is not theoretical. Here are the numbers from our anchor organizations:

These numbers translate directly to money. At SecureFirst, the annual savings exceeded $837,000 against a first-year investment of $480,000 — a payback period of seven months. At CNB, eliminating the source access wait time alone saved an estimated 2,400 developer-hours per year, equivalent to more than one full-time developer.

The numbers also translate to risk reduction. A deployment-related incident in a banking system can cost anywhere from $50,000 (minor processing error caught quickly) to $50 million (regulatory violation with customer impact). Reducing incident frequency by 88% is not just an efficiency gain — it's an existential risk mitigation.

The Regulatory Imperative

For organizations in regulated industries — banking, insurance, healthcare, government — DevOps is increasingly not optional. Regulators are raising the bar on what they expect:

OCC (banking): Expects traceable, auditable change management with evidence of testing. Manual processes that rely on sign-off sheets are increasingly questioned.

SOX (publicly traded companies): Requires internal controls over financial reporting, including controls over the systems that produce financial data. Automated, repeatable pipelines provide stronger controls than manual processes.

PCI DSS (payment card industry): Requires documented change control procedures with testing evidence. Automated pipelines produce this evidence as a natural byproduct.

HIPAA (healthcare): Requires access controls and audit trails for systems handling protected health information. Git-based change management with pull request approval provides stronger audit trails than traditional mainframe change management.

At Federal Benefits, Sandra's conversations with the agency's Inspector General revealed that the IG's office was actually more comfortable with automated, pipeline-based change management than with the traditional manual process. "The automated pipeline does the same thing every time," the IG examiner told Sandra. "The manual process depends on humans remembering to follow every step. I know which one I trust more."

36.2 Git for z/OS: Replacing SCLM and Endevor

Why Git?

Git is the standard version control system for software development across all platforms. It is used by over 95% of professional developers worldwide. By adopting Git for COBOL source management, mainframe teams gain:

• Branching and merging: Multiple developers can work on the same program simultaneously, in branches, and merge their changes when ready. This is impossible with PDS-based source management.
• Full history: Every change, by every developer, forever. No more "who changed this and when?" — Git knows.
• Code review: Pull request workflows enable peer review before changes are merged. This is a quantum leap from "I promoted it and it's in QA now."
• Integration: Every CI/CD tool, every IDE, every project management tool integrates with Git. Endevor integrates with Endevor.
• Distributed development: Developers can work offline, from home, from any location. The pandemic made this essential.

The SCLM/Endevor-to-Git Migration

Migrating from a traditional mainframe source management system to Git is not trivial, but it is well-understood. Here is the process that CNB followed under Kwame's leadership:

Phase 1: Parallel Operation (3 months)

Both systems run simultaneously. The existing SCLM/Endevor system remains the system of record. All source is exported to Git as a baseline. Developers continue using the existing system for production changes while learning Git on the side.

SCLM/Endevor (System of Record)
     |
     | nightly sync
     v
Git Repository (Read-Only Mirror)

This phase lets everyone get comfortable with Git without any risk. Developers explore the repository, try branching and merging, and learn the command line or GUI tools — all with production source they recognize.

Phase 2: Dual-Write (3 months)

Developers begin making changes in Git first, then applying those changes to SCLM/Endevor for production promotion. Both systems must stay in sync. This is painful but necessary — it builds muscle memory for the new workflow while maintaining the existing safety net.

Developer Workstation
     |
     |--- commit --> Git (Primary)
     |
     |--- also apply --> SCLM/Endevor (Production Promotion)

Phase 3: Git as System of Record (ongoing)

SCLM/Endevor is retired for active development. Git is the sole system of record. Production promotion is driven from Git through the CI/CD pipeline. SCLM/Endevor may be retained in read-only mode for historical access.

Developer Workstation
     |
     | commit/push
     v
Git Repository (System of Record)
     |
     | CI/CD Pipeline
     v
z/OS Build/Test/Deploy

Repository Structure

Organizing a Git repository for COBOL requires thoughtful structure. Here is the layout used at CNB:

ha-banking-system/
├── src/
│   ├── cobol/              # COBOL source programs
│   │   ├── HATXN1000.cbl
│   │   ├── HADEP1000.cbl
│   │   ├── HAWDL1000.cbl
│   │   └── ...
│   ├── copybook/           # Copybooks (COPY members)
│   │   ├── ACCTMSTR.cpy
│   │   ├── TXNREC.cpy
│   │   └── ...
│   ├── bms/                # BMS map sources
│   │   ├── HATXNM01.bms
│   │   └── ...
│   └── sql/                # DB2 DDL and stored procedures
│       ├── tables/
│       └── procedures/
├── jcl/                    # JCL procedures and jobs
│   ├── compile/
│   ├── batch/
│   └── deploy/
├── test/
│   ├── unit/               # Unit test cases
│   ├── integration/        # Integration test cases
│   └── data/               # Test data sets
├── config/
│   ├── dbb/                # IBM Dependency Based Build config
│   ├── jenkins/            # Jenkins pipeline definitions
│   └── zowe/               # Zowe CLI configuration
├── docs/                   # Generated and maintained documentation
├── Jenkinsfile             # Main CI/CD pipeline definition
├── .gitattributes          # Encoding and line-ending rules
└── README.md

The Encoding Problem

COBOL source on z/OS is stored in EBCDIC encoding. Git expects UTF-8 or ASCII. The .gitattributes file tells Git how to handle the conversion:

# .gitattributes for COBOL z/OS project
*.cbl   text working-tree-encoding=IBM-1047 zos-working-tree-encoding=IBM-1047
*.cpy   text working-tree-encoding=IBM-1047 zos-working-tree-encoding=IBM-1047
*.bms   text working-tree-encoding=IBM-1047 zos-working-tree-encoding=IBM-1047
*.jcl   text working-tree-encoding=IBM-1047 zos-working-tree-encoding=IBM-1047

When Zowe CLI or IBM's Rocket Git for z/OS transfers files between the z/OS Unix System Services (USS) file system and PDS members, the encoding conversion is handled transparently. Developers on Windows or Linux see ASCII/UTF-8 in their IDE; the z/OS build process sees EBCDIC in the PDS.

Branching Strategy

For mainframe development, we recommend a simplified Git Flow:

• main: Production code. Only merged to via pull requests from release branches. Tagged with version numbers.
• develop: Integration branch. Feature branches merge here after review.
• feature/JIRA-###-description: One branch per change ticket. Short-lived (days, not weeks).
• release/v#.#: Cut from develop when a release is ready. Bug fixes go here, then merge back to both main and develop.
• hotfix/JIRA-###: Emergency fixes branching from main, merged back to main and develop.

At Pinnacle, Ahmad initially resisted branching: "COBOL developers don't branch. We lock the member and work on it." After three months with feature branches, he became the team's strongest advocate: "I can work on my change without worrying about what Lisa is doing to the same copybook. When we're both done, we merge. If there's a conflict, Git shows us exactly where. In the old system, we'd just overwrite each other and not know until something broke."

Pull Requests and Code Review

The pull request workflow is perhaps the single most transformative practice that Git brings to mainframe development. In the traditional model, code review happens after deployment to a test environment (if it happens at all). In the Git model, code review happens before the change is merged into the integration branch — before it can affect anyone else.

A pull request for a COBOL change includes:

1. The diff: Exactly what lines changed, in context. Git's diff makes it impossible to miss a change that was made — unlike reviewing a full PDS member where you have to spot what's different.
2. The commit message: Why the change was made, what ticket it addresses, what the developer intended.
3. CI/CD results: The automated build and test results for the change, before it's merged.
4. Discussion thread: Reviewers can ask questions, request changes, and discuss the approach — all recorded permanently.

At CNB, Lisa Park was the first to notice the secondary benefit: "Pull request reviews are the best training tool we've ever had. When a junior developer reviews a senior developer's code, they learn patterns they'd never pick up from a textbook. When a senior developer reviews a junior developer's code, they catch issues before they become habits."

The pull request workflow also addresses the "bus factor" problem — the risk that knowledge is concentrated in one person. When every change is reviewed by at least one other person, knowledge spreads naturally. No change goes in that only one person understands.

Handling COBOL-Specific Git Challenges

Several challenges are specific to managing COBOL source in Git:

Line length enforcement. COBOL has strict column rules: columns 1-6 are sequence numbers (often blank in modern source), column 7 is the indicator area, columns 8-11 are Area A, columns 12-72 are Area B, and columns 73-80 are identification (traditionally ignored). Git doesn't enforce column positions, so a pre-commit hook should validate that COBOL source conforms to column rules.

Copy member naming. COBOL COPY statements reference member names without file extensions. Git needs file extensions to identify file types. The convention is to use .cpy for copybooks and .cbl for programs, with the build configuration stripping extensions when uploading to PDS members.

Case sensitivity. COBOL is case-insensitive, but Git on Linux is case-sensitive. Standardize all filenames to uppercase and enforce this with a pre-commit hook to avoid "file not found" errors during the build.

Binary artifacts. Load modules, object decks, and DBRM files are binary. They should never be committed to Git (use .gitignore). The CI/CD pipeline produces these artifacts fresh from source on every build — that's the whole point of CI.

Zowe CLI Integration

Zowe CLI is the bridge between the Git-based development workflow and the z/OS platform. Key commands:

# Upload COBOL source from Git working directory to z/OS PDS
zowe zos-files upload file-to-data-set src/cobol/HATXN1000.cbl \
  "DEVL.COBOL(HATXN1000)" --encoding IBM-1047

# Download a compiled listing for review
zowe zos-files download data-set "DEVL.LISTING(HATXN1000)" \
  --file build/listings/HATXN1000.lst

# Submit a compile JCL and get the job output
zowe zos-jobs submit local-file jcl/compile/COMPILE.jcl \
  --wait-for-active --wait-for-output

# Check job return code
zowe zos-jobs view job-status-by-jobid JOB12345

These commands are the building blocks of the CI/CD pipeline we'll construct in Section 36.3. The power of the Zowe CLI is that it turns z/OS into a scriptable platform. Anything you can do interactively from a 3270 terminal — submit a job, browse a dataset, check a job's return code — you can do from a shell script. And anything you can do from a shell script, you can do from a CI/CD pipeline. This is the technical mechanism by which the threshold concept becomes reality: the mainframe becomes just another platform that your automation talks to through a CLI.

36.3 CI/CD Pipelines: Jenkins, IBM DBB, Zowe CLI

The Pipeline Architecture

A CI/CD pipeline for z/OS COBOL follows the same fundamental pattern as any CI/CD pipeline:

Source       Build        Test         Deploy
  |            |            |            |
  v            v            v            v
Git Push --> Compile --> Unit Test --> Promote
             Link        Integration   to Target
             Bind        Regression    Library

The difference is in the "how," not the "what." Instead of javac for build, we use the Enterprise COBOL compiler. Instead of JUnit for testing, we use zUnit or a COBOL testing framework. Instead of Kubernetes deployment, we use library promotion.

IBM Dependency Based Build (DBB)

IBM DBB is the key technology that makes mainframe CI/CD practical. DBB replaces traditional JCL-based compilation with a build system that:

• Understands dependencies: DBB analyzes COBOL source, COPY statements, and CALL relationships to build a dependency graph. When you change a copybook, DBB knows which programs need recompilation — and it only recompiles those programs, not everything.
• Integrates with Git: DBB reads source from USS directories (where Git stores it) and produces load modules in z/OS datasets.
• Produces build reports: DBB generates machine-readable build reports that Jenkins can parse for pass/fail determination.
• Supports incremental builds: On a system with 500 programs, changing one copybook might require recompiling 12 programs. DBB does exactly that, not 500 compiles.

DBB configuration for the HA banking system:

// dbb-build.groovy — IBM DBB build script
@groovy.transform.BaseScript com.ibm.dbb.groovy.ScriptLoader baseScript

import com.ibm.dbb.build.*
import com.ibm.dbb.dependency.*
import com.ibm.dbb.repository.*

// Build properties
def properties = BuildProperties.getInstance()
properties.load(new File("${properties.workspace}/config/dbb/build.properties"))

// Source directories
def sourceDir = "${properties.workspace}/src/cobol"
def copyDir = "${properties.workspace}/src/copybook"

// Dependency resolution
def resolver = new DependencyResolver()
resolver.setSourceDir(sourceDir)
resolver.setCopyDir(copyDir)

// Get list of changed files from Git
def changedFiles = getChangedFiles(properties.gitHash)

// Resolve impacted programs (programs that COPY changed copybooks)
def impactedPrograms = resolver.resolveImpacts(changedFiles)

// Build only what changed plus impacted programs
def buildList = changedFiles + impactedPrograms
buildList.each { program ->
    def compile = new CompileStep()
    compile.setSource("${sourceDir}/${program}")
    compile.setCopyPath(copyDir)
    compile.setLoadLib(properties.loadLib)
    compile.execute()
}

The Jenkinsfile

The Jenkinsfile defines the complete CI/CD pipeline. Here is the pipeline used at SecureFirst, developed by Yuki and Carlos (see code/example-01-jenkinsfile.groovy for the complete, annotated version):

pipeline {
    agent { label 'zos-agent' }

    environment {
        ZOWE_PROFILE = 'ha-banking-dev'
        DBB_HOME = '/usr/lpp/dbb/v2'
        BUILD_WORKSPACE = "/u/jenkins/builds/${env.BUILD_NUMBER}"
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
                sh "mkdir -p ${BUILD_WORKSPACE}"
                sh "cp -r . ${BUILD_WORKSPACE}/"
            }
        }

        stage('Build') {
            steps {
                sh """
                    ${DBB_HOME}/bin/groovyz \
                        ${BUILD_WORKSPACE}/config/dbb/dbb-build.groovy \
                        --workspace ${BUILD_WORKSPACE} \
                        --hlq BLDHA.${env.BUILD_NUMBER}
                """
            }
            post {
                always {
                    archiveArtifacts 'build/reports/**'
                }
                failure {
                    notifyTeam('Build failed')
                }
            }
        }

        stage('Unit Test') {
            steps {
                sh """
                    ${DBB_HOME}/bin/groovyz \
                        ${BUILD_WORKSPACE}/test/run-unit-tests.groovy \
                        --loadlib BLDHA.${env.BUILD_NUMBER}.LOAD
                """
            }
            post {
                always {
                    junit 'test/results/*.xml'
                }
            }
        }

        stage('Integration Test') {
            steps {
                sh """
                    ${BUILD_WORKSPACE}/test/run-integration-tests.sh \
                        --region HATESTP \
                        --loadlib BLDHA.${env.BUILD_NUMBER}.LOAD
                """
            }
        }

        stage('Deploy to QA') {
            when {
                branch 'develop'
            }
            steps {
                sh """
                    zowe zos-files upload dir-to-pds \
                        ${BUILD_WORKSPACE}/build/load \
                        'QA.HA.LOADLIB' --encoding binary
                """
            }
        }

        stage('Deploy to Production') {
            when {
                branch 'main'
            }
            input {
                message 'Deploy to production?'
                submitter 'release-managers'
            }
            steps {
                sh """
                    ${BUILD_WORKSPACE}/jcl/deploy/promote-to-prod.sh \
                        --build ${env.BUILD_NUMBER} \
                        --approver ${env.APPROVED_BY}
                """
            }
        }
    }

    post {
        success {
            notifyTeam("Pipeline SUCCESS: Build #${env.BUILD_NUMBER}")
        }
        failure {
            notifyTeam("Pipeline FAILED: Build #${env.BUILD_NUMBER}")
        }
    }
}

Build Optimization: Making CI Practical

For CI to work, builds must be fast. A full recompilation of 500 COBOL programs takes hours and is unacceptable for a pipeline that should complete in minutes. DBB's dependency-based incremental build is the key optimization, but there are additional techniques:

Parallel compilation. DBB can compile multiple programs simultaneously, limited by the number of compiler address spaces the LPAR can sustain. At SecureFirst, Carlos configured DBB for eight parallel compilations, reducing a 12-program rebuild from 18 minutes to under 4 minutes.

Caching compiled objects. Load modules that haven't changed don't need to be relinked. DBB tracks which programs are current and which need relinking based on whether their object code or any linked subprogram changed.

Separating DB2 binds. DB2 BIND PACKAGE is often the slowest step in the build. DBB can identify which programs have SQL changes and only rebind those packages, while reusing existing packages for programs where only non-SQL code changed.

Build workspace reuse. Creating build datasets for every pipeline run is expensive (DASD allocation, catalog updates). A pool of pre-allocated build workspaces, assigned to pipeline runs on demand and released after completion, eliminates this overhead.

With these optimizations, CNB's pipeline completes a typical incremental build (1-5 changed programs) in under 3 minutes. Even a major copybook change affecting 30 programs completes in under 12 minutes.

Pipeline Stages Explained

Checkout: The pipeline pulls the latest code from Git. On z/OS, the Git repository lives in USS (Unix System Services), where standard Git commands work. The code is copied to a build-specific workspace to avoid conflicts between concurrent builds.

Build: IBM DBB compiles all changed programs and their dependents. The build produces load modules in a build-specific High-Level Qualifier (HLQ). If any program fails compilation, the pipeline stops and the team is notified. The build report is archived for audit.

Unit Test: Automated unit tests run against the newly compiled load modules. Tests are executed in an isolated test environment. Results are published in JUnit XML format, which Jenkins displays as test reports.

Integration Test: Integration tests exercise the programs in a realistic environment — CICS region, DB2 subsystem, VSAM files — but with test data. These tests verify that programs work together correctly, not just individually.

Deploy to QA: If the build and tests pass on the develop branch, the load modules are automatically promoted to the QA library. The QA team can then run their acceptance tests.

Deploy to Production: Production deployment requires manual approval from a release manager. This is the human gate in an otherwise automated pipeline. The approval is recorded in Jenkins for audit purposes. The production stage also includes a backup step (preserving the current load modules before overwriting them with the new version) and a smoke test step (verifying that the deployed programs respond correctly). If the smoke test fails, an automatic rollback restores the backed-up modules.

Pipeline as Code: Why It Matters

The Jenkinsfile is not just a convenience — it's a fundamental architectural decision. By defining the pipeline in a file that lives in the Git repository alongside the COBOL source, you gain several critical capabilities:

Version control of the pipeline itself. When you change the pipeline (add a test stage, modify compiler options, change the promotion path), the change is tracked in Git with the same rigor as source code changes. You can see who changed the pipeline, when, and why. You can revert a pipeline change that caused problems. You can compare today's pipeline to last month's pipeline.

Branch-specific pipelines. Different branches can have different pipelines. The develop branch pipeline deploys to QA automatically. The main branch pipeline requires production approval. A feature branch pipeline might skip deployment entirely and only build and test. This flexibility comes naturally when the pipeline definition is part of the branch.

Pipeline review. Changes to the Jenkinsfile go through the same pull request review process as code changes. When someone proposes adding a new pipeline stage or changing a compiler option, the team reviews and approves the change before it takes effect. This prevents accidental pipeline modifications that could affect deployment quality.

Reproducibility. Given a specific Git commit, you can reconstruct exactly what the pipeline did — because the pipeline definition at that commit is the one that executed. This is critical for audit: regulators can ask "what build process produced the code running in production?" and the answer is deterministic.

At Pinnacle, Diane initially kept her pipeline configuration in a shared Jenkins folder rather than in Git. After a junior developer accidentally modified a shared pipeline that affected all projects, Diane moved all pipeline definitions to Jenkinsfiles in each project's repository. "The Jenkinsfile is code," she told her team. "It gets the same discipline as code."

The Zowe Pipeline Script

For shops that prefer shell scripting over Jenkins declarative pipelines, the Zowe CLI enables pipeline automation through bash scripts (see code/example-02-zowe-pipeline.sh for the complete version). The key advantage of the Zowe approach is simplicity — any CI/CD tool that can execute shell commands can drive the pipeline.

36.4 Automated Testing on z/OS: Unit, Integration, and Regression

The Testing Gap

The single biggest obstacle to mainframe DevOps is not Git, not Jenkins, not Zowe — it's the absence of automated tests. You cannot have continuous integration without tests. You cannot have confidence in continuous deployment without tests. And most mainframe shops have few or no automated tests for their COBOL programs.

This is not because testing is impossible on z/OS. It's because the mainframe development culture evolved before automated testing was the norm, and the inertia has been enormous.

Unit Testing with IBM zUnit

IBM zUnit (formerly COBOL Unit Test Framework) enables unit testing of individual COBOL paragraphs in isolation. A zUnit test:

1. Sets up input data in WORKING-STORAGE and LINKAGE SECTION fields
2. PERFORMs the target paragraph
3. Verifies that the output fields contain expected values

TEST CASE: 2100-VALIDATE-ACCT-STATUS — Active Account No Hold
  SETUP:
    WS-ACCT-STATUS-CD = 'A'
    WS-ACCT-HOLD-CD = SPACES
    WS-VALID-FLAG = 'N'
    WS-HOLD-OVERRIDE-AUTH = 'N'
    WS-HOLD-OVERRIDE-CTR = 0
    WS-HOLD-REJECT-CTR = 0
  PERFORM: 2100-VALIDATE-ACCT-STATUS
  VERIFY:
    WS-VALID-FLAG = 'Y'
    WS-HOLD-OVERRIDE-CTR = 0
    WS-HOLD-REJECT-CTR = 0
    WS-ERROR-MSG = SPACES

The test case is defined in XML or a proprietary format, and zUnit generates COBOL code to execute the test. The test program PERFORMs the target paragraph with the specified inputs and asserts the expected outputs.

Integration Testing

Integration tests verify that programs work together correctly. In the HA banking system, an integration test for deposit processing would:

1. Insert a test transaction record into the input queue
2. Start the transaction processing program (CICS or batch)
3. Verify that the account balance was updated correctly in DB2
4. Verify that the audit log record was written
5. Verify that the response to the terminal or output file is correct

Integration testing on z/OS requires:

• A dedicated test CICS region with its own DB2 subsystem and VSAM files
• Test data management — the ability to set up, verify, and tear down test data
• Transaction isolation — tests must not interfere with each other or with other test activity

At Pinnacle, Diane's team implemented integration testing using a combination of Zowe CLI (to submit transactions and check results) and custom COBOL test harness programs (to manage test data setup and verification):

#!/bin/bash
# Integration test: Deposit processing

# Setup: Insert test account and transaction
zowe zos-files upload stdin-to-data-set "TEST.DEPOSIT.INPUT" <<EOF
1234567890DP000010000020260315BRANCH01TELLER01
EOF

# Execute: Run the deposit processing batch step
JOB_ID=$(zowe zos-jobs submit local-file test/jcl/TEST-DEPOSIT.jcl \
    --rff jobid --rft string)

# Wait for completion
zowe zos-jobs view job-status-by-jobid $JOB_ID --wait-for-output

# Verify: Check account balance in DB2
BALANCE=$(zowe zos-files download data-set "TEST.DEPOSIT.RESULT" \
    --rfj | jq -r '.data')

# Assert
if [ "$BALANCE" = "0000010000" ]; then
    echo "PASS: Deposit processing correct"
    exit 0
else
    echo "FAIL: Expected balance 0000010000, got $BALANCE"
    exit 1
fi

Regression Testing

Regression tests verify that existing functionality still works after changes. These are the tests generated in Chapter 35 using AI assistance, now automated in the CI/CD pipeline.

The regression test suite for the HA banking system runs as part of every pipeline execution:

1. Setup: Load test data into the test environment (VSAM files, DB2 tables)
2. Execute: Run each test scenario (batch jobs, CICS transactions)
3. Verify: Compare actual output to expected output
4. Report: Generate JUnit-format XML results for Jenkins
5. Cleanup: Remove test data (or reset to baseline)

The complete regression suite for the HA banking system targets:

At SecureFirst, Carlos's team runs 3,100 regression tests nightly. The test suite takes 47 minutes to execute — compare this to the two weeks it used to take for manual regression testing before each quarterly release. More importantly, the automated tests run after every change, not just before releases. This means regression bugs are caught within hours of introduction, when the developer still has the context fresh in mind, rather than weeks later when the developer has moved on to other work and must context-switch back to understand the failure.

The cost comparison is illuminating. Manual regression testing at SecureFirst required three testers working full-time for two weeks — 240 person-hours per quarterly release, or 960 person-hours per year. The automated test suite runs unattended in 47 minutes and produces results that are more comprehensive, more consistent, and available every day rather than four times a year. The testers weren't laid off — they were redeployed to exploratory testing, performance testing, and test design, where their expertise adds more value than executing the same manual test scripts for the hundredth time.

Mocking and Stubbing on z/OS

Unit testing COBOL paragraphs that access external resources (DB2, VSAM, CICS services) requires mocking those resources. Common approaches:

DB2 mocking: Replace EXEC SQL statements with test-controlled data using compile-time conditional compilation or test harness programs that intercept SQL calls.

VSAM mocking: Use test-specific VSAM files loaded with known data. The test JCL points DD names to test datasets rather than production datasets.

CICS mocking: IBM's CICS TS provides a testing facility that simulates CICS API calls without requiring a full CICS region. For more complex scenarios, use a dedicated test CICS region.

CALL mocking: Replace called subprograms with stub programs that return predetermined values. This isolates the program under test from its dependencies.

At CNB, Rob built a mocking framework that uses JCL DD overrides to redirect all file I/O to test datasets:

//UNITTEST  EXEC PGM=TESTHRNS,PARM='HADEP1000,2100-VALIDATE'
//STEPLIB   DD DSN=TEST.LOAD.LIB,DISP=SHR
//          DD DSN=PROD.LOAD.LIB,DISP=SHR
//ACCTMSTR  DD DSN=TEST.ACCTMSTR.DATA,DISP=SHR    * Test data, not production
//TESTCASE  DD DSN=TEST.CASES.VALIDATE,DISP=SHR    * Test case definitions
//TESTRSLT  DD DSN=TEST.RESULTS.VALIDATE,DISP=OLD  * Test results output
//SYSOUT    DD SYSOUT=*

The test harness program (TESTHRNS) loads the target program, sets up working storage per the test case, PERFORMs the target paragraph, and writes the results for comparison.

Test Environment Management

One of the chronic problems in mainframe testing is test environment contention. In many shops, there's one test CICS region shared by all developers, and accessing it requires coordination (or luck). DevOps demands dedicated, isolated test environments.

There are three approaches to test environment isolation:

Dedicated regions per pipeline. Each pipeline run gets its own test CICS region and DB2 subsystem. This provides perfect isolation but is expensive in terms of resources. Only practical for shops with substantial LPAR capacity.

Shared region with data isolation. All pipeline runs share a test CICS region but use different DB2 schemas or VSAM file qualifiers. The pipeline assigns a unique qualifier (e.g., BUILD0042) to each run, and all test data is created under that qualifier. This provides data isolation without region multiplication.

Containerized testing. Using IBM's Wazi as a Service or Zowe's containerized z/OS emulation, test environments can be spun up on demand in a cloud or private cloud environment. This is the most flexible approach but requires investment in container infrastructure and may not perfectly replicate production z/OS behavior.

At SecureFirst, Carlos implemented the shared-region approach with a twist: the pipeline allocated test data at the start of each run, ran all tests, and then deallocated the test data — leaving no residue. This meant the test environment was always clean at the start of every run, eliminating the "my test failed because someone else's data was in the way" problem that had plagued the team for years.

Test Reporting and Visibility

Automated tests only provide value if their results are visible. The CI/CD pipeline should publish test results in a standard format (JUnit XML is the de facto standard) that the CI server can display.

Jenkins displays JUnit results as: - A test result summary (passed, failed, skipped, total) - A trend chart showing test results over time - Drill-down to individual failed tests with failure messages - Comparison to previous run (new failures, fixed failures)

This visibility transforms the team's relationship with testing. Instead of testing being something that happens in a black box (the tester's head), test results are visible to everyone — developers, managers, operations, and auditors. When the test trend chart shows a decline in pass rate, the team investigates before it becomes a production problem.

36.5 Deployment Automation: Promotion Paths and Rollback

The Promotion Path

Traditional mainframe deployment follows a library promotion model:

DEV Libraries --> QA Libraries --> STAGING Libraries --> PROD Libraries

In a DevOps pipeline, the promotion is automated but the gates between environments can be automated (for lower environments) or manual (for production):

Git develop  --(auto)--> Build --> Test --> DEV Libraries
Git release  --(auto)--> Build --> Test --> QA Libraries
QA Sign-off  --(manual)--> STAGING Libraries
Release Mgr  --(manual)--> PROD Libraries

Each promotion copies load modules, DBRMs (DB2 resource modules), and BMS maps from the source library to the target library. The promotion script also:

• Binds DB2 plans and packages in the target subsystem
• Installs CICS programs (NEWCOPY or PHASEIN)
• Updates VSAM file definitions if changed
• Records the promotion in the change management system

Blue-Green Deployment

Blue-green deployment — running two identical production environments and switching traffic between them — is challenging but possible on z/OS.

The approach at Pinnacle uses two sets of CICS regions:

Load Balancer
    |
    |--- Blue CICS Regions (current production)
    |       AOR1-BLUE, AOR2-BLUE, AOR3-BLUE
    |
    |--- Green CICS Regions (new version)
            AOR1-GREEN, AOR2-GREEN, AOR3-GREEN

Deployment process:

1. Deploy new code to the Green regions (currently inactive)
2. Run smoke tests against the Green regions
3. Shift traffic gradually: 10% -> 25% -> 50% -> 100%
4. Monitor for errors at each increment
5. If errors detected, shift all traffic back to Blue immediately
6. If successful, Blue becomes the idle set for the next deployment

This requires duplicate CICS infrastructure, which is expensive. For batch processing, blue-green is simpler: run the new version of a batch job in parallel with the old version, compare outputs, and switch when satisfied.

Rollback Strategy

Every deployment must have a rollback plan. In the CI/CD pipeline, rollback is automated:

For CICS programs: Use CICS PHASEIN to install the new version. If rollback is needed, PHASEIN the previous version from the backup library. No CICS restart required.

For batch programs: Keep the previous version in a backup library. If the new version fails, the restart JCL points to the backup library.

For DB2 changes: This is the hard part. Schema changes (ALTER TABLE) are not easily rolled back. The rollback strategy for DB2 changes must be defined in advance, and may involve backward-compatible schema changes deployed in advance of the code that uses them.

For copybook changes: If a copybook change affects multiple programs, all affected programs must be deployed together or rolled back together. This is why DBB's dependency analysis is critical — it ensures that all affected programs are included in the build.

At Federal Benefits, Sandra implemented a "deployment package" concept: every production deployment is a named, versioned package containing all load modules, DB2 changes, JCL changes, and copybook changes that go together. Rolling back means deploying the previous package. The package is immutable once created — you never modify a deployment package, you create a new one.

CICS Deployment Strategies: NEWCOPY vs. PHASEIN

The choice between CICS NEWCOPY and PHASEIN has significant implications for deployment risk:

NEWCOPY immediately replaces the program in the CICS program library. Any transaction that starts after the NEWCOPY uses the new version. However, transactions already in flight may be using the old version. If the old and new versions have different data expectations (different copybook layouts, for example), in-flight transactions could abend.

PHASEIN is the safer option. It installs the new version but allows in-flight transactions using the old version to complete normally. New transactions get the new version. There's no abend risk from version mismatch during the transition, but there's a brief period where both versions are running concurrently.

For the HA banking system, we always use PHASEIN for production deployments. The brief concurrent-version period is far preferable to the risk of abending in-flight transactions that might be in the middle of updating account balances.

Canary Deployments

Beyond blue-green, a canary deployment strategy deploys the new version to a single CICS region first (the "canary"), monitors it for a defined period (typically 15-30 minutes), and then rolls out to the remaining regions if no issues are detected.

At Pinnacle, Diane implemented canary deployments for their online transaction system:

1. Deploy to AOR-1 (the canary)
2. Monitor for 15 minutes: transaction response time, error rate, abend rate
3. If metrics are within tolerance, deploy to AOR-2 through AOR-6
4. If metrics exceed tolerance, rollback AOR-1 and investigate

The canary approach limits the blast radius of a bad deployment to approximately 16% of transaction volume (one of six AORs), rather than exposing all users simultaneously. The first time the canary caught a problem — a DB2 deadlock introduced by a SQL change that only manifested under real production concurrency — Diane's team avoided what would have been a system-wide outage.

Emergency Fix (Hotfix) Process

Even with a well-designed CI/CD pipeline, there will be times when an emergency fix must be deployed immediately — a production-down situation where waiting for the full pipeline is unacceptable.

The hotfix process maintains the pipeline's integrity while allowing urgency:

1. Branch from main: Create a hotfix branch directly from main (not develop)
2. Minimal change: The fix should be as small as possible — fix the bug, nothing else
3. Abbreviated pipeline: Skip integration tests if necessary, but never skip compilation and unit tests
4. Emergency approval: A single senior approver (on-call manager) can approve production deployment
5. Immediate deployment: Deploy to production using the standard automated promotion
6. Post-deployment: Merge the hotfix back to both main and develop. Run the full pipeline including integration tests. Create a follow-up ticket for any cleanup needed.

The key principle: the hotfix process is a faster path through the pipeline, not a path around it. Every production change, including emergencies, goes through automated build and basic test. The pipeline is never bypassed entirely — that way lies the kind of untracked, untested changes that cause the next emergency.

36.6 Monitoring and Observability in DevOps

The Feedback Loop

DevOps is not just about delivery — it's about feedback. The monitoring and observability practices from Chapter 27 become the right half of the DevOps infinity loop:

Plan --> Code --> Build --> Test --> Deploy --> Monitor --> Learn --> Plan

Without the monitoring/learn/plan stages, you have continuous delivery but not continuous improvement.

Metrics That Matter

For the HA banking system's DevOps pipeline, we track:

Deployment Metrics: - Deployment frequency: How often do we deploy to production? (Target: weekly or better) - Lead time: How long from code commit to production deployment? (Target: under 1 week) - Change failure rate: What percentage of deployments cause incidents? (Target: under 5%) - Mean time to recovery (MTTR): When a deployment fails, how quickly do we recover? (Target: under 1 hour)

These are the "DORA metrics" (DevOps Research and Assessment), and they apply to mainframe just as much as to distributed systems.

Pipeline Metrics: - Build success rate (target: >95%) - Test pass rate (target: >99%) - Build time (track and optimize) - Test execution time (track and optimize) - Pipeline queue time (indicates capacity constraints)

Application Metrics (from Chapter 27): - Transaction response time - Batch job elapsed time - CPU consumption per transaction - DASD I/O rates - DB2 deadlock frequency

Integrating z/OS Monitoring with Enterprise Observability

Modern observability platforms (Splunk, Datadog, Dynatrace, Elastic) can ingest z/OS metrics via SMF (System Management Facilities) records and Zowe-based metric exporters. This puts mainframe performance data alongside distributed system data in a single dashboard.

At CNB, Kwame's team implemented a Grafana dashboard that shows:

• Jenkins pipeline status (left panel)
• z/OS system health (center panel — CPU, memory, I/O)
• Application performance (right panel — transaction response times, batch durations)
• Deployment timeline (bottom panel — when each deployment occurred, correlated with metric changes)

This unified view lets the team see the impact of deployments immediately. "When we deploy at 2 PM and the response time graph spikes at 2:05 PM, we know to roll back," Kwame says. "Before the dashboard, we wouldn't know about the spike until operations called at 3 PM."

Alerting and Incident Response

The DevOps pipeline should be integrated with the incident response process:

1. Pipeline failure alert: When a build or test fails, the team is notified immediately (Slack, email, PagerDuty)
2. Deployment monitoring alert: For the first hour after deployment, monitoring thresholds are tightened. Any anomaly triggers an alert.
3. Automatic rollback trigger: If critical metrics (error rate, response time) exceed thresholds within 30 minutes of deployment, automatic rollback is initiated without waiting for human intervention.
4. Post-incident review: Every deployment-related incident triggers a blameless post-mortem that feeds improvements back into the pipeline.

The Blameless Post-Mortem

The blameless post-mortem is a practice borrowed from site reliability engineering (SRE) that is essential for DevOps maturity. After any deployment-related incident, the team conducts a structured review that focuses on what happened, why it happened, and how to prevent it — without blaming individuals.

The post-mortem template used at CNB:

INCIDENT POST-MORTEM: [Incident ID]
Date: [Date]
Duration: [Start time to resolution]
Impact: [What was affected and how]

Timeline:
  [Chronological sequence of events]

Root Cause:
  [Technical cause, not "person X made an error"]

Contributing Factors:
  [Process gaps, tooling issues, monitoring blind spots]

What Went Well:
  [Detection speed, response effectiveness, communication]

What Can Be Improved:
  [Pipeline changes, test additions, monitoring enhancements]

Action Items:
  [Specific, assigned, time-bound improvements]

The "blameless" part is critical. If developers fear being punished for deployment incidents, they will resist automated deployment (preferring the safety of the old manual process where blame was diffused). If they know that incidents are treated as learning opportunities, they embrace the faster deployment cadence because they know the system will improve with every incident.

At SecureFirst, Yuki made blameless post-mortems a weekly practice — even for near-misses caught by the pipeline before reaching production. "We learn as much from the bugs the pipeline catches as from the ones it doesn't," she explains. "The pipeline caught it this time. The post-mortem asks: how do we catch similar bugs earlier?"

36.7 Cultural Change Management

The People Problem

The technology for mainframe DevOps exists and works. The real challenge is cultural. You are asking people who have successfully managed z/OS systems for decades to change how they work. This requires empathy, patience, and strategic change management.

The Change Management Framework

Yuki Tanaka developed a change management framework at SecureFirst that has since been adopted by CNB and Pinnacle:

Phase 1: Awareness (Month 1-2)

• Demonstrate the DevOps toolchain to the entire team
• Show real examples from other mainframe shops (not just distributed examples)
• Address concerns honestly — don't dismiss legitimate worries
• Identify champions: developers who are curious and willing to try new things
• Key message: "This is coming. Let's shape it together rather than have it imposed on us."

Phase 2: Education (Month 2-4)

• Git training for all developers (hands-on, with COBOL examples)
• Jenkins/pipeline training for build and deployment leads
• Testing framework training for the test team
• Training must use mainframe examples — not Java or Python examples that feel irrelevant
• Key message: "The concepts are the same. The tools are slightly different."

Phase 3: Pilot (Month 3-6)

• Select one low-risk application for the full DevOps treatment
• The pilot team includes champions plus one skeptic (skeptics who convert become the strongest advocates)
• Success criteria defined in advance (deployment frequency, defect rate, developer satisfaction)
• Allow the pilot team to struggle — learning requires making mistakes in a safe environment
• Key message: "This is an experiment. We'll learn from what works and what doesn't."

Phase 4: Expansion (Month 6-12)

• Pilot team members become mentors for the next wave of applications
• Standardize the pipeline configuration (Jenkinsfile templates, DBB configuration, test frameworks)
• Address the issues discovered during the pilot
• Begin migrating source from SCLM/Endevor to Git for the expanding set of applications
• Key message: "The pilot proved it works. Now let's scale it."

Phase 5: Normalization (Month 12+)

• DevOps is how we work, not a special initiative
• All new development uses the Git/Jenkins/automated-test pipeline
• Legacy applications are migrated on a risk-prioritized schedule
• Continuous improvement of the pipeline based on team feedback
• Key message: "This is just how we work now."

Handling Resistance

Resistance comes in predictable patterns. Here's how to handle each:

"I'm too old to learn new tools." Response: "Git has a learning curve, but it's shorter than you think. You learned ISPF, you learned SCLM, you learned each new version of Enterprise COBOL. This is no different. And we're providing training and mentoring — you're not doing this alone."

"Automated testing can't replace manual testing." Response: "You're right — automated testing and manual testing catch different things. We're not eliminating manual testing. We're adding automated testing to catch the routine regressions so your manual testing can focus on the complex scenarios that require human judgment."

"This works for web apps, not mainframe." Response: "That's what mainframe shops said about version control in 2000, about IDEs in 2010, and about cloud integration in 2020. Each time, the tools adapted and the mainframe benefited. DevOps is the same."

"What if the pipeline breaks?" Response: "The pipeline is code. When it breaks, we fix it, just like we fix application code. And while the pipeline is down, we can always fall back to the manual process. The pipeline doesn't eliminate the manual process — it automates it."

Measuring Cultural Adoption

Track these indicators to measure cultural adoption:

• Git commit frequency: Are developers committing daily? Weekly? Monthly?
• Branch usage: Are developers using feature branches, or committing directly to develop?
• Test contribution: Are developers adding tests when they make changes?
• Pipeline engagement: Are developers monitoring their pipeline runs, or ignoring them?
• Tool satisfaction surveys: Quarterly anonymous surveys on tool satisfaction and pain points

At SecureFirst, Yuki's quarterly surveys showed developer satisfaction with the DevOps toolchain rising from 28% at introduction to 71% at twelve months. The biggest satisfaction driver was "time saved on deployment" — developers appreciated getting back the hours they used to spend on manual promotion procedures.

Training: What Works and What Doesn't

Based on the experience of all four anchor organizations, here is what works for DevOps training on mainframe teams:

What works:

Hands-on labs using the team's own code. Abstract Git exercises with sample projects don't transfer. When developers practice branching and merging with COBOL programs they recognize, the learning sticks. At CNB, Lisa designed a training exercise where each developer created a feature branch, modified a real (but low-risk) program, submitted a pull request, reviewed a colleague's pull request, and resolved a merge conflict — all in an afternoon.

Pairing junior and senior developers. The junior developer teaches Git; the senior developer teaches the business context. Both learn from the exchange. This pairing model was used successfully at CNB (Marcus and Rob), SecureFirst (Priya and Jose), and Pinnacle (Ahmad mentoring new hires). Rob later said the pairing with Marcus was the most productive learning experience of his career — in both directions.

Incremental introduction. Don't try to teach Git, Jenkins, Zowe, testing, and deployment automation all at once. Start with Git (weeks 1-4), add the pipeline (weeks 5-8), add testing (weeks 9-16), add deployment automation (weeks 17-24). Each layer builds on the previous.

Celebrating mistakes in safe environments. The pilot project should be low-risk enough that mistakes don't cause anxiety. When a developer causes a merge conflict in the pilot, it's a learning opportunity. When a pipeline fails because of a misconfiguration, the team debugs it together and the team's collective knowledge grows.

What doesn't work:

Vendor-led training with generic examples. Mainframe developers disengage when shown Java CI/CD pipelines. The training must use COBOL, JCL, and z/OS-specific tooling. If your vendor can't provide COBOL-specific training materials, create your own.

Mandating adoption without training. Telling developers "use Git starting Monday" without providing hands-on training produces fear and resentment. Budget a minimum of two full days of hands-on training per developer.

Ignoring the ISPF comfort zone. Many veteran developers live in ISPF. Forcing them to abandon ISPF entirely for VS Code or Eclipse is unnecessary and counterproductive. Modern tools like Zowe Explorer provide Git and pipeline access from within familiar environments. Meet developers where they are, and let them migrate to modern IDEs at their own pace.

Treating DevOps as an IT initiative rather than a team initiative. DevOps works when the team owns it. It fails when it's imposed by a management directive or an external consulting team that doesn't understand mainframe culture. The champions must come from within the team.

The Long View: Where Mainframe DevOps Is Heading

Looking ahead, mainframe DevOps is converging with several other trends:

AI-assisted development (Chapter 35) integrates naturally with the CI/CD pipeline. AI-generated documentation is maintained by the pipeline. AI-generated test cases run in the pipeline. AI-assisted code review augments the pull request workflow. The DevOps pipeline provides the infrastructure that makes AI-assisted development practical at scale.

Cloud integration means the CI/CD pipeline will increasingly deploy to both z/OS and cloud platforms. A single pipeline might compile COBOL for z/OS, build a Java API gateway for Kubernetes, and deploy both as a coordinated release. The mainframe is one node in a multi-platform deployment topology.

Platform engineering is the practice of building internal developer platforms that abstract away infrastructure complexity. For mainframe teams, this means building self-service environments where developers can provision test regions, run pipelines, and deploy to dev — all without filing tickets to the systems programming team.

GitOps extends Git's role from source control to infrastructure management. The desired state of the z/OS environment (CICS configuration, DB2 schemas, VSAM definitions) is declared in Git, and automation reconciles the actual state with the declared state. This is the logical endpoint of the DevOps journey for the mainframe.

These trends are not distant futures — they're actively being implemented at organizations across the financial services, insurance, and government sectors. The organizations that invest in mainframe DevOps now are building the foundation for these next-generation practices.

Applying DevOps to the HA Banking System

For our progressive project, implementing DevOps means:

1. Git repository: Migrate all HA banking system source to Git with the directory structure from Section 36.2
2. Jenkinsfile: Implement the CI/CD pipeline from Section 36.3, customized for the HA banking system's specific build and test requirements
3. Automated tests: Integrate the test suites from Section 36.4 (unit, integration, regression) into the pipeline
4. Deployment automation: Implement the promotion path from Section 36.5 for the HA banking system's libraries
5. Monitoring integration: Connect the pipeline to the monitoring infrastructure from Chapter 27

These tasks are detailed in the project checkpoint (code/project-checkpoint.md).

Chapter Summary

DevOps for the mainframe is not a future aspiration — it is a present necessity. The tools exist (Git, Zowe, Jenkins, IBM DBB, zUnit), the patterns are proven (CI/CD, automated testing, deployment automation), and the benefits are measurable (faster delivery, fewer defects, better traceability).

The threshold concept — "the mainframe is just another platform in the pipeline" — is the key to unlocking these benefits. Once you stop treating z/OS as a special case that requires special processes, you can leverage the same DevOps practices that have transformed software delivery across every other platform.

The real challenge is cultural, not technical. Veteran mainframe developers have legitimate concerns about new tools and processes, and those concerns deserve respectful, honest engagement. The most successful mainframe DevOps transformations are led by mainframe veterans who see the value, not by distributed systems developers who see the mainframe as a problem to be fixed.

As Rob Chen puts it: "I didn't adopt DevOps because someone told me the mainframe was doing it wrong. I adopted DevOps because I saw that we could do it better. There's a difference."

The key principles to carry forward:

1. Start with source control. Git is the foundation for everything else. Without Git, you can't have CI/CD. Without CI/CD, you can't have automated testing. Without automated testing, you can't have confidence in frequent deployment. Git first, always.

2. Automate the build before anything else. A developer who can push code and see it compile automatically — within minutes, without submitting JCL — has crossed the threshold. That first automated build is the moment DevOps becomes real for the team.

3. Tests are the investment that pays compound returns. Every test you write catches bugs forever. The test you write today catches bugs in the code change you make tomorrow, and the change next month, and the change next year. A test suite of 3,000 tests running nightly is like having 3,000 developers reviewing every change.

4. Deploy small and often. Large quarterly releases are high-risk because they contain many changes, making it hard to identify which change caused a problem. Small, frequent deployments are low-risk because each contains few changes. If something breaks, you know what caused it.

5. Measure everything, share everything. Pipeline metrics, test results, deployment success rates — make them visible to the entire team. Transparency drives improvement. When everyone can see that the last three deployments were green and the test pass rate is 99.2%, they trust the process. When they see a trend heading the wrong direction, they intervene before it becomes a crisis.

6. Respect the humans. DevOps is not about replacing people with automation. It's about freeing skilled developers from manual drudgery so they can apply their expertise to work that matters. The 25-year veteran who knows the batch schedule, the business rules, and the operational context — that person is irreplaceable. The 25-year veteran who spends four hours every quarter manually promoting load modules — that person's time is being wasted. DevOps fixes the latter while honoring the former.

The mainframe has another fifty years of life in it. DevOps is how we make those fifty years sustainable — not by replacing the platform or the people, but by giving both the modern tools and practices they deserve.

Next chapter: We continue our modernization journey by examining cloud integration patterns for COBOL, connecting mainframe transactions to cloud-native services and microservices architectures.