Case Study 1: Continental National Bank's PCI-DSS Compliance Program

Background

Continental National Bank processes 500 million card transactions per day across its four-LPAR Parallel Sysplex. Every one of those transactions contains cardholder data — Primary Account Numbers (PANs), expiration dates, and in some cases Card Verification Values (CVVs). Under PCI-DSS, this makes CNB a Level 1 Service Provider, subject to the strictest compliance requirements: an annual on-site assessment by a Qualified Security Assessor (QSA), quarterly network vulnerability scans, and continuous compliance monitoring.

In Q3 2022, CNB underwent its annual PCI-DSS v4.0 assessment. The QSA firm — a top-five assessor with extensive mainframe experience — spent three weeks on-site. This case study documents what they found, what passed, what failed, and how Kwame Mensah's team remediated the failures.

The Cardholder Data Environment

The first assessment task was defining the Cardholder Data Environment (CDE) — the set of systems, networks, and processes that store, process, or transmit cardholder data.

CNB's CDE scope:

Scope reduction strategy: CNB's most significant scope reduction decision was tokenizing PANs in the development and test environments. By replacing real PANs with tokens that are not reversible without access to the production token vault, they removed all non-production systems from PCI scope. This reduced the assessment footprint by approximately 40%.

Rob Calloway was initially skeptical: "The test team screamed bloody murder — they said they needed real PANs for end-to-end testing." Kwame's response: "No. They need data that behaves like a PAN. The token vault generates tokens with valid Luhn check digits and realistic BIN ranges. The test programs can't tell the difference. And I don't have to explain to the QSA why a developer's ISPF session has access to 10 million live PANs."

Assessment Findings

Passed: Requirements 1, 2, 5, 9 (Network, Configuration, Malware, Physical)

The QSA validated CNB's network segmentation, system hardening, malware resistance, and physical security controls without significant findings.

Network segmentation (Requirement 1): CNB's z/OS Communications Server IP filtering rules restrict inbound traffic to the CDE LPARs to authorized IP ranges only. The VLAN architecture segments the CDE from the corporate network. AT-TLS enforces TLS 1.3 on all inbound connections to CICS web service ports.

The QSA noted: "The z/OS IP filtering configuration is more granular than what we typically see in the distributed environment. The ability to restrict TCP/IP stack access by address space (SERVAUTH class) is a control that has no equivalent on Linux."

Malware protection (Requirement 5): The QSA accepted CNB's argument that z/OS is inherently resistant to commodity malware, supplemented by APF authorization controls (preventing unauthorized program execution) and RACF PROGRAM class profiles (controlling which load libraries are authorized). The assessor's comment: "This is the only platform where we accept 'the architecture is the anti-malware control' as a valid response."

Passed with Observations: Requirements 3, 4, 6, 8 (Encryption, Secure Dev, Authentication)

Data at rest (Requirement 3): CNB passed, but the QSA made two observations:

Observation 1: The PAN column in the CNB.CARDHOLDER table used DB2 column-level encryption with AES-128. The QSA recommended upgrading to AES-256 to align with current NIST guidance. Kwame's team completed this within 60 days by rotating the encryption key to a 256-bit key label.

Observation 2: Backup tapes containing cardholder data were encrypted at the dataset level (z/OS tape encryption), but the key management documentation did not explicitly state the key rotation schedule for tape encryption keys. Ahmad (on loan from Pinnacle as a cross-industry advisor for this audit) drafted the documentation within two weeks.

Secure development (Requirement 6): CNB's code review process checked for SQL injection and buffer overflows, but the QSA wanted evidence that the checks were systematic, not ad-hoc. Lisa Tran's automated scan (which flags STRING statements feeding into EXEC SQL PREPARE) was accepted as evidence, but only after she demonstrated it on a sample of recently changed programs and provided the scan output as evidence.

Failed: Requirements 7, 10, 11 (Access Control, Logging, Testing)

Three findings required formal remediation.

Finding 1: Requirement 7 — Excessive batch job access (CRITICAL)

The QSA discovered that the batch settlement program (CNBSET01) ran under userid CNBBAT01, which had UPDATE access to all production DB2 tables — not just the settlement-related tables. This violated the least-privilege principle.

Root cause: When the batch userid was originally configured (in 2008), the DBA team granted broad DB2 access to simplify initial deployment. The original intent was to tighten the permissions after go-live. They never did.

Kwame's reaction: "I knew this was there. Every architect has a skeleton in the closet. Ours was a 14-year-old DB2 GRANT that nobody wanted to touch because 'it works and we might break something.'"

Remediation:

-- Step 1: Identify what CNBBAT01 actually accesses
-- (Lisa ran a DB2 audit trace for 30 days to capture
-- every table and column actually referenced)

-- Step 2: Revoke broad access
REVOKE ALL ON TABLE CNB.CARDHOLDER FROM CNBBAT01_AUTHID;
REVOKE ALL ON TABLE CNB.ACCOUNTS FROM CNBBAT01_AUTHID;
-- (repeated for 47 tables)

-- Step 3: Grant minimum necessary access
GRANT SELECT, UPDATE ON TABLE CNB.SETTLEMENT_BATCH
  TO CNBBAT01_AUTHID;
GRANT SELECT ON TABLE CNB.ACCOUNTS
  TO CNBBAT01_AUTHID;
GRANT INSERT ON TABLE CNB.SETTLEMENT_LOG
  TO CNBBAT01_AUTHID;
-- (8 tables total, down from 47)

Timeline: 90-day remediation. The first 30 days were spent running the DB2 audit trace to identify actual table usage. The next 30 days were spent testing the revised permissions in the pre-production environment. The final 30 days were the controlled production rollout, monitored daily for -551 SQLCODEs.

Finding 2: Requirement 10 — Incomplete audit logging (HIGH)

The QSA found that RACF was configured to audit failures only (AUDIT(FAILURES(READ))) for datasets containing cardholder data. PCI-DSS Requirement 10.2 requires logging all access to cardholder data — both successful and failed.

Root cause: The original RACF configuration was designed for security monitoring (detecting unauthorized access attempts), not for compliance auditing (documenting all access to cardholder data). The operations team had resisted AUDIT(ALL(READ)) because of the SMF volume it would generate.

Remediation:

-- Change audit settings for cardholder data profiles
ALTDSD 'CNB.PROD.CHD.**' AUDIT(ALL(READ))
ALTDSD 'CNB.PROD.CARD.**' AUDIT(ALL(READ))

-- Increase SMF buffer allocation to handle increased volume
-- (in SMFPRMxx member)
BUFUSAGE(80)
SYS(NOTYPE(0:19,21:39,41:69,81:89,91:255),
    TYPE(20,40,70:80,90,100:120))

-- Deploy additional SMF dump datasets
-- (Rob's JCL change for the SMF dump/clear schedule)

Impact: SMF volume increased by approximately 200 GB per day across the Sysplex. Rob Calloway had to adjust the SMF dump schedule from every 8 hours to every 4 hours and provision additional DASD for SMF storage. "Two hundred gig a day," Rob said. "That's the cost of 'log everything.' But the auditor is right — you can't prove compliance with an incomplete audit trail."

Finding 3: Requirement 11 — No regular penetration testing of z/OS services (MEDIUM)

CNB performed annual penetration testing of its distributed web applications but had never penetration-tested the CICS web services or z/OS Connect APIs running on the mainframe.

Root cause: "Nobody does mainframe pentesting" was the conventional wisdom. The QSA disagreed: "If the mainframe exposes HTTP endpoints — and yours does, through z/OS Connect and CICS web services — those endpoints are in scope for penetration testing."

Remediation: CNB engaged a penetration testing firm with mainframe experience (there are approximately five such firms globally). The initial test found no critical vulnerabilities but identified two medium-severity findings: - z/OS Connect API response headers leaked the z/OS release level (information disclosure) - CICS web service error responses included DB2 SQLCODE values in cleartext (information leakage)

Both were remediated by configuring custom error responses in the CICS web service pipeline and the z/OS Connect server configuration.

Lessons Learned

Kwame presented the audit lessons to CNB's technology leadership in a 30-minute briefing. His three key points:

1. Security debt compounds like financial debt. The 14-year-old overly broad DB2 GRANT was a known issue that nobody wanted to fix. Every year it wasn't fixed, it became harder to fix — more programs depended on the broad access, more institutional memory about the original intent was lost, more risk accumulated. Fix security debt when you find it, not when the auditor finds it.

2. Compliance and security are different things. CNB's mainframe was secure — no breaches, no unauthorized access. But it was not compliant — the audit logging was incomplete, the access controls were overly broad, and the testing was inadequate. Security is the absence of compromise. Compliance is the ability to prove security to an external auditor. You need both.

3. The audit is not an event — it's a lifestyle. After the remediation, Kwame established a quarterly self-assessment program. Every 90 days, the security team runs the same checks the QSA runs: access control reviews, audit log completeness, encryption key rotation status, and penetration test scheduling. "By the time the QSA arrives next year," Kwame told his team, "we should already know every finding they'll make — because we found it first."

Discussion Questions

1. CNB's batch settlement program had UPDATE access to 47 production tables for 14 years. This is a common pattern in mainframe shops — broad access granted during initial deployment that is never tightened. What organizational and technical barriers prevent least-privilege tightening? How would you overcome them?

2. The QSA accepted z/OS's architecture as sufficient for PCI-DSS Requirement 5 (malware protection). Do you agree with this assessment? What insider threat scenarios could circumvent the architectural protections?

3. CNB's scope reduction through PAN tokenization removed 40% of systems from PCI scope. What are the risks of tokenization? Under what circumstances could a token be used to reconstruct the original PAN?

4. The SMF volume increase of 200 GB/day for comprehensive audit logging is a real operational cost. How would you architect the SMF collection and analysis pipeline to handle this volume without impacting production system performance?

5. Only approximately five firms globally offer mainframe penetration testing. What does this say about the mainframe security market? Is the scarcity of testers a risk (fewer eyes on the platform) or a strength (smaller attack surface)?