Chapter 22 Key Takeaways

1. The Mainframe Is the System of Record — Integration Follows from That

Every integration pattern you design must respect the fundamental asymmetry: the mainframe holds the authoritative data. Distributed systems hold copies, caches, and derived views. When there's a conflict, the mainframe wins. This means your integration architecture must prioritize data flowing outward from the mainframe with consistency guarantees, not treat all systems as equal peers.

2. Three Patterns, Not One

File-based, message-based, and API-based integration each have distinct sweet spots. Files handle high-volume batch efficiently. Messages handle near-real-time event distribution with decoupling. APIs handle low-latency, synchronous request-reply. No pattern is universally superior. Most production mainframe environments use all three simultaneously, and the best architects know which pattern fits which requirement without defaulting to their personal favorite.

3. File-Based Integration Is Not Legacy — It's Fit for Purpose

Connect:Direct with GDG-based handoff patterns remains the most reliable, recoverable, and auditable method for moving large volumes of structured data. Checkpoint/restart, compression, and guaranteed delivery make it the right choice for nightly data warehouse loads, regulatory reporting extracts, and bulk data exchanges. Dismissing files as "legacy" is a sign of inexperience with high-volume data movement.

4. GDGs Are Your Integration Safety Net

Generation Data Groups provide automatic versioning, rotation, and recovery capability for integration files. The dual-GDG acknowledgment pattern (data GDG paired with acknowledgment GDG) catches silent feed failures. Always use GDGs for integration files — never bare sequential datasets. The small setup cost pays dividends the first time you need to retransmit or audit a prior generation.

5. Canonical Data Models Reduce Integration Complexity from N-Squared to Linear

With N downstream consumers, point-to-point transformations require up to N(N-1) format converters. A canonical data model reduces this to 2N by defining a single intermediate format. Every producer converts to canonical; every consumer converts from canonical. Adding a new consumer requires one new transformer instead of N-1. The upfront design cost is repaid within two or three integration additions.

6. Message-Based Integration Requires Idempotent Consumers and Poison Message Handling

MQ guarantees delivery, but network failures and application errors can cause message redelivery. Every consumer must handle duplicates gracefully by tracking processed message IDs. Every consumer must detect poison messages (via backout count) and route them to a review queue instead of entering an infinite redelivery loop. These are not optional safety features — they are required for production reliability.

7. APIs Are Powerful but Expensive on the Mainframe

z/OS Connect transforms COBOL programs into REST APIs elegantly, but every API call consumes mainframe CPU (MIPS), which is metered and expensive. Caching, rate limiting, pagination, and versioning are not optional — they're economic necessities. Use APIs for low-volume, real-time interactions. Use files or messages for everything else.

8. Data Format Translation Causes More Integration Bugs Than Anything Else

EBCDIC/ASCII conversion, packed decimal translation, binary endianness, and date format normalization are the most common sources of integration failures. Build a standardized transformation utility (like Federal Benefits' XFORMUTIL) and use it everywhere. Test edge cases exhaustively: zero values, maximum values, negative values, null dates, sentinel dates, and special characters.

9. Change Data Capture Solves the Incremental Data Problem

Full-file extracts are wasteful when only a small percentage of records change daily. CDC captures and transmits only the changes. Log-based CDC (reading database recovery logs) is the most reliable approach but requires product licensing. Application-level CDC (modifying COBOL programs to write change records) is more accessible but requires discipline to ensure every update program participates. Combine CDC with periodic full reconciliation to catch any gaps.

10. Monitor for Absence, Not Just Failure

The most dangerous integration failure is the one that doesn't happen. If a file never arrives, no error occurs. If a message is never sent, no alert fires. Build monitoring that expects activity and alerts on silence — expected files that don't arrive, expected message volumes that drop below baseline, expected confirmations that never come. Proactive absence monitoring catches an entire class of failures that reactive error monitoring misses.

11. The Pattern Selection Framework: Five Questions Before You Design

Before choosing an integration pattern, answer: (1) What is the latency requirement? (2) What is the volume? (3) What happens when the consumer is down? (4) Who initiates the exchange? (5) Does the exchange require a response? These five questions point you toward the right pattern — or the right hybrid — every time.

12. Hybrid Patterns Combine Strengths

The "near-real-time with batch reconciliation" pattern — CDC/MQ for real-time changes plus nightly full-file extract for reconciliation — combines the low latency of messaging with the reliability of file-based integration. Most production architectures are hybrids, and the reconciliation step is what ensures consistency across patterns.

Integration Pattern Quick Reference