Key Takeaways — Chapter 38: Batch Processing Patterns

Core Concepts

  1. Checkpoint/restart is not optional in enterprise batch. Any program processing significant volumes must be able to resume from a failure point without reprocessing or double-processing records.

  2. Control totals catch errors that logic cannot detect. Record counts, hash totals, and financial totals provide independent verification that every record is accounted for. The control total equation — Records Read = Records Written + Records Rejected + Records Skipped — must always balance.

  3. Hash totals verify completeness, not accuracy. A hash total is a sum of a non-financial field (such as account numbers). Two systems computing the same hash total over the same records confirms they processed the same set of records.

  4. Audit trails record what changed, when, and why. Field-level before/after images provide the most useful audit information. Both the individual changes and the batch run itself should be audited.

  5. The balanced update pattern is the foundation of master file processing. Two sorted files — master and transaction — are processed together by comparing keys. Handle EOF with HIGH-VALUES, and always account for multiple transactions per master key.

  6. Multi-step job streams require inter-step verification. Every step must verify that its input matches the prior step's output control totals. A successful return code from a prior step does not guarantee correct output.

  7. GDGs (Generation Data Groups) provide automatic dataset versioning. Relative references (+1, 0, -1) make JCL generic. The GDG limit controls automatic rolloff of old generations.

  8. Return codes communicate program status to the scheduler. Use 0 for success, 4 for warnings, 8 for errors, 12 for severe errors, and 16 for catastrophic failures. Always display the return code before exiting.

  9. Never use floating-point for financial totals. COMP-3 (packed decimal) preserves exact decimal values. COMP-1 and COMP-2 introduce rounding errors that accumulate across millions of records.

  10. Error thresholds prevent runaway processing. Set configurable limits on error counts and error percentages. Abort the program when thresholds are exceeded rather than processing garbage data.

Patterns to Remember

Pattern Purpose Key Technique
Checkpoint/restart Resume after failure Periodic state snapshots to checkpoint file
Control totals Verify completeness Record counts + hash totals + financial totals
Audit trail Track changes Before/after images for each field change
Balanced update Master file processing Sorted key comparison with HIGH-VALUES EOF
GDG Dataset versioning Relative references in JCL (+1, 0, -1)
Error threshold Prevent bad data Configurable count/percentage limits
Run control Track step completion Status table for each step + business date

Common Mistakes to Avoid

  • Ignoring checkpoint write failures (if you cannot checkpoint, you cannot restart)
  • Writing the master record before processing all transactions for that key
  • Using WRITE instead of REWRITE for subsequent checkpoints
  • Forgetting to accumulate control totals for rejected records
  • Using COMP-1/COMP-2 for financial calculations
  • Not verifying input control totals at the start of each step
  • Hardcoding dataset names instead of using GDG relative references
  • Setting return code 0 when warnings or errors occurred