Appendix H: Cloud Service Translation Guide

This book teaches concepts and demonstrates them on a laptop (Appendix A). This appendix maps them to the three major clouds.

🧭 This is the fastest-dating page in the book. Product names change, services are renamed and superseded, and the pricing shape moves. Verify the current name and price before relying on any row. What does not date is the left-hand column.

H.1 The Core Mapping

Snowflake and Databricks run on all three and are frequently the actual answer, which is why they are not in a column.

H.2 Object Storage

ADLS Gen2's hierarchical namespace is the real difference: a rename is a metadata operation rather than a copy, so an atomic directory rename works — which removes one of the reasons table formats exist (ch10). It also means directory-level ACLs are possible, which S3 and GCS emulate with prefix policies.

All three now have strong read-after-write consistency, which invalidates a lot of pre-2021 advice about eventual consistency in lakes.

H.3 Warehouses

The pricing model decides what you optimize (ch33 §33.2):

BigQuery on demand     per TiB scanned  -> optimize the SCAN. CPU is free.
Snowflake              per credit-second -> optimize DURATION and SUSPEND.
Redshift provisioned   per node-hour     -> optimize UTILIZATION and scheduling.

Auto-suspend is the single highest-value setting on the list. Kestrel's disabled one cost $5,040 a month.

H.4 Streaming

MSK and Event Hubs speak the Kafka protocol, so Chapter 15's code runs against both with a configuration change. Kinesis and Pub/Sub do not, and porting is real work.

Pub/Sub's model differs most: subscriptions rather than consumer groups, and per-message acknowledgement rather than offsets. It is genuinely easier to operate and genuinely harder to reason about replay.

H.5 Orchestration

Both managed Airflow services have a meaningful cost floor, because the environment runs whether or not a DAG does. Below about ten DAGs, self-hosting or a smaller tool is cheaper, and Chapter 24's threshold applies before either.

Step Functions is the interesting outlier: near-zero idle cost, genuinely serverless, and a poor fit for data pipelines because its state is small and its retry semantics are per-state rather than per-data interval.

H.6 Identity and Secrets

The pattern is the same on all three and the names differ:

                     AWS                GCP                    Azure
workload identity    IAM role for       Workload Identity      Managed Identity
                     service account    Federation
CI without a secret  OIDC -> AssumeRole OIDC -> Workload       OIDC -> federated
                     WithWebIdentity    Identity Federation    credential
secret storage       Secrets Manager    Secret Manager         Key Vault

The important property is the same everywhere: a running workload should have an identity, not a stored key. Chapter 27 §27.8 and Chapter 28 §28.9 — OIDC in CI removes the long-lived credential entirely, and it is configuration rather than code.

H.7 Cost Shapes That Surprise People

Cross-AZ traffic. AWS charges for traffic between availability zones inside a region — about $0.01/GB each way. A Kafka cluster spanning three AZs pays this on every replica, which is why an MSK bill is larger than the broker-hours suggest.

NAT gateway. ~$0.045/hour plus ~$0.045/GB processed. A private-subnet job pulling from the internet pays twice, and this line surprises people more than any other on an AWS bill.

BigQuery on-demand versus slots. On demand is $6.25/TiB scanned with no floor; slots are a committed capacity with no per-query charge. The crossover is roughly 200 TiB scanned a month, and it is worth computing rather than assuming.

Snowflake's minimum billing increment. 60 seconds per warehouse resume, then per second. A warehouse resumed 400 times a day for 5-second queries bills 400 minutes.

Egress. Every cloud charges to leave and none charges to enter. This is the lock-in that is real, and it is worth knowing your total data volume as a one-off egress cost before it matters.

H.8 Porting This Book's Code

Almost everything is one configuration change.

# S3 -> the sandbox: only the endpoint differs
s3 = boto3.client("s3",
                  endpoint_url="http://localhost:9000",       # remove for real S3
                  aws_access_key_id=os.environ["MINIO_ROOT_USER"],
                  aws_secret_access_key=os.environ["MINIO_ROOT_PASSWORD"])

# GCS
from google.cloud import storage; client = storage.Client()   # ADC handles auth

# ADLS
from azure.storage.filedatalake import DataLakeServiceClient

dbt is an adapter swap: dbt-duckdb → dbt-snowflake, dbt-bigquery, dbt-redshift, dbt-databricks. The models are unchanged except for the dialect differences in Appendix B §B.9.

Spark reads s3a://, gs://, and abfss:// with the right jars on the classpath and the right credentials provider. The DataFrame code does not change.

Kafka code runs unmodified against MSK and Event Hubs. Kinesis and Pub/Sub need a rewrite of the producer and consumer, and nothing else.

H.9 Choosing, Briefly

Most organizations do not choose a cloud on data-platform merits — the decision is made elsewhere, and that is usually fine, because the concepts transfer.

Where it does matter:

BigQuery is genuinely differentiated — serverless, no cluster to size, and a pricing model that makes scan reduction the only optimization. The best default for a team without a data platform engineer.

S3's ecosystem is the widest. Every tool supports it first.

Azure is the right answer when the organization already runs on Entra ID and Fabric, and fighting that is a losing argument regardless of technical merit.

And Snowflake or Databricks on any of them is the answer for many organizations, because it moves the choice from three clouds to one platform — at the cost of a second vendor and a second bill.