Case Study: Training GPT — The Environmental Bill

Background

In June 2020, OpenAI released GPT-3, a large language model with 175 billion parameters. It was, at the time, the largest language model ever made publicly available. GPT-3 could write essays, answer questions, generate code, translate languages, and produce creative fiction — all from a simple text prompt. It was a commercial and cultural sensation, and it set the stage for the generative AI boom that followed.

What received less attention was the environmental cost of creating GPT-3.

Training a model with 175 billion parameters is not a weekend project. It requires thousands of specialized GPUs running continuously for weeks. It requires enormous amounts of electricity to power those GPUs. It requires cooling systems — typically using water — to keep the hardware from overheating. And it requires manufacturing the hardware in the first place, a process with its own significant environmental footprint.

This case study examines what we know (and what we can reasonably estimate) about the environmental bill for training large language models.

The Energy Numbers

OpenAI has not publicly disclosed the exact energy consumption or carbon emissions from training GPT-3. However, multiple independent researchers have produced estimates based on available information about the model's architecture, training duration, and hardware.

Training computation: GPT-3 required an estimated 3,640 petaflop-days of computation — a measure of the total number of floating-point operations performed during training. To put this in context: a modern high-end GPU can perform roughly 300 teraflops per second. Training GPT-3 would require approximately 1,000 such GPUs running continuously for about 14 days — or fewer GPUs running for longer.

Energy consumption: Based on the estimated hardware and training duration, researchers have estimated that training GPT-3 consumed approximately 1,287 megawatt-hours (MWh) of electricity. For comparison: - An average American household uses about 10.5 MWh per year - So GPT-3's training consumed roughly the annual electricity of 120 homes

Carbon emissions: The carbon impact depends critically on the energy source. If the training ran in a data center powered by the U.S. average grid mix (roughly 400g CO2/kWh in 2020), the estimated emissions would be approximately 502 metric tons of CO2. If the data center ran on cleaner energy (as Microsoft's Azure facilities increasingly do), the number would be lower. If it ran on coal-heavy electricity, it would be higher.

For reference, 502 metric tons of CO2 is roughly equivalent to: - The annual emissions of about 100 passenger cars - Approximately 1,220 one-way flights from New York to San Francisco - The lifetime emissions of about one American, from birth to death

Scaling: From GPT-3 to GPT-4 and Beyond

GPT-3 was large. Its successors are much larger.

While OpenAI has not disclosed exact specifications for GPT-4 (released March 2023), independent analysts and leaked reports suggest it is significantly larger than GPT-3 — potentially involving multiple models working together in a "mixture of experts" architecture. Training costs for GPT-4 have been estimated at $100 million or more, suggesting computational requirements roughly 10–100 times greater than GPT-3.

If those estimates are accurate, training GPT-4 may have consumed energy equivalent to thousands of homes for a year and produced carbon emissions on the order of thousands of metric tons of CO2 — though the exact numbers depend heavily on the energy source and hardware efficiency.

The trend is clear: each generation of frontier models is dramatically more expensive to train than the last. And this is just one company. Google, Meta, Anthropic, Mistral, and other organizations are training their own large models, each with comparable or greater computational requirements.

The Inference Side

Training is a one-time cost. But GPT models are used for inference millions of times per day.

A single GPT-4 query is estimated to consume roughly 0.001–0.01 kWh of electricity — a tiny amount. But multiplied by hundreds of millions of daily queries across OpenAI's ChatGPT service and API customers, the annual inference energy consumption likely exceeds the one-time training cost many times over.

This pattern — training as a large one-time expenditure and inference as a small per-use cost that dominates at scale — is characteristic of large AI systems and is a critical factor in assessing their total environmental impact.

The Water Bill

Researchers at UC Riverside estimated that training GPT-3 consumed approximately 700,000 liters (185,000 gallons) of fresh water for data center cooling — enough to fill roughly 17 standard residential swimming pools. This does not include the water used to generate the electricity consumed during training.

For GPT-4, with its estimated 10–100x greater computational requirements, the water consumption for training alone could be in the millions of liters.

As AI inference scales to billions of daily queries, the ongoing water consumption of AI data centers is becoming a significant concern in water-stressed regions. Microsoft, which hosts OpenAI's infrastructure, reported a 34% increase in global water consumption from 2021 to 2022, attributing the increase in part to AI growth.

The Other Side of the Ledger

Any fair assessment must also consider what GPT-3 and its successors have enabled. Large language models have been applied to:

  • Scientific research: Assisting with literature review, hypothesis generation, data analysis, and code writing, potentially accelerating research in fields including climate science, materials science, and drug discovery.
  • Education: Providing personalized tutoring, explaining complex concepts, and making educational content more accessible.
  • Accessibility: Generating alt text for images, translating between languages, simplifying complex documents, and other tasks that improve access for people with disabilities.
  • Productivity: Automating routine writing tasks, summarizing documents, assisting with code development, and other efficiency gains across the economy.

Quantifying these benefits in environmental terms is difficult. If LLM-assisted research accelerates the development of better solar panels, batteries, or carbon capture technologies, the indirect environmental benefit could be enormous. But such claims are speculative and hard to verify.

Analysis Framework

1. Cost-Benefit Assessment

  • What are the direct environmental costs of training and deploying GPT-scale models? (Consider energy, carbon, water, hardware, e-waste.)
  • What are the direct and indirect environmental benefits? (Consider research acceleration, efficiency gains, climate applications.)
  • Can the costs and benefits be meaningfully compared? What information would you need to make a rigorous comparison?

2. Distributional Questions

  • Who benefits from GPT-scale models? (Primarily users in wealthy countries and companies in the technology sector.)
  • Who bears the environmental costs? (Data center host communities, mineral-mining communities, e-waste-processing communities, and everyone affected by climate change.)
  • Is this distribution equitable? What would a more equitable distribution look like?

3. Alternatives Analysis

  • Could the same benefits be achieved with a smaller, more efficient model? At what point does model scale deliver diminishing returns?
  • What role do green AI techniques (distillation, pruning, quantization) play in reducing costs?
  • What role does energy sourcing play? How much would the environmental assessment change if training ran on 100% renewable energy?

4. Governance Questions

  • Should there be mandatory environmental disclosure for large training runs?
  • Should there be limits on the scale of AI training — either through regulation or carbon pricing?
  • Who should make these decisions — companies, governments, international bodies, or some combination?

Discussion Questions

  1. The proportionality question: Training GPT-3 produced roughly 500 tons of CO2. The global airline industry produces roughly 900 million tons of CO2 per year. Does the comparison make GPT-3's impact seem trivial — or does it miss the point? What is the right frame of reference for evaluating AI's environmental impact?

  2. The disclosure question: OpenAI did not publicly disclose the energy consumption or carbon emissions from training GPT-3 or GPT-4. Should such disclosure be mandatory for models above a certain size? What are the arguments for and against?

  3. The trajectory question: If each generation of frontier models requires 10–100 times more computation than the last, is this trajectory sustainable — even with efficiency improvements and renewable energy? Where does it end?

  4. The equity question: Large language models are trained primarily by companies in the United States and a few other wealthy nations. The environmental costs are borne globally. Is this a form of environmental injustice? How might it be addressed?

  5. The personal question: You have almost certainly used a large language model. Knowing what you now know about the environmental cost, does it change how you think about your usage? Should it?

Mini-Project Options

Option A: Carbon estimation. Using the Python estimator from Section 18.7 or a tool like the ML CO2 Impact Calculator, estimate the carbon emissions for training a model at three different scales: small (100 GPU-hours), medium (10,000 GPU-hours), and large (100,000 GPU-hours). For each, calculate the emissions using three different grid carbon intensities (clean: 50 g/kWh, average: 400 g/kWh, coal-heavy: 900 g/kWh). Present your results in a table and write a paragraph analyzing the key variables.

Option B: Corporate comparison. Research the environmental disclosures of two companies that train large AI models (e.g., Google, Microsoft, Meta, OpenAI, Anthropic). Compare: what do they disclose? What do they not disclose? How do their commitments compare? Write a 500-word analysis.

Option C: Policy proposal. Draft a one-page policy proposal for mandatory environmental impact disclosure for large AI training runs. Specify: what must be disclosed, by whom, to whom, in what format, and what the consequences of non-compliance would be.

References and Sources

  • Strubell, E., Ganesh, A., & McCallum, A. (2019). "Energy and Policy Considerations for Deep Learning in NLP." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–3650. [Tier 1 — The seminal study on AI energy consumption]
  • Patterson, D., Gonzalez, J., Le, Q., et al. (2021). "Carbon Emissions and Large Neural Network Training." arXiv preprint arXiv:2104.10350. [Tier 1 — Detailed analysis by Google researchers]
  • Li, P., Yang, J., Islam, M. A., & Ren, S. (2023). "Making AI Less Thirsty: Uncovering and Addressing the Secret Water Footprint of AI Models." arXiv preprint arXiv:2304.03271. [Tier 1 — UC Riverside water consumption study]
  • International Energy Agency. (2024). "Electricity 2024: Analysis and Forecast to 2026." [Tier 1 — Global energy consumption data and projections]
  • Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2019). "Green AI." arXiv preprint arXiv:1907.10597. [Tier 1 — Foundational paper on computational efficiency in AI]