Case Study 2: The Lamb Shift — Where Quantum Mechanics Meets Quantum Field Theory

Overview

The Lamb shift — the splitting between the $2S_{1/2}$ and $2P_{1/2}$ levels of hydrogen, which the Dirac equation predicts to be exactly degenerate — is one of the pivotal measurements in the history of physics. Its discovery in 1947 demonstrated that the Dirac equation, for all its triumphs, was incomplete. Its explanation required the quantization of the electromagnetic field itself and launched the development of quantum electrodynamics (QED), the most precisely tested theory in the history of science.

This case study traces the experiment, the theoretical crisis it provoked, the resolution through QED, and the extraordinary precision measurements that continue to test fundamental physics today.

Part 1: The Dirac Prediction and Its Assumption

What Dirac's Equation Says

The exact Dirac energy levels for hydrogen depend on the principal quantum number $n$ and the total angular momentum quantum number $j$:

$$E_{n,j} = mc^2\left[1 + \left(\frac{\alpha}{n - j - \frac{1}{2} + \sqrt{(j + \frac{1}{2})^2 - \alpha^2}}\right)^2\right]^{-1/2}$$

Crucially, the energy does not depend on the orbital angular momentum quantum number $l$. This means that states with the same $n$ and $j$ but different $l$ are degenerate. For $n = 2$:

• $2S_{1/2}$ (with $l = 0$, $j = 1/2$)
• $2P_{1/2}$ (with $l = 1$, $j = 1/2$)

have exactly the same energy according to Dirac. The $2P_{3/2}$ level ($l = 1$, $j = 3/2$) is higher, separated from the $2S_{1/2}$/$2P_{1/2}$ pair by the fine-structure interval of about $10,969$ MHz.

The Hidden Assumption

The Dirac equation treats the electromagnetic field as a classical background — the Coulomb potential $V(r) = -e^2/r$ is a fixed function, not a quantum operator. This approximation is called the external field approximation, and for most purposes it is excellent. But it neglects three effects:

1. The electromagnetic field itself has quantum fluctuations (zero-point energy), even in the vacuum.
2. The electron can emit and reabsorb virtual photons, temporarily modifying its own properties.
3. The vacuum can spontaneously produce virtual electron-positron pairs that partially screen the proton's charge.

These effects are tiny — of order $\alpha^5mc^2 \sim 10^{-6}$ eV, compared to the fine structure at $\alpha^4mc^2 \sim 10^{-4}$ eV — but they are not zero. Detecting them required a new level of experimental precision.

Part 2: The Experiment (1947)

The Experimenters

Willis Eugene Lamb Jr. was a 34-year-old physicist at Columbia University who had spent the war years working on microwave radar at the university's Radiation Laboratory. Robert Curtis Retherford was his graduate student. The wartime development of high-frequency microwave technology — magnetrons, klystrons, waveguides — had given experimentalists tools of unprecedented precision for manipulating and detecting electromagnetic radiation in the GHz range.

The Method

Previous attempts to measure the $2S_{1/2}$-$2P_{1/2}$ splitting had used optical spectroscopy, looking at the Balmer-$\alpha$ line (the $n = 2 \to n = 3$ transition) for evidence of fine-structure anomalies. These measurements were limited by the Doppler broadening of the spectral lines (the thermal motion of hydrogen atoms smears out the line by about $10,000$ MHz, comparable to the fine structure itself). It was impossible to resolve the $\sim 1000$ MHz Lamb shift within a $\sim 10,000$ MHz Doppler-broadened line.

Lamb and Retherford's breakthrough was to avoid optical spectroscopy entirely. Instead of observing light emitted by hydrogen atoms, they used microwave spectroscopy to directly drive transitions between the $2S_{1/2}$ and $2P_{1/2}$ states.

Their method exploited a crucial property of the $2S_{1/2}$ state: it is metastable. An atom in the $2S_{1/2}$ state cannot decay to the ground state ($1S_{1/2}$) by emitting a single photon, because this would require a transition with $\Delta l = 0$, violating the electric dipole selection rule $\Delta l = \pm 1$. The $2S_{1/2}$ state decays instead by a two-photon process, with a lifetime of about $1/7$ second — an eternity in atomic physics. By contrast, the $2P_{1/2}$ state decays to the ground state in about $1.6 \times 10^{-9}$ seconds.

The Experimental Setup

1. A beam of hydrogen atoms is produced and bombarded by electrons with energy just above the $n = 2$ excitation threshold ($\sim 10.2$ eV). This populates both the $2S_{1/2}$ and $2P$ states, but the $2P$ states decay almost instantly, leaving a beam of metastable $2S_{1/2}$ atoms.

2. The metastable atoms travel through a region of microwave radiation. If the microwave frequency matches the $2S_{1/2} \to 2P_{1/2}$ transition frequency, the atoms are stimulated to transition to $2P_{1/2}$, from which they immediately decay to the ground state.

3. The metastable $2S_{1/2}$ atoms that survive the microwave region strike a tungsten detector. When a $2S_{1/2}$ atom hits tungsten, it ejects an electron (the excitation energy of the metastable state is enough to overcome the work function). An atom in the ground state does not eject an electron. The detector current therefore measures the number of metastable atoms reaching the detector.

4. When the microwave frequency is resonant with the $2S_{1/2} \to 2P_{1/2}$ transition, the detector current decreases because metastable atoms are being depopulated.

The Result

Lamb and Retherford found a clear resonance at a frequency of:

$$\nu_{\text{Lamb}} = 1057 \pm 3\,\text{MHz}$$

The $2S_{1/2}$ state was higher in energy than the $2P_{1/2}$ state by this amount. The Dirac equation predicted zero. The measurement was unambiguous.

Part 3: The Theoretical Crisis and Resolution

The Shelter Island Conference (June 1947)

The news of Lamb's result (initially $\sim 1000$ MHz, refined to $1057$ MHz shortly after) was presented at the historic Shelter Island conference on June 2-4, 1947. The conference, organized by Robert Oppenheimer and attended by Feynman, Schwinger, Bethe, Kramers, von Neumann, and other luminaries, was electrified by Lamb's announcement.

The theoretical challenge was clear: explain the Lamb shift using quantum electrodynamics, the theory of quantized electromagnetic fields interacting with quantized matter. QED had been developed in the 1930s by Dirac, Heisenberg, Pauli, and others, but it was plagued by infinities — integrals that diverged to infinity in higher-order calculations. Many physicists had concluded that QED was fundamentally flawed.

Bethe's Calculation (June 1947)

Hans Bethe, riding the train from Shelter Island back to Schenectady, New York, performed the first successful calculation of the Lamb shift. His approach was non-relativistic but captured the essential physics.

The dominant contribution comes from the electron self-energy: the electron emits a virtual photon and reabsorbs it, and during the brief interval when the virtual photon exists, the electron's effective position is shifted (or "smeared out") by a small amount. The shift is:

$$\langle(\delta r)^2\rangle \sim \frac{2\alpha}{3\pi}\left(\frac{\hbar}{mc}\right)^2\ln\left(\frac{mc^2}{\langle E_{\text{binding}}\rangle}\right)$$

This smearing affects $S$-states (which have $|\psi(0)|^2 \neq 0$) more than $P$-states (which have $|\psi(0)|^2 = 0$), because only $S$-state electrons spend time at $r = 0$ where the Coulomb potential is strongest. The resulting energy shift is:

$$\Delta E_{nS} \approx \frac{4\alpha^5mc^2}{3\pi n^3}\ln\left(\frac{1}{\alpha^2}\right)$$

Bethe's estimate: $\Delta E_{2S} \approx 1040$ MHz — within 2% of the experimental value!

The integral in Bethe's calculation was formally divergent (the virtual photon's energy can be arbitrarily large). Bethe handled this by renormalization: the divergent part of the self-energy shift is the same for a free electron and a bound electron, so it can be absorbed into the observed electron mass. Only the difference between the bound and free self-energies — which is finite — contributes to the Lamb shift.

The Full QED Calculation

The complete relativistic QED calculation was carried out by several groups over the following two years:

1. Electron self-energy (dominant): $+1017$ MHz. The electron's interaction with virtual photons smears its charge over a distance $\sim \alpha\lambda_C/\pi$, effectively weakening the Coulomb potential seen by $S$-state electrons.

2. Vacuum polarization: $-27$ MHz. Virtual $e^+e^-$ pairs near the proton partially screen its charge at short distances, slightly strengthening the effective potential at larger distances. This effect opposes the self-energy and is smaller.

3. Vertex correction (anomalous magnetic moment): $+68$ MHz. The electron's modified interaction with the magnetic field of its own orbital motion.

4. Higher-order corrections: smaller contributions from two-loop and higher diagrams.

Total theoretical prediction: $1057.864 \pm 0.014$ MHz.

Current experimental value: $1057.845(9)$ MHz.

The agreement is spectacular.

Part 4: The Three Contributions in Detail

Electron Self-Energy: Virtual Photon Cloud

The electron is constantly emitting and reabsorbing virtual photons. This creates a "cloud" of virtual photons surrounding the electron, which modifies its effective interaction with external fields.

In free space, the self-energy is infinite but unobservable — it is absorbed into the electron's measured mass (mass renormalization). In a bound state, however, the self-energy depends on the electron's environment. The difference between the bound and free self-energies is finite and observable.

Physically, the virtual photon cloud causes the electron's position to fluctuate rapidly (Zitterbewegung-like). An electron in an $S$-state, which has substantial probability at $r = 0$, "samples" the rapidly varying Coulomb potential over a range $\delta r$. Since the Coulomb potential curves upward near $r = 0$ (it goes as $-e^2/r$, which is concave when averaged), the average potential seen by the smeared electron is slightly less negative than the point-like Coulomb potential. This raises the $S$-state energy.

For $P$-states, $|\psi(0)|^2 = 0$, so the effect is much smaller (it enters at a higher power of $\alpha$).

Vacuum Polarization: The Seething Vacuum

The QED vacuum is not empty. It seethes with virtual particle-antiparticle pairs that briefly appear and then annihilate. Near a charged particle like the proton, the virtual $e^+e^-$ pairs become polarized: the virtual positrons are attracted toward the proton, and the virtual electrons are repelled. This creates a partial screening of the proton's charge.

At large distances (larger than the Compton wavelength $\hbar/m_ec \approx 386$ fm), the screening is nearly complete, and the electron sees the renormalized charge $e$. At very short distances (smaller than $\hbar/m_ec$), the electron penetrates inside the screening cloud and sees a larger effective charge. This strengthens the Coulomb potential at short range, which lowers the $S$-state energy (opposite sign to the self-energy).

The vacuum polarization contribution to the Lamb shift is $-27$ MHz, partially canceling the self-energy contribution of $+1017$ MHz. The net Lamb shift is positive (the $2S_{1/2}$ state is higher than $2P_{1/2}$) because the self-energy dominates.

Vertex Correction: The Anomalous Magnetic Moment in Action

The same virtual photon exchange that gives rise to the electron self-energy also modifies the electron's magnetic moment from the Dirac value $g_s = 2$ to the Schwinger value $g_s \approx 2(1 + \alpha/2\pi)$. This anomalous magnetic moment changes the spin-orbit interaction, which in turn shifts the energy levels.

The vertex correction contributes $+68$ MHz to the Lamb shift of the $2S_{1/2}$ state.

Part 5: Precision Frontier — The Lamb Shift Today

Modern Measurements

The Lamb shift has been measured with steadily increasing precision over the decades:

The current theoretical value, computed to order $\alpha^7$ in QED with nuclear-size corrections, is:

$$\Delta E_{\text{theory}} = 1057.8446 \pm 0.0020\,\text{MHz}$$

The agreement between theory and experiment is at the level of a few parts per million.

The Proton Radius Puzzle

In 2010, a measurement of the Lamb shift in muonic hydrogen (a proton orbited by a muon instead of an electron) produced a startling result. Because the muon is 207 times heavier than the electron, its Bohr radius is 207 times smaller, and it spends much more time near the proton. This makes the Lamb shift in muonic hydrogen exquisitely sensitive to the proton's charge radius $r_p$.

The muonic hydrogen Lamb shift measured at the Paul Scherrer Institute yielded:

$$r_p = 0.84087 \pm 0.00039\,\text{fm}$$

This was about 4% smaller than the previously accepted value from electronic hydrogen spectroscopy and electron-proton scattering:

$$r_p = 0.8751 \pm 0.0061\,\text{fm}$$

The discrepancy, known as the proton radius puzzle, generated enormous excitement and prompted a decade of re-measurements. By 2022, improved measurements of electronic hydrogen spectroscopy converged toward the smaller muonic value, and the puzzle appears to be largely resolved — the earlier electronic measurements had systematic uncertainties that were underestimated. But the episode illustrates how precision measurements of the Lamb shift continue to push the frontiers of physics.

What the Lamb Shift Tests

The Lamb shift is sensitive to: - QED (virtual photons, vacuum polarization, vertex corrections) - Nuclear structure (the finite size of the proton modifies the Coulomb potential at short range) - Hadronic vacuum polarization (virtual quark-antiquark pairs, in addition to $e^+e^-$) - Recoil corrections (the proton is not infinitely heavy)

Every improvement in precision tests these contributions at deeper levels. The Lamb shift is a laboratory where atomic physics, quantum field theory, and nuclear physics all intersect.

Part 6: The Lamb Shift's Place in Physics History

The Birth of Modern QED

The Lamb shift measurement in 1947 is often cited as the event that launched modern quantum field theory. It demonstrated that:

1. The electromagnetic vacuum has measurable physical effects.
2. The infinities in QED can be systematically handled through renormalization.
3. QED, properly renormalized, makes predictions of extraordinary precision.

The theoretical tools developed to explain the Lamb shift — Feynman diagrams, renormalization, the operator formalism of QED — became the foundation for the Standard Model of particle physics.

The Hierarchy of Hydrogen Physics

The Lamb shift sits in a beautiful hierarchy of effects in the hydrogen atom, each corresponding to a deeper level of physical understanding:

Each level reveals a new layer of reality. The hydrogen atom, the simplest atom in nature, has been the Rosetta Stone for decoding the laws of physics at every scale.

Nobel Prizes from the Lamb Shift

The Lamb shift, directly or indirectly, contributed to several Nobel Prizes:

• Willis Lamb (1955): "for his discoveries concerning the fine structure of the hydrogen spectrum"
• Polykarp Kusch (1955, shared with Lamb): "for his precision determination of the magnetic moment of the electron" (the anomalous magnetic moment, closely related to the Lamb shift)
• Julian Schwinger, Richard Feynman, Sin-Itiro Tomonaga (1965): "for fundamental work in quantum electrodynamics, with deep-ploughing consequences for the physics of elementary particles" — developed to explain the Lamb shift

Discussion Questions

1. Lamb and Retherford's experiment was made possible by wartime radar technology. What does this tell us about the relationship between applied and fundamental research? Can you think of other examples where technology developed for practical purposes enabled fundamental discoveries?

2. Bethe's calculation on the train from Shelter Island gave an answer within 2% of the experimental value, despite being non-relativistic and using a crude cutoff for the divergent integral. Why was this crude estimate so effective? What does this tell us about the physics of the Lamb shift?

3. The proton radius puzzle illustrates how a discrepancy between two measurements at the 4% level can generate enormous scientific activity. Why was this small discrepancy taken so seriously? What would the implications have been if the discrepancy had been real (i.e., if muonic and electronic hydrogen genuinely gave different proton radii)?

4. The Lamb shift is a purely quantum field-theoretic effect with no classical or single-particle analogue. It arises from the interaction of the electron with the fluctuating quantum vacuum. Does this mean the vacuum is "real" — a physical entity with measurable properties — or is it just a convenient way to organize a perturbation expansion?

5. The hierarchy of hydrogen energy corrections (Bohr $\to$ Dirac $\to$ QED $\to$ ...) spans six orders of magnitude in energy scale. Each level requires a deeper theoretical framework. Do you think there are further levels waiting to be discovered (beyond QED), or has the hydrogen atom been completely understood? What would a new, undiscovered level of hydrogen structure tell us about physics?