Chapter 26 Further Reading: A/B Testing Content and Offer Strategy

Books

1. "Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing" by Ron Kohavi, Diane Tang, and Ya Xu (2020, Cambridge University Press) This is the definitive technical reference on A/B testing, written by researchers from Microsoft, Google, and LinkedIn who have collectively run tens of thousands of experiments. It is thorough, somewhat dense, and not specifically written for creators — but Chapters 1–4 and Chapter 7 (on common pitfalls) are accessible and directly relevant. If you want to understand why the stopping rules and sample size calculations in this chapter are structured the way they are, this book provides the rigorous foundations.

2. "You Should Test That" by Chris Goward (2013, Sybex) Goward's LIFT model (Value Proposition, Relevance, Clarity, Anxiety, Distraction, Urgency) provides a structured framework for identifying what to test on landing pages and conversion-oriented content. Written for digital marketers but highly applicable to creator landing pages, email sequences, and product offers. The case studies throughout the book translate directly to creator contexts.

3. "Influence: The Psychology of Persuasion" by Robert Cialdini (1984, updated 2021, Harper Business) Understanding why certain test variants win requires understanding how people make decisions. Cialdini's principles — reciprocity, commitment, social proof, authority, liking, scarcity — underlie most of what makes Version B outperform Version A. Reading this alongside your test results helps you build better hypotheses (not just empirical "what" findings but theoretical "why" explanations).

Online Resources

4. HubSpot Research Blog — A/B Testing Case Studies (blog.hubspot.com) HubSpot has published hundreds of posts documenting specific test findings across email marketing, landing pages, CTAs, and content. Searchable by topic. Their level of transparency about methodology and results — including negative results and failed tests — is unusual in the industry. Start with their posts on email subject line testing and CTA button optimization for direct creator relevance.

5. CXL Institute Blog — Conversion Rate Optimization (cxl.com/blog) CXL (formerly ConversionXL) publishes rigorous, data-oriented content about testing and conversion optimization. Their articles on statistical significance, sample size calculation, and common testing mistakes are consistently high quality and appropriately skeptical of hype. Free. Their paid courses are excellent if you want deeper statistical training.

6. Optimizely Learning Center (optimizely.com/optimization-glossary) Optimizely's public glossary and learning resources cover every A/B testing concept in this chapter with clear definitions and examples. Particularly useful for: understanding the difference between frequentist and Bayesian testing approaches (a topic this chapter touches on implicitly), p-value interpretation, and multi-armed bandit testing as an alternative to classic A/B.

Academic and Research Resources

7. "Online Experimentation at Microsoft" — Kohavi et al. (2009, Proceedings of the 15th ACM SIGKDD) Available free via Google Scholar. This early academic paper documents the lessons from running thousands of controlled experiments at Microsoft, including the pitfalls most commonly observed in practice. The section on "trustworthiness of metrics" (are you measuring what actually matters?) is particularly relevant for creators choosing their optimization targets.

8. "Peeking at A/B Tests: Why It Matters and What to Do About It" — Johari et al. (2015, LinkedIn Engineering Blog) Available free at engineering.linkedin.com. LinkedIn's data scientists documented the specific mathematical consequences of peeking at A/B tests, including how much it inflates false positive rates. This is the technical foundation for the stopping rules discussed in Section 26.6.

Tools

9. Evan Miller's Sample Size Calculator (evanmiller.org/ab-testing/sample-size.html) A free, clean sample size calculator for A/B tests that accepts baseline rate, minimum detectable effect, significance level, and statistical power. Faster to use than running the Python script for quick back-of-envelope calculations. Evan Miller's site also includes excellent explainers on A/B testing statistics.

10. VWO (Visual Website Optimizer — vwo.com) A full-featured A/B testing platform for landing pages and websites. More accessible than Optimizely for solo creators. The free tier allows basic A/B tests. The hypothesis library feature — where you can browse test ideas organized by element type — is particularly useful for creators who are new to landing page testing and need inspiration for what to test first.

11. Beehiiv (beehiiv.com) — Newsletter Platform with Built-in A/B Testing For creators building email newsletters, Beehiiv offers subject line A/B testing on all paid tiers. What makes it notable is the quality of the test reporting — including statistical significance indicators — which is more rigorous than many competing email platforms. The platform is worth evaluating if you are choosing or switching email platforms and A/B testing is a priority.

12. Google Optimize Successor / Firebase A/B Testing (firebase.google.com) Google Optimize was sunset in 2023, but Firebase A/B Testing (for app-based creators and products) and Google's broader experimentation infrastructure remain available. For creators building web-based products or membership platforms, Firebase A/B Testing offers free, sophisticated experiment management.