Case Study 02: OkCupid's Public A/B Testing Disclosure (2014)

Christian Rudder's "We Experiment on Human Beings" and the Public Reaction

Background

In July 2014, one month after the Facebook emotional contagion controversy had generated widespread outrage about undisclosed platform experimentation, OkCupid co-founder Christian Rudder published a blog post on the OkCupid company blog with a title that seemed deliberately provocative given the surrounding news environment: "We Experiment on Human Beings!"

The exclamation mark in the title was not accidental. Rudder's post was written with a defiance that contrasted sharply with Facebook's apologetic response to the emotional contagion controversy. While Facebook had issued measured, careful statements expressing regret for the "concern" generated by its study, Rudder was asserting not merely that OkCupid experimented on its users, but that it was entitled to, that this was the right and normal thing for digital companies to do, and that any product that claimed otherwise was either lying or failing to take its product seriously.

The post described three specific experiments OkCupid had conducted on its users: a study of whether the site's compatibility matching algorithm actually predicted relationship success; an experiment that removed profile photos from all accounts for a day; and, most controversially, an experiment in which OkCupid told users that they were highly compatible with randomly selected potential matches regardless of what OkCupid's actual algorithm computed.

The OkCupid disclosure case is instructive not because OkCupid's experiments were more harmful than the Facebook emotional contagion study — arguably they were less harmful — but because of what Rudder's defiant framing, and the public reaction to it, reveal about the norms, ethics, and politics of platform behavioral experimentation. Rudder made explicit arguments that most platforms practice implicitly, and the public's response to those arguments illuminated the gap between platform-side and user-side expectations of what the platform relationship entails.

Timeline

2004 — OkCupid founded by Chris Coyne, Sam Yagan, Max Krohn, and Christian Rudder, all Harvard alumni. The site is notable from the beginning for its data-driven, algorithmically-transparent approach to online dating.

2011 — OkCupid acquired by IAC/InterActiveCorp for approximately $50 million. The site continues operating under its own brand with its original leadership team largely intact.

2012 — Rudder publishes "Dataclysm"-era blog posts on the OkCupid blog, sharing statistical analyses of user behavior on the platform. These posts, which include analyses of messaging patterns, racial preferences, and match success rates, generate significant attention and establish Rudder's reputation as a data-transparent voice in the technology industry.

June 2014 — Facebook emotional contagion paper is published. Controversy erupts. Facebook issues a statement. Regulatory investigations are opened in multiple jurisdictions.

July 28, 2014 — Rudder publishes "We Experiment on Human Beings!" The post describes three OkCupid experiments in detail. The post simultaneously generates admiration (for transparency) and criticism (for the content of the experiments, particularly the false match compatibility disclosure).

August 2014 — Media and academic commentary on the OkCupid disclosure continues. Comparisons with the Facebook emotional contagion study are widespread. The Federal Trade Commission is reportedly monitoring the disclosures.

2014 — Rudder publishes "Dataclysm: Who We Are (When We Think No One's Looking)," which extends the OkCupid data analysis approach to a full book. The book is generally well received; the experimentation disclosure continues to generate commentary.

2015 onward — The OkCupid case is regularly cited in academic and policy discussions about platform experimentation ethics as a counterpoint to the Facebook emotional contagion case. Rudder's arguments become touchstones in debates about the legitimacy of commercial behavioral research.

The Experiments Described

Rudder's blog post described three experiments in detail, each with a specific purpose and a specific finding.

Experiment 1: Does OkCupid's match percentage actually work?

OkCupid's core product feature is a compatibility matching algorithm that computes a numerical "match percentage" between users based on their responses to a questionnaire. The algorithm is central to the site's value proposition — users are supposed to be more likely to form successful relationships with high-match partners than with low-match ones.

To test whether the algorithm actually worked, OkCupid conducted an experiment in which users were shown false match percentages — users who had a high actual match percentage were sometimes told they had a low match percentage, and users who had a low actual match percentage were sometimes told they had a high match percentage.

The finding was that the actual match percentage mattered: users with a high actual match percentage who were told they had a low match percentage still messaged and connected at higher rates than users with a low actual match percentage who were told they had a high match percentage. But the communicated match percentage also had an independent effect: being told you are compatible makes you act as if you are compatible, regardless of whether you actually are.

Experiment 2: Removing profile photos

OkCupid briefly removed all profile photos from the site for a single day, giving the site the appearance of a text-only dating service. The experiment was designed to test the hypothesis that photos dominate the user experience to such a degree that textual profile information is effectively ignored.

The finding confirmed this hypothesis: conversations during the no-photo period were substantially longer, more substantive, and more likely to include exchange of contact information than typical conversations. Users were more engaged with textual self-presentation when visual information was unavailable. When photos were restored at the end of the experiment, users were notified — and many of the conversations that had begun during the no-photo period ended abruptly when photos became visible.

The no-photo experiment was arguably the least ethically problematic of the three: it temporarily changed a product feature in a way that most users would not have found objectionable, and it produced findings that had genuine value for understanding what drives meaningful connection on dating platforms.

Experiment 3: Telling users they are "good matches" when they are not

The third experiment, and the one that generated the most ethical concern, involved telling users that randomly selected partners were "extremely good matches" when OkCupid's algorithm computed that they were not compatible. Users who were told they were high matches interacted with those partners at significantly higher rates than users who were told nothing about their compatibility.

This experiment raised the most direct ethical concerns because it involved actively providing false information to users in a context where accurate information was the core service promise. OkCupid's value proposition was that it would help users find genuinely compatible partners; the experiment deliberately undermined this value proposition for subjects by providing inaccurate compatibility information.

Rudder framed the finding as evidence that "suggestion is powerful" — that telling people they are compatible makes them behave compatibly, regardless of underlying compatibility. This is an interesting psychological finding. But it was obtained by deceiving users in a context directly relevant to significant personal decisions (romantic connection, potentially long-term relationships) in a way that users had not consented to and would not have expected.

Rudder's Ethical Argument

The most important aspect of the OkCupid disclosure was not the experiments themselves — it was Rudder's explicit defense of platform experimentation as a general practice, which made visible the unstated ethical position that most platforms hold but rarely articulate.

Rudder's argument had several components.

First, all products are experiments. Every time a new product ships, designers are testing whether their product choices produce the intended outcomes. A/B testing is simply a more rigorous, more formal version of this universal practice. Singling out explicit A/B testing for ethical criticism while accepting the continuous implicit experimentation of product development is inconsistent.

Second, experimentation is how products improve. Without testing, companies cannot know whether their choices are right. The alternative to experimentation is either stagnation (keeping the product unchanged) or blind change (making changes without evidence about their effects). Neither is better for users than informed, evidence-based change.

Third, users implicitly accept this by using the product. When a user creates an OkCupid account, they are participating in a system that has always been data-driven and experiment-based. The platform's data-driven character is not hidden; it is publicly known. Users who find this problematic can choose not to use the platform.

Fourth, and most provocatively: "OkCupid doesn't really know what it's doing. Neither does any other website." Rudder argued that the uncertainty about what actually works in product design makes experimentation not merely acceptable but obligatory — that any company that claims to know what is best for users without testing that claim is engaged in a more egregious self-deception than companies that acknowledge uncertainty through empirical testing.

The Public Reaction

The public reaction to Rudder's post was sharply bifurcated along lines that revealed the fundamental values conflict at the heart of platform experimentation ethics.

A significant portion of the reaction was positive. Technology journalists, data scientists, and product design professionals largely praised Rudder's transparency. After years of platform experimentation conducted invisibly, here was a company executive openly acknowledging what his company did, explaining why, and defending the practice. Even if you disagreed with the ethical conclusions, the transparency was refreshing and rare.

The more negative reaction came from privacy advocates, ethicists, and users who focused on the content of the experiments rather than the transparency of the disclosure. The false match compatibility experiment, in particular, was widely criticized for actively deceiving users in a context directly relevant to their emotional lives and romantic decisions. Critics noted that the "users accept this by using the product" argument was inadequate: users accept OkCupid's data-driven approach, but they do not accept being provided with actively false information about compatibility.

The philosophical tension between these two reactions maps onto a fundamental disagreement about what the platform-user relationship entails. The Rudder position holds that using a commercial digital service creates no expectation of freedom from experimentation — that the platform is entitled to test its products on its users as part of the normal operation of a data-driven business. The critic position holds that certain kinds of experimentation — particularly deception in contexts of emotional significance — violate user trust in ways that the platform relationship does not authorize, regardless of what terms of service say.

Comparison to the Facebook Emotional Contagion Study

The simultaneity of the OkCupid disclosure and the Facebook emotional contagion controversy created a natural comparison that commentators extensively developed.

In many respects, the OkCupid experiments were less ethically problematic than the Facebook study. They were shorter in duration. They were less massive in scale. They involved a platform with a clearly commercial character (matching romantic partners) rather than a platform that positions itself as a neutral social infrastructure. And — crucially — they were disclosed voluntarily and publicly, rather than becoming known only through academic publication that users could not access or evaluate.

In other respects, the false match compatibility experiment was more directly problematic. The Facebook manipulation adjusted the emotional valence of content users saw; it did not provide users with explicitly false information. The OkCupid experiment explicitly told users that an algorithm-computed poor match was an excellent match — a direct factual deception in a context where accurate information was the central product promise.

This comparison illuminates an important analytical point: the ethics of platform experimentation cannot be reduced to a single variable. Scale, consent, disclosure, the nature of the manipulation, the context in which it occurs, and the potential for harm are all relevant dimensions. Different experiments occupy different positions on these dimensions, and ethical assessment requires attention to the multidimensional character of the conduct.

The Argument for Transparency: What Rudder Got Right

Rudder's post, whatever its ethical limitations, made a genuinely important argument: transparency about experimentation is better than opacity. The norm in the technology industry was — and to a large extent remains — to conduct behavioral experiments without disclosure, treating the experimental character of product development as proprietary information to be protected rather than as a relevant fact about the user experience that users are entitled to know.

Rudder's transparency, by contrast, gave users actual information on which to base their decisions about platform use. Users who read his post and found the described experiments unacceptable had the information they needed to choose not to use OkCupid. Users who found the experiments reasonable — or who found OkCupid's transparency refreshing relative to the undisclosed experimentation they suspected other platforms of — could make an informed choice to continue using the platform.

This informational value of disclosure should not be dismissed simply because what is disclosed is ethically problematic. A world in which platforms disclose their experimentation practices is a world in which users, regulators, and the public can form accurate judgments about those practices and respond accordingly. A world in which practices are concealed is one in which accurate judgment is impossible. From this perspective, Rudder's defiant transparency, even while defending practices that deserved ethical criticism, was a genuine contribution to the broader project of public accountability for platform behavioral practices.

What This Means for Users

Transparency is valuable even when what is disclosed is ethically problematic. OkCupid's disclosure was more valuable to users than Facebook's opacity, even though OkCupid's experiments were in some respects more directly objectionable. Users deserve information about how they are being treated by platforms, and disclosure — even defensive disclosure — is better than silence.

The "you accepted the terms of service" argument is legally coherent but ethically inadequate. Rudder's argument that users implicitly accept experimentation by using a commercial platform is a legally defensible position in most jurisdictions. It is not an adequate ethical response to the specific concerns raised by active deception, emotional manipulation, or harm to vulnerable users. Legal permissibility and ethical acceptability are different standards.

The no-photo experiment suggests that platform design choices have significant effects on the quality of human connection. OkCupid's finding that conversations were more substantive and successful when visual information was unavailable suggests that visual-first design choices — prioritizing profile photos — may systematically undermine the quality of connection on dating platforms. This is a finding with implications beyond OkCupid: platform design choices that appear neutral may have significant effects on the kinds of human connection the platform facilitates.

Ask "what are they experimenting with?" not just "are they experimenting?" The ethical question for platform users should not be whether their platform conducts A/B tests — virtually all of them do — but what they are testing, with what optimization targets, with what potential for harm, and with what oversight. Rudder's transparency gave OkCupid users the information to answer this question. The absence of comparable transparency on most other platforms means users cannot.

Discussion Questions

Rudder's defiant transparency contrasted sharply with Facebook's apologetic opacity. Which approach is more ethically defensible? Does transparency about ethical problems make them more or less serious? Does voluntary disclosure of problematic practices reduce or increase moral accountability?
Evaluate Rudder's core argument: "All products are experiments." Is this argument persuasive as a defense of explicit A/B testing? Does the universality of implicit product experimentation make explicit, randomized experimentation ethically equivalent to it?
The false match compatibility experiment provided users with actively false information about romantic compatibility. Evaluate this against the standard of "the platform is entitled to test its products." Is there a principled distinction between experiments that adjust what information users see (Facebook's emotional contagion study) and experiments that provide users with actively false information (OkCupid's false match study)?
The no-photo experiment produced findings with genuine value for understanding connection quality on dating platforms. If the experiment itself was ethically acceptable (a brief product change with no deceptive element), does this suggest a framework: experiments that involve only product changes (even without disclosure) are acceptable, while experiments that involve deception are not? Where would you draw the line?
Rudder's transparency enabled users to make informed choices about OkCupid. Is individual user exit — choosing not to use a platform because you object to its experimental practices — an adequate response to the ethical concerns raised by platform experimentation? Or do the network effects of platforms make exit a meaningfully less available choice than it appears in theory?