Case Study: The Billion-Dollar Gamble — EdTech Without Evidence
The Promise
The educational technology (EdTech) narrative has been remarkably consistent across decades: technology will transform education. In the 1980s, it was personal computers. In the 1990s, it was the internet. In the 2000s, it was interactive whiteboards. In the 2010s, it was tablets and one-to-one device programs. In the 2020s, it is AI-powered adaptive learning.
Each technology cycle follows the same pattern:
- The announcement: A new technology is presented as transformative for education
- The investment: Billions of dollars flow into development, deployment, and marketing
- The adoption: School districts purchase the technology, often with limited pilot testing
- The ambiguous results: Research produces mixed findings — some positive in specific contexts, others neutral or negative
- The narrative preservation: The ambiguity is interpreted as "we haven't done it right yet" rather than as evidence of limited effectiveness
- The next technology: Attention shifts to the next transformative technology, and the cycle repeats
The Evidence
The most comprehensive evidence on educational technology comes from large-scale international studies:
OECD (2015): Students, Computers and Learning: Making the Connection analyzed the relationship between technology use and educational outcomes across OECD countries. The findings were striking: "Students who use computers moderately at school tend to have somewhat better learning outcomes than students who use computers rarely. But students who use computers very frequently at school do a lot worse in most learning outcomes." More technology was not better. The relationship was inverted-U at best.
The OECD report concluded: "Despite considerable investments in computers, internet connections and software for educational use, there is little solid evidence that greater computer use among students leads to better scores in mathematics and reading."
One-to-one laptop programs: Multiple evaluations of programs that provide every student with a laptop have produced mixed results. Some studies find modest positive effects on writing and engagement; others find no effect on core academic outcomes; some find negative effects on student attention and behavior. The consistent finding: providing the device, without substantial changes to instruction, does not improve learning.
Interactive whiteboards: Promoted heavily in the 2000s, interactive whiteboards were installed in classrooms across the UK, U.S., and other countries at enormous expense. Research consistently found that interactive whiteboards improved engagement (students enjoyed using them) but had little or no measurable effect on learning outcomes. The technology was measuring — and optimizing for — the wrong metric.
Capital-Sustained Error in Education
The EdTech industry operates through the same capital-sustained error dynamic identified in Chapter 29, adapted to the education market:
Narrative-market fit. The narrative "technology transforms education" has powerful narrative-market fit. Parents expect technology in classrooms. Politicians want to appear modern and innovative. Administrators want to demonstrate forward-thinking leadership. EdTech companies have products to sell. The narrative serves everyone's interests — except the students whose learning outcomes are the ostensible purpose.
The evidence vacuum. Because education research is structurally difficult (Section 30.3), the evidence base for most EdTech products is thin. This creates a vacuum that marketing fills. An EdTech product can be marketed as "research-based" on the strength of a single small study, a pilot program with self-selected participants, or even just a theoretical alignment with learning science principles. The standard of evidence for EdTech procurement is far lower than the standard of evidence for pharmaceutical approval.
The missing feedback loop. In technology markets outside education, products face market feedback — if a product doesn't work, customers stop buying it. In education, the feedback loop is broken: - Students (the end users) don't make purchasing decisions - Teachers (the daily users) often have limited input on procurement - Administrators (the purchasers) may not use the product or measure its impact - EdTech companies (the sellers) have no obligation to demonstrate efficacy - Learning outcomes (the relevant measure) are difficult to attribute to any single intervention
The absence of a functioning feedback loop means that EdTech products can persist in schools for years without evidence of effectiveness — sustained by contracts, institutional inertia, and the sunk cost of implementation.
The COVID-19 Acceleration
The COVID-19 pandemic forced the largest unplanned experiment in educational technology in history. When schools closed in March 2020, education moved online almost overnight. The results provided a massive natural experiment in EdTech effectiveness.
The findings were sobering. Extensive research on pandemic-era learning documented significant learning losses — particularly among disadvantaged students. While technology enabled continuity of some form of instruction, the quality of that instruction and its effect on learning were, in most contexts, substantially inferior to in-person teaching.
This should have been a crisis point (Chapter 19) for the EdTech narrative — undeniable evidence that technology alone, without the social infrastructure of schools, produces significantly worse outcomes. Instead, the narrative adapted: "The problem wasn't the technology — it was the emergency context. Given more time and better implementation, technology-mediated learning will work."
This is the unfalsifiable defense pattern. If technology works in classrooms, it validates the narrative. If technology fails during a pandemic, the failure is attributed to context rather than to limitations of the approach. No outcome can disprove the core thesis.
Failure Mode Analysis
Capital-sustained error (Ch.29). EdTech companies have raised billions in venture capital and public market funding. The capital creates an ecosystem — companies, employees, sales teams, conferences, media coverage — with financial incentives to maintain the narrative that technology transforms education, regardless of evidence.
Streetlight effect (Ch.4). EdTech products measure what is measurable — engagement, time-on-task, completion rates — and present these metrics as evidence of learning. But engagement is not learning. A student who spends more time on a platform is not necessarily learning more — they may be distracted, clicking through content, or practicing skills at a level too easy to produce growth.
Incentive structures (Ch.11). Every actor in the EdTech ecosystem has incentives that diverge from student learning: companies maximize sales, administrators maximize visible innovation, politicians maximize constituent approval, and teachers minimize disruption to their existing practice.
Plausible story (Ch.6). "Technology transforms everything else — of course it will transform education" is a compelling narrative that draws on genuine evidence from other sectors (technology has transformed communication, commerce, entertainment) and extrapolates to education. But education is structurally different from these sectors — learning is a human process that depends on relationships, motivation, and cognitive development in ways that resist technological substitution.
Analysis Questions
1. The OECD found that moderate computer use was associated with somewhat better outcomes, but heavy computer use was associated with significantly worse outcomes. Apply the complexity-hiding-in-simplicity framework (Chapter 15): how does the "technology transforms education" narrative hide this nuanced finding?
2. Compare the EdTech evidence vacuum with the crypto evidence vacuum (Chapter 29). Both involve massive capital investment with limited evidence of the claimed benefits. What structural similarities make both markets susceptible to capital-sustained error? What differences, if any, make one more susceptible than the other?
3. Design an evidence-based procurement process for educational technology that would require EdTech companies to demonstrate efficacy before adoption. What evidence standard would you set? Who would evaluate the evidence? What resistance would you face from EdTech companies, administrators, and politicians?
4. The COVID-19 experience provided massive evidence that technology alone does not substitute for in-person schooling. Apply the crisis-driven correction framework (Chapter 19): will this evidence produce lasting change in EdTech investment, or will the field return to pre-pandemic levels of technology enthusiasm? Predict the trajectory using the Correction Speed Model.