The Era of Edtech Efficacy

Graham Forman, Managing Director and Founder, Edovate Capital

As an operator and investor in edtech for more than 20 years, I have heard about and discussed the need for edtech “efficacy” for as long as I can remember. The vision was simple—let’s find out what products/services work and for which students and teachers. Efficacy in edtech is about the ability to produce (and prove) an intended result for students and teachers, and momentum around the concept got a boost with the passage of No Child Left Behind in 2001.

Just a year later, the U.S. Department of Education started the What Works Clearinghouse, which established national standards for measuring efficacy in education. Two types of studies—quasi-experimental (QED) studies and randomized controlled trials (RCTs)—were allowed, and the ecosystem was energized to gather robust, actionable data on efficacy. Unfortunately, the reality of proving edtech efficacy was much more difficult than the perception since the “gold standard” of educational research—the randomized control trial or “RCT”—turned out to be more difficult to execute in education than in other fields, such as medicine.

Educational conditions are more complex than those in a medical clinic, and RCTs can take several years and be cost-prohibitive for the vast majority of edtech companies, especially those at the early stages. On top of that, many schools and district buyers preferred to make buying decisions based more on testimonials from peers than the research bonafides of edtech providers. That has changed in recent years, and we’re at an inflection point where an increasing number of well-researched, effective edtech products are now available to schools and districts. I believe this is the moment for edtech efficacy, and there’s a confluence of factors driving this momentum.

1) The ESSA Tiers of Evidence. Under the Every Student Succeeds Act (ESSA) passed in 2015, schools and districts are encouraged to choose evidence-based interventions that have been shown to improve student outcomes. ESSA’s Tiers of Evidence give schools, districts, and edtech providers a framework for determining which programs, practices, strategies, and interventions work in which contexts and for which students. Importantly, the Tiers of Evidence recognize a continuum of efficacy appropriate to each stage of the product. Early-stage products are more likely to meet basic requirements (such as having a logic model), while more mature products are vetted by deeper research (such as one or more well-designed, well-implemented randomized control experiments).

2) Demand from educators is more than just compliance with ESSA. Educators value research and increasingly ask for it as they evaluate edtech products. I see more and more evidence requests from administrators and teachers who are evaluating different products. U.S. Department of Education leadership and guidance on the importance of evidence has helped increase awareness and demand, which has trickled down to state departments of education and local education agencies across the country.

3) With so many tools to choose from, evidence is a way to differentiate. As schools and districts have digitized their learning environments, there’s been an explosion of edtech tools available to them. There are now more than 1,400 (!) edtech products in use in a typical school district (source: LearnPlatform). With so many tools to choose from, educators increasingly sort through the pack by demanding evidence-based products.

4) Access to affordable research partners. RCTs are still the gold standard for research, but they’re complex, intensive, time-consuming, and can cost hundreds of thousands of dollars. They’re accessible for mature, later-stage companies and products but less so for early-stage companies with fewer resources. This is why companies such as LearnPlatform have stepped in and provided rapid-cycle evaluations through services like evidence-as-a-service. Other leading companies serving the research needs of edtech providers and schools include WestEd and SRI. The work of the Edtech Evidence Exchange has also been instrumental in promoting evidence and accessible research.

"We’re at an inflection point where an increasing number of well-researched, effective edtech products are now available to schools and districts"

5) More entrepreneurs are baking research and evidence into their products from the start. Jennifer Carolan and Molly B. Zielezinski wrote an outstanding article entitled “Debunking the ‘Gold Standard’ Myths in Edtech Efficacy”, where they argue that the concept of an “efficacy portfolio” is a better frame for demonstrating efficacy over time. They write, “An efficacy portfolio is a collection of evidence gathered over time that captures different elements of whether and how a product is ‘working.’ Some of the evidence may include a QED or RCT—but those approaches no longer bear the full weight of demonstrating efficacy than a QED or RCT alone.” This framing is useful because it helps enable entrepreneurs to think about how to include research and evidence in their products from the beginning. We see more and more early-stage companies such as Bamboo Learning, Boddle, and Levered Learning committing to research and evidence from the start.

Edtech efficacy is still in its early days, and there are still edtech companies that dress up anecdotes or testimonials from customers and call them “evidence,” but such tactics are fading as educators demand research and evidence and as research becomes more accessible and affordable to entrepreneurs. The bottom line is that efficacy is good for school districts, edtech providers, students, and the entire education ecosystem. As an active edtech investor, my thesis is that the most effective tools will be the most used by educators and students and become the best businesses over the long run. It’s not a guarantee of success, but it’s an advantage. The means to prove product efficacy are not out there in the future but are accessible now. For founders just getting started, make sure to build evidence of efficacy into your product from the ground up, take advantage of existing research, and build an efficacy portfolio over time. You and your company will win in the long run, and, most importantly, your educators and students will be better served.

EdTech