Peer-reviewed article

Raising the Bar for Evaluating Effectiveness of Early Childhood Interventions

Jack P. Shonkoff, Dana Charles McCoy · 20 Sept 2021

Abstract

Despite growing recognition of the foundational importance of the early childhood period, the persistence of significant socioeconomic, racial, and ethnic disparities in school readiness remains a challenge.1 This challenge is reflected in decades of program evaluations across multiple sectors revealing modest effect sizes, limited replication of promising interventions, and minimal evidence of significant impact at scale.2,3 In this issue of Pediatrics, Chamberlain et al4 address an important question facing pediatricians: What can be done in a primary care practice to strengthen the building blocks of literacy for children who are not reached by other programs? The contributions of this article fall within two categories. The first includes the findings of the TipsByText (TbT) evaluation. The second, and arguably the more instructive, includes the questions that the study leaves unresolved.The premise for this investigation is based on the proposition that young children’s access to primary health care in the United States is near universal, thus offering an opportunity to engage families who are difficult to reach through other services (eg, preschool). The rationale for testing an existing intervention that is easy to incorporate within the health care setting is well founded, and the positive overall impact of TbT on children’s knowledge of literacy fundamentals is encouraging.Although the potential benefits of text messaging for isolated families are promising, a more important contribution of this article is the opportunity to consider the questions it is unable to answer. Namely, what is the value of a statistically significant mean difference between a treatment and control group that is not embedded in an explicit theory of change that connects the intervention to the measured outcome? When that precision and linkage are missing, what are the implications of the findings for either practical application or further investigation?More specifically, the theory of change for an intervention that uses text messages to empower parents to improve their child’s early literacy skills assumes that the messages will (1) be received and viewed by parents; (2) inform and motivate them; and (3) lead them to engage in greater interactivity in language exchanges with their children, which in turn will improve early literacy skills. The lack of data in this report on program take-up and parental empowerment, combined with null impacts on parents’ self-reported reading and literacy interactions with their children, generates more questions than answers about the mechanisms through which the intervention affected children’s literacy knowledge.Similarly, although Chamberlain et al do take the important step to examine subgroup differences in outcomes, the absence of a conceptual framework to guide these analyses makes their interpretation challenging. What meaning can be attributed to the relatively larger intervention effects associated with first-born status, 2-parent families, English-speaking caregivers, and children >48 months of age, with no differences on the basis of caregiver education? Without hypothesized causal mechanisms, how can pediatricians be expected to modify their practice in response to potential variation in intervention effectiveness?5The challenges faced by Chamberlain et al in explaining how and why the TbT program led to stronger versus weaker impacts on child literacy are not unique. Indeed, in a recent systematic review of 277 high-quality early childhood program evaluations spanning 5 decades, we found that information on program take-up was reported in fewer than half of the studies, whereas process outcomes and tests of moderation were each reported in fewer than 20%.6 Collectively, this lack of consideration for whether programs are used by intended recipients, how and why they work, and for whom they are most effective has led to more questions than answers about how to maximize both impact and equity in early childhood intervention.In summary, although its average effect was confirmed, the conclusion that parent-directed texting produced “three months of literacy gains” in young children without any evidence of take-up, impact on parenting behaviors or parent-child interaction, or explanation of subgroup heterogeneity makes it difficult to identify the active ingredients and, in turn, generalize the stated conclusions about the TbT program. Significant mean differences between treatment and control groups on outcomes that are not linked to the intervention through a clear mechanism to explain the changes limits our understanding of how to match services to children and parents who will benefit from them.The most important contribution of this instructive article is not its recommendation for text messaging to improve early literacy in the context of pediatric primary care. Rather, its power lies in the opportunity it provides to underscore a fundamental challenge faced by early childhood services broadly for half a century. Reporting average effects and subgroup differences without exploring causal explanations in programs that target child development is a major limitation affecting many similar studies. The entire early childhood field would benefit substantially from confronting this important challenge, which impedes our ability to develop efficient and effective interventions that can be targeted to those who would most benefit. Although the TbT program was found to lead to some improvement, we need further studies designed explicitly to elucidate the complex mechanisms underlying its reported effects on early literacy to achieve sustainable, population-level impacts.