Now Available: Impact Evaluation in Practice handbook (2nd Edition)

The second edition of the ​I​mpact Evaluation in Practice handbook is now available. The handbook is a comprehensive introduction to impact evaluation for policymakers and development practitioners. The updated version covers the latest techniques for evaluating programs, with expanded case studies.

---

New Publications
see publications page for full list

Road Maintenance and Local Economic Development: Evidence from Indonesia’s Highways
Journal of Urban Economics (conditionally accepted)
(Working Paper, 2023)
Abstract: This paper estimates the local welfare impacts of highway maintenance investments. We instrument road quality exploiting Indonesia’s two-step budgeting process for allocating funding to local road authorities. Using comprehensive data on road quality from 1990-2007, we find evidence that better roads help manufacturers create new jobs, enabling worker transitions out of informal employment, and increasing labor income. Road quality also changes the cost of living, reducing perishable food prices but also raising housing prices. We estimate the elasticity of household welfare with respect to road quality to be 0.09 and the benefit/cost ratio for road maintenance investments to be1.8.

Digital Collateral
Quarterly Journal of Economics (2024) 
(working paper, 2023)
Abstract: A new form of secured lending using “digital collateral” has recently emerged, most prominently in low- and middle-income countries. Digital collateral relies on lockout technology, which allows the lender to temporarily disable the flow value of the collateral to
the borrower without physically repossessing it. We explore this new form of credit both in a model and in a field experiment using school-fee loans digitally secured with a solar home system. Securing a loan with digital collateral drastically reduces default rates (by 19 pp) and increases the lender’s rate of return (by 38 pp). Employing a variant of the Karlan and Zinman (2009) methodology, we  decompose the total effect on repayment and find that roughly one-
third is attributable to adverse selection, and two-thirds is  attributable to moral hazard. In addition, access to digitally secured school-fee loans significantly increases school enrollment and school-related expenditures without detrimental effects to households’ balance sheet.

Financing Municipal Water and Sanitation Services in
Nairobi’s Informal Settlements
Review of Economics and Statistics (forthcoming)
(Working Paper, 2023)
Abstract: We test two ways to improve revenue collection efficiency for water and sanitation utilities: (i) face-to-face engagement between utility staff and customers and (ii) contract enforcement for service disconnection due to nonpayment in the form of transparent and credible disconnection notices. Engagement has no effect, while enforcement significantly increases payment. We find no effect
on access to water, perceptions of the utility, relationships between tenants and property owners, or on tenant mental well-being nine months after the interven-tion. These results suggest that  transparent contract enforcement was effective at improving revenue collection efficiency without incurring significant observed social or political costs.

Vulnerability and Clientelism
American Economic Review ( 2022)
(Publication)
Abstract:This study argues that economic vulnerability causes citizens to participate in clientelism, a phenomenon with various pernicious consequences. To examine how reduced vulnerability affects citizens’ participation in clientelism, we employ two exogenous shocks to vulnerability. First, we designed a randomized control trial to reduce household vulnerability: our development intervention constructed residential water cisterns in drought-prone areas of Brazil. Second, we exploit rainfall shocks. We find that reducing vulnerability significantly decreases requests for private goods from politicians, especially among citizens likely to be in clientelist relationships. Moreover, reducing vulnerability decreases votes for incumbent mayors, who typically have more resources for clientelism.

Trust and saving in financial institutions by the poor
Journal of Development Economics (2022)
(Publication)
Abstract: We randomly assigned beneficiaries of a conditional cash transfer program in Peru to attend a 3 h training session designed to build their trust in financial institutions. We find that the intervention: (a) increased trust in banks, but had no effect on financial literacy, and (b) increased savings over a ten month period. The increase
in savings represents a 1.4 percentage point increase in the savings rate out of the cash transfer deposits, and a 0.4 percentage point increase in the savings rate out of household income.

Another Brick in the Wall: Effect of Non-contributory Pensions on Material and Psychological Well-Being
(Journal of Behavioral and Economic Behavior (2022).
(Publication, Working Paper)
Abstract: We explore the effect of non-contributory pensions on the well-being with a field experiment in Paraguay. Households with a beneficiary increased consumption by 44 percent and older adults increased their leisure by reducing labor supply. The program also improved a psychological well-being index by 0.48 standard deviations.

How scheduling systems with automated appointment reminders improve health clinic efficiency                           Journal of Health Economics (2022).
(Publication, Working Paper)
Abstract: Missed clinic appointments burden health care systems through inefficient use of staff time and resources. Scheduling software with automatic appointment reminders shows promise to improve clinics’ management through timely cancellations and re-scheduling, but at-scale evidence is missing. We study a nationwide text message appointment reminder program in Chile implemented at primary care clinics for patients with chronic disease. We find that visits by 5.0% in the first year and 7.4% in the second.

Do private providers give patients what they demand, even if it is inappropriate? A randomized study utilizing unannounced standardized patients in Kenya
British Medical Journal - Open (2022). 
(Publication)
Abstract: Some quality improvement strategies encourage patient engagement; however, patients demanding inappropriate medicines can favor the selection of resistant microbial strains. This study examines the effects when patients demand different types of inappropriate medicines. We conducted an experiment where unannounced standardized patients (SPs), locally recruited individuals trained to simulate a standardized case, present at private clinics. SPs portraying caretakers of a watery diarrhea childhood case scenario (in absentia) conducted visits at 200 private, primary care clinics. Half of the clinics were randomly assigned to receive an SP demanding amoxicillin (an antibiotic); the other half, an SP demanding albendazole (an antiparasitic drug often used for deworming). Providers significantly increased the dispensing rate for those who demand albendazole, but did for amoxicillin.

How Debit Cards Enable the Poor to Save More
]Journal of Finance (2021) 
(Publication, Working Paper)
Abstract: We study an at-scale natural experiment in which debit cards are given to cash transfer recipients who already have a bank account. We find that beneficiaries accumulate a savings stock equal to 2 percent of annual income after two years with the card. The increase in formal savings represents an increase in overall savings, financed by a reduction in current consumption.

Air Conditioning and Inequality
Global Environmental Change (2021) 
(publication, working paper)
Abstract: We use household-level microdata from 16 countries to characterize empirically the relationship between climate, income, and residential air conditioning. Not only do richer countries have much more air conditioning than poorer countries, but within countries adoption is highly concentrated among high-income households. The pattern of adoption is particularly stark in relatively low-income countries like India, where the vast majority of adoption will be concentrated among the upper income tercile. We use our model to forecast future adoption, show how patterns vary across countries and income levels, and then discuss what these patterns mean for health, productivity, and educational inequality.

Aspiration Adaptation in Resource-Constrained Environments 
Journal of Urban Economics (2021).
(Publication, Working Paper)
Abstract: We use a multi-country field experiment to test the effect of a slum-housing intervention on the evolution of housing aspirations. Initially after the intervention, we observe a large housing gap in aspirations to upgrade their dwelling relative to the treatment group, echoing an aspiration to “keep-up” with the treated Joneses’. However, after 2 years of treatment exposure, the aspirational gap completely disappears and no effects are found on housing investment. Our evidence suggests that simply fostering housing aspirations may be insufficient to encourage housing investment in poor neighborhoods, and thus slum-upgrading policies designed to indirectly stimulate housing expansion may not be as effective as they promise to be.

New Working Papers
See working papers page for full list

New How Managers Can Use Purchaser Performance Information to Improve Procurement Efficiency                 (working paper 2024)
We examine the effect of performance monitoring in public procurement through the lens of organizational culture in a principal-agent model where the manager (principal) and buyers (agents) may have different beliefs about how much the government values efficiency. We show that the effect of performance information not only increases efficiency but is greater when the buyer’s belief is stronger than the manager’s belief. We leverage a new e-procurement system in Chile to test these ideas by randomizing monthly reports on the purchasing performance of buyers and further whether the individual performance reports were disclosed to managers. We find that the reports generated sizable reductions in overspending — with savings reaching a 15% reduction or 0.1% of GDP — but only when individual performance was observable to managers. This is consistent with extrinsic motivation rather than intrinsic motivation driving buyer behavior. Consistent with the theoretical model, we also find that the gain in efficiency is concentrated in procurement units where buyer belief that the government cares about efficiency is stronger than manager belief. Our results highlight the key role played by organizational culture in mediating the impact of purchasing performance
information on preventing the misuse of public resources.

New The Essential Role of Altruism in Medical Decision Making (working paper 2024)
Patients rely on medical care providers to act in their best interests because providers understand disease pathology and appropriate treatment much better than patients. Providers, however, not only give advice (diagnose) but also deliver (sell) treatments based on that advice.  This creates a moral hazard dilemma where provider financial interests can diverge from patient interests, especially when providers are motivated by more by profits than by altruism. We investigate how profit motivated versus altruistic preferences influence medical care decision making in the context of malaria in Kenya. We measured the appropriateness of care using data from an audit study that employed standardized patients (SP) who pretended to be real patients and presented the identical clinical case scenario to providers. The SPs were confirmed to be malaria negative before and after field work with a very reliable and sensitive blood test at a high-quality laboratory. We measured provider preferences using a lab in the field, real stakes, modified version of the dictator game. We find that more profit-motivated providers report higher rates of false-positive malaria test results than do more altruistic providers. Specifically, purely profit motivated providers report 30 percentage points more positives than providers who altruistically motivated, and providers likely knew that the positive results that they reported to their patients were false. We also find that more profit motivated providers sold more unnecessary antimalarial drugs than did more altruistic providers. Based on mediation analysis, more profit-oriented providers sold more drugs not only because they knowingly reported more false positives, but also because they promoted drugs sales more conditional on a positive test result. Thus, more profit motivated providers seem to have chosen to misrepresent test results to sell more unnecessary malaria-related drugs.

New Version - The Sweet Life: Long-Term Effects of a Sugar-Rich Early Childhood
Journal of Political Economy - Revision requested
(Working Paper, 2023)
We provide new evidence on the effects of exposure to sugar-rich diet in early life on adult health and economic well-being. Sugar-rich diet early in life may impact theseou tcomes through physiological programming and/or elevated sugar consumption acrossvblifespan as preference for sweetness forms early and sugar may be addictive. As a source of variation in early-life sugar intake, we exploit the end of the post-WW2 rationing of sugar and sweets in 1953 in the United Kingdom, where rationed limits were comparable to today’s dietary guidelines. We use regression discontinuity design informed by the sharp increase at more than 100% in sugar consumption - but not other foods - shortly after the sugar rationing ended. We find that children with greater early-life exposure to sugar-rich diet experienced higher prevalence in chronic inflammation, diabetes, elevated cholesterol, or arthritis as adults. Though early sugar-rich diet did not affect the likelihood of employment, it decreased post-secondary schooling attendance, and made it less likely to have a skilled occupation, or to accumulate above median wealth. Elevated sugar consumption across lifespan is a possible pathway: the end of rationing increased intake of free sugar and sweets more than five decades into adulthood. This is concerning given that majority of American pregnant women as well as infants and toddlers consume free sugars above recommended amounts.

New Version - Using Lotteries to Attract Deposits 
Journal of Finance - Revision requested
(working paper, 2023)
Abstract: Despite the importance of deposit financing for lending, banks in developing countries struggle to attract deposits. In a randomized experiment across 110 bank branches through-
out Mexico, a lottery incentive based on net monthly deposits caused a 36% increase in the number of accounts opened and a 21% increase in the number of deposits during the lottery
months. Nearly all new accounts (96%) were opened by households previously unbanked at any bank. The temporary two-month incentive had a persistent 2–3 year impact on the flow of deposits and stock of savings, and increased the present value of branch profits by 6

New Version - Mortality from Nestlé’s Marketing of Infant Formula in Low and Middle-Income Countries
(working paper, 2023)
Abstract: Infant formula use has been implicated in millions of infant deaths in low and middle-income countries over the past several decades, but causal evidence of formula’s link with mortality remains elusive. We combine data on over 2.6 million infant births and deaths from the Demographic and Health Surveys (DHS) across 38 countries with data on the timing of Nestlé entrance into country infant formula markets taken from annual investor reports.
Consistent with the hypothesis that formula mixed with unclean water acts as a disease vector, we find that infant mortality increased in households with unclean water sources by 19.5 per thousand births following Nestlé market entrance, but had no effect among other households. This rate is equivalent to a 27% increase in mortality and amounts to about 212,000 thousand excess deaths per year at the peak of the Nestle controversy in 1981

New Version - Making Entrepreneurs: The Returns to Training Youth in Hard versus Soft Business Skills          (working paper, 2023)
Abstract: We study the medium- and long-term impacts of the Skills for Effective Entrepreneurship Development (SEED) program, an innovative in-residence 3-week mini-MBA program for high school students modeled after western business school curricula and adapted to the Ugandan context. The program featured two
separate treatments: the hard skills MBA features a mix of approximately 75% hard skills and 25% soft skills; the soft skills curriculum has the reverse mix. Using data on 4,400 youth from a nationally representative sample in a 3-arm field experiment in Uganda, the 4-year follow-up demonstrated that training
was effective in improving both hard and soft skills, but only the soft skills curriculum was directly linked to improvements in self-efficacy, persuasion, and negotiation. Four years after the intervention, youth in both groups were more likely to start enterprises and more successful in ensuring the survival of their businesses. Businesses started by the treatment group were also of higher quality, with higher likelihoods of formality, of having employees, and of being collaborations with other entrepreneurs. The program also led to better business performance, with significantly higher revenues (SEED hard: 31%, SEED soft: 35.8%) and profits (SEED hard: 25.7%, SEED soft: 31.9%). Nine years after the intervention, the entrepreneurial efforts of the treatment and control groups double: approximately 61% of the sample owns an active business. Although there are no longer effects along the extensive margin of entrepreneurship, ventures led by SEED graduates maintain their edge in terms of quality. They are still more likely to be formal, to have (paid) employees, and to feature partners. They are also larger in scale as proxied by revenues (SEED hard: 50.6%, SEED soft: 36.4%) and have higher profits (SEED hard: 32.0%, SEED
soft: 19%). Both SEED curricula were highly cost effective if benefits are proxied by business earnings. In the medium run, total costs ($ 160.14) are exceeded by 0.8 (1.38) months worth of extra earnings
resulting from attending the SEED hard (soft) program. The business-related net benefits of the two curricula continued to grow over the five years between the two follow-ups, with annualized growth rates of the effect on profits of 25.6% and 8.4% for SEED hard and SEED soft, respectively.

New Version - Promoting Parental Involvement in Schools: Evidence from Two Randomized Experiments
(working paper, 2023)
Abstract: We examine the effects of a parental involvement program in Mexico, which provides parent associations with grants and information. Grants to parent associations did not improve educational outcomes. Information to parent associations reduced disciplinary actions in schools, mainly by increasing parental involvement in schools and changing parenting behavior at home. The divergent results from grants and information are partly explained by significant changes in perceptions of trust between parents and teachers.

Encouraging Preventative Care to Manage Chronic Disease at Scale 
(working paper, 2023)
Abstract: We study how reminding high-risk patients with chronic disease of their upcoming primary care appointments impacts their health care and behaviors. We leverage a natural experiment
in Chile’s public healthcare system that sent reminders before preventative care appointments to over 300,000 patients with type 2 diabetes and hypertension across 315 public primary care
clinics between 2015-2018 in Chile. Employing both a difference-in-difference and instrumental variable approach on national administrative patient-level data, we show that reminders
increased preventative care visits, which led to more health screenings and improved medication adherence. In this at-scale program, we find substantial variation in implementation fidelity
across clinics, which, once accounted for increases our estimates by over a third. Reminders also increased hospitalizations and a reduced in-hospital mortality, suggesting an improvement in
timely care-seeking behavior among high-risk patients. Our findings inform healthcare settings where patients must first visit their primary care provider for approval before undergoing tests,
receiving medication prescriptions, or getting referrals to other specialists. Through intervening at the first step in the cascade of care, we find that a simple intervention like reminders can
have large and meaningful downstream effects.


Effect of the Jamaica Early Childhood Stimulation Intervention on Labor Market Outcomes at Age 31
(working paper, appendicies, 2022)
Abstract: We report the labor market effects of the Jamaica Early Childhood Stimulation intervention at age 31. Implemented in 1987-1989, treatment consisted of a two-year home-based intervention designed to improve nutrition and the quality of mother-child interactions to foster cognitive, language and psycho-social skills. The original sample is 127 stunted children between 9 and 24 months old. Our study is able to track and interview 75% of the original sample 30 years after the intervention. We find large and statistically significant effects on income and schooling; the treatment group had 43% higher hourly wages and 37% higher earnings than the control group.