• Stories
• Kenya
• Evaluating the Millennium Villages

• Share this Story
• Comment
• Print
• Reprint

Evaluating the Millennium Villages

I met with Gabriel Demombynes, a World Bank economist stationed here in Nairobi, this evening. Recently, he pointed out that there was a fundamental flaw in the way in which progress in the Millennium Villages is evaluated. Essentially, the evaluation looked at changes in measures of performance over time, but it did not make comparisons to any type of baseline.

For example, suppose an evaluation finds that the percentage of people engaging in a negative behavior falls from 20 percent to 10 percent. If the percentage of people doing this falls, that's a good thing, right? Yes, but it doesn't mean that the Villages should necessarily pat themselves on the back.

To see why, suppose that in other comparative villages--which are not Millennium Villages--the percentage also falls from 20 percent to 10 percent, or perhaps even lower. Now things don't look as good. Measured against the baseline, performance has stayed the same, or even worsened.

That's essentially what Demombynes and his co-authors showed in their academic paper on this topic. The measures for Kisumu in the report are very highly correlated with measures at the national level.

The response from Village supporters such as Jeff Sachs was to say that the evaluation wasn't the actual evaluation, and the real one would come soon. But when the second evaluation arrived, its major finding--that infant mortality had fallen significantly in the Villages--was subject to similar questions.

Gabriel, via an email, points to further discussions of these points:

"Have a great time in Kisumu. An hour or so away is Sauri, the first Millennium Village. I've had a long back-and-forth with Jeff Sachs about the project's evaluation. Here's an early blog post I wrote about my visit to Sauri. More recently I wrote this blog post which led to the MV project retracting its major finding in a paper in The Lancet; here's the retraction on the MVP website, the retraction in The Lancet, a letter from The Lancet editors, and my letter with others published in The Lancet."

Demombynes also recommended these related stories:

• "Millennium Villages Project forced to correct Lancet paper on foreign aid as leader leaves team," Retraction Watch

• "The best story in development," The Economist

• "Millennium bugs," The Economist

• "Millennium Villages Project," Give Well

Being here emphasizes how big the problem is, and how important it is to get these measures correct.

• Reader Comments
• Post a Comment