Evidence matters to policymaking

The challenge for the Modi government is to evaluate what works in development programming before making large spending decisions

Governments in India have always launched big-ticket social development programmes with ambitious goals. The Narendra Modi government is no exception. Swachh Bharat Abhiyan aims to make India open defecation-free by 2019. The Pradhan Mantri Jan Dhan Yojana’s financial inclusion programme set a target of opening 10 crore bank accounts by January 26, 2015. But are ambitious targets enough for achieving desired outcomes on the ground?

Most governments across the world run social development programmes that focus on increasing supply. So, the emphasis is on building toilets, creating bank accounts, providing loans and so on. For politicians, development needs to be visible for it to count as an achievement. It is harder to assess if these programmes are really achieving the results that matter and whether they are improving the lives of the poor. The focus on supply, while ignoring demand, is an approach that has often failed. For example, will toilets be used if they are built? The Ministry of Drinking Water and Sanitation is now asking people to post pictures of unused toilets on its website. Low usage of toilets is clearly an issue. Are the toilets that are being built improving sanitation and child health by reducing diarrhoea — a highly preventable condition that is the leading cause of death of children under five? The answers to these questions are not forthcoming. Nor can they be without rigorous evaluations.

Need for evaluations

Flagship development programmes in India have not been designed or modified based on evidence from evaluations. There has been a continuity of some social programmes over decades, although governments may have given them different names and made a few changes in design. Swachh Bharat Abhiyan builds on Nirmal Bharat Abhiyan (2012), the Total Sanitation Campaign (1999) and the Central Rural Sanitation Programme (1986). But politics rather than evidence is the key driver for programmes getting closed, changed or renamed.

Synthesised evidence from impact evaluations of water and sanitation programmes in low and middle-income countries shows that building toilets is not enough for improving sanitation. An impact evaluation of the Total Sanitation Campaign in Odisha in 2010 showed that demand-oriented programmes raised latrine use from just six per cent to 35 per cent. This seems like a useful increase, but not one that made any difference to people’s health, and left 65 per cent still not using the new facility. A more recent 3ie-supported impact evaluation in Odisha also showed a substantial increase in latrine coverage at the village level. But this increase was also insufficient to reduce child diarrhoea, most likely because the villages in the study were not open defecation-free. It is well established that open defecation supports various transmission channels by which children consume faecal matter: direct contact or indirectly from pets or other animals, and through flies crawling on both excrement and the food children may eat. According to the World Health Organization and UNICEF, diarrhoea causes the deaths of more than 7,50,000 children under the age of five every year. A study published in the Lancetin 2012 reports that in India alone, an average of 2, 12,000 children under the age of five die of diarrhoea every year.

Global evaluation evidence on sanitation interventions highlights the need to emphasise behavioural factors for promoting the use and sustained adoption of water, sanitation and hygiene technologies. Making people pay even a very small fee has put them off from using water treatment technologies. Changing behaviour can take years and decades. Sanitation programmes in India need to take this evidence into account.

Evidence in financial inclusion

In the same vein, financial inclusion programmes also need to consider evaluation evidence. According to a 2014 report brought out jointly by the World Bank, the Better than Cash Alliance and the Bill & Melinda Gates Foundation, over 2.5 billion adults around the world are ‘unbanked’. Evidence shows that just opening bank accounts in someone’s name, as the Jan Dhan programme is doing, doesn’t work. Many people don’t even access the account to take out the initial deposit placed there for them. Jan Dhan’s goal of opening accounts for 10 crore poor people is not enough by itself. There are many reasons people don’t use bank accounts: lack of trust in financial institutions, distance to the bank, illiteracy and reliance on informal and traditional banking sources. In addition to tackling these issues, there may be a need for complementary activities to ensure that the poor can benefit from having bank accounts.

“Plenty of inspiration can be found for evidence-based policymaking from Latin American countries such as Mexico, which is known for its system of evaluating social programmes”

The challenge for the Modi government now is to evaluate what works in development programming before making large spending decisions. This would mean putting in place rigorous evaluations of the new programmes it is launching. Existing programmes need to also be evaluated before major decisions are made about their future. Building a culture of evidence-informed policymaking would also require moving towards institutionalising evaluation within government.

Randomised controlled trials

There is plenty of inspiration that can be found for this from other countries, particularly those in Latin America. Mexico is widely known in the international development community for its system of evaluating social programmes. In the mid-90s, the Mexican government launched Progresa, one of the world’s first conditional cash transfer programmes. Progresa handed out cash to poor families on the condition that they send their children to school and young mothers seek antenatal care. Giving cash directly to the poor was not the only innovative aspect of Progresa. The government put in place a rigorous randomised evaluation of the programme. Just as clinical trials have informed evidence-based medicine, randomised controlled trials of social programmes provide rigorous evidence of whether programmes are working or not.

The impact evaluation showed that Progresa increased school enrolments and use of health services and reduced poverty. Given the strong evidence on the programme’s impact, the new government that came into power soon after the evaluation decided to retain the programme — although they renamed it as Opportunidades. It continues to survive today as one of the world’s largest social programmes. Mexico has since also set up CONEVAL, the National Council for the Evaluation of Social Development Policy. It is a governmental body that has the mandate to independently evaluate government programmes and policies. Based on the evaluations, programmes are scaled up, closed down or modified.

India can have the same reputation as Mexico. It is home to many bold social development programmes. But the failure to evaluate them is a lost opportunity. It is a lost opportunity for governments to achieve recognition for what they are doing. A lost opportunity for those working for social programmes to know they are making a difference. Most importantly, it is a lost opportunity for the millions of poor to have better lives.