Methods of Measuring the Impacts of Social Policy in Political, Economic and Social Dimensions

Over the past decade the world has seen a sea change in the role of evidence informing social policy design and implementation. In the social protection sector in particular, rigorous quantitative impact assessments have changed how policy analysts and implementers measure success. Mainstream evaluations increasingly rely on experimental approaches, and sometimes quasi-experiments, requiring important compromises. Given the complexity of many critical policy questions, evaluation designers often face a trade-off between ensuring the most rigorous attribution of impact and illuminating the critical policy questions that policy makers are asking. An evaluation approach that balances the prerequisites for analytical rigour with the demands of policy makers for relevant answers is required to design and implement more effective social policies and strategies.

This paper reviews current impact assessment methods and builds on Amartya Sen’s framework of comprehensive and culmination outcomes to identify elements of a comprehensive framework that enables a systems approach to the analysis of social policy. Discussing how mainstream evaluation methods have assessed the outcomes of social security systems, the underlying assumptions of these methods and the associated challenges for the realization of comprehensive outcomes, the paper argues that inclusion of the processes, institutions and actors of social policy interventions that importantly affect programme objectives—along with the actually realized outcomes—should be integrated into a comprehensive approach to better inform social policies.

The emerging framework recognizes the value of a cross-sectoral analysis that studies social, political and economic aspects across a variety of dimensions. It recognizes the importance of both short- and long-term analysis within a policy environment driven by a multiplicity of strategic objectives. In particular, the framework explicitly recognizes that the impact of one sectoral intervention on a specific outcome depends critically on the related interventions across a range of sectors. A comprehensive evaluation approach should inform the optimal balancing of multiple interventions to achieve a range of joint outcomes.

This paper is organized as follows: Section 1 introduces the concepts of culmination and comprehensive outcomes, following a framework proposed by Sen. Section 2 discusses the current mainstream impact evaluation methods for social protection. Several key aspects of a comprehensive evaluation that are not present in current mainstream methods are identified. Section 3 discusses in further depth the features of a comprehensive evaluation, and identifies steps that are already being taken toward a comprehensive evaluation. Emphasizing that inclusion of the processes, institutions and actors of social policy interventions that affect programme objectives in important ways—along with the actually realized outcomes—is central to alternative approaches to expand the scope of the assessment, section 4 concludes with the exposition of elements of a framework for comprehensive evaluation, and discusses future challenges and opportunities.

Michael Samson is Director of Research, Economic Policy Research Institute, South Africa. At the time of writing, Sasha van Katwyk, Maarten Fröling, Rumbidzai Ndoro, Cara Meintjes and Bryant Renaud were Fellows at EPRI and Lien Buts was an Intern there.