The use of randomized control trials (RCTs) for evaluating development programs has been much debated with strong arguments on both sides of the coin. This debate is often couched in a larger discussion about the importance of impact evaluations, which can take the form of RCTs or other quasi-experimental designs that establish some sort of counterfactual. While there is considerable difference of opinion on whether or not experimental methods should be used in evaluation, there is no doubt that impact evaluations can be costly and difficult to implement. So, from a practitioner’s perspective, the more critical question that development organizations should be asking is when, or under what conditions, should IEs be considered.
To attack this hot topic in international development, we asked a diverse group of practitioners to tell us about their experiences and perspectives on some key criteria that an organization should think through when considering an IE. First, though, we identified five of the most important issues and outlined them below. Keep in mind that these issues are largely related to resource constraints and logistical issues; any organization looking at an IE should also consider the myriad technical questions that researchers wrestle with in designing rigorous evaluations.
- Is there a clear research question? Defining a clear research question is instrumental to identifying the respondents, techniques for sampling, options for evaluation methodology, and data collection instruments. Just as importantly, if you don’t have a clear research question, you’re not likely to come up with a convincing “answer” in the end. This question is important in deciding to undertake any serious evaluation, but it becomes especially key when determining whether or not to expend significant resources and time on an IE.
- What is the intervention and how does it address the development problem? Understanding the theoretical framework (or theory of change) on how the intervention will have an impact and knowing the mechanisms by which the intervention will have an impact helps to decide on the feasibility of, and inform the process for, an impact evaluation.
- Will the project reach enough of the “target audience” to achieve sufficient sample size for an IE? A well-designed IE often requires a large number of “treatment units” to achieve the necessary power to make statistical inferences about outcomes. Thus, a good time to look at an IE is when a program or organization is about to go through a significant expansion. Often, these expansions result in a much larger population base than in a small pilot or a more mature, stable project.
- What is the organization’s human resource capacity for implementing an impact evaluation? Does the organization have the staff required to plan and oversee a complex experiment and related evaluation? If not, does the organization have the resources to bring in third-party researchers? What will be the division of labor in implementing the impact evaluation?
- Does the organization have the logistical and financial capacity to effectively implement an impact evaluation? Once an organization decides that an impact evaluation makes sense, it must consider logistical issues to implementation. Some questions that must be considered are: Does the intervention have sufficient resources to achieve scale and reach full coverage of all eligible beneficiaries? Are resources available from the beginning to take the pilot to scale if/when it is successful?
Our guest bloggers will take the next several blog posts on this site to unpack these issues and provide their own stories on the importance of each criterion in deciding whether or not to pursue RCTs or other types of impact evaluation. Stay tuned for the next series of posts where we’ll be hearing from a mix of development practitioners, from academics to NGO workers to independent evaluators on their experiences in determining when an IE should be considered.
In the meantime, please send us your comments directly on this site or you can email us at email@example.com.