The use of randomized control trials (RCTs) for evaluating development programs has been much debated with strong arguments on both sides of the coin.  This debate is often couched in a larger discussion about the importance of impact evaluations, which can take the form of RCTs or other quasi-experimental designs that establish some sort of counterfactual. While there is considerable difference of opinion on whether or not experimental methods should be used in evaluation, there is no doubt that impact evaluations can be costly and difficult to implement.  So, from a practitioner’s perspective, the more critical question that development organizations should be asking is when, or under what conditions, should IEs be considered.

 

To attack this hot topic in international development, we asked a diverse group of practitioners to tell us about their experiences and perspectives on some key criteria that an organization should think through when considering an IE.  First, though, we identified five of the most important issues and outlined them below. Keep in mind that these issues are largely related to resource constraints and logistical issues; any organization looking at an IE should also consider the myriad technical questions that researchers wrestle with in designing rigorous evaluations.

 

1. Is there a clear research question?  Defining a clear research question is instrumental to identifying the respondents, techniques for sampling, options for evaluation methodology, and data collection instruments. Just as importantly, if you don’t have a clear research question, you’re not likely to come up with a convincing “answer” in the end. This question is important in deciding to undertake any serious evaluation, but it becomes especially key when determining whether or not to expend significant resources and time on an IE.
2. What is the intervention and how does it address the development problem? Understanding the theoretical framework (or theory of change) on how the intervention will have an impact and knowing the mechanisms by which the intervention will have an impact helps to decide on the feasibility of, and inform the process for, an impact evaluation.
3. Will the project reach enough of the “target audience” to achieve sufficient sample size for an IE? A well-designed IE often requires a large number of “treatment units” to achieve the necessary power to make statistical inferences about outcomes. Thus, a good time to look at an IE is when a program or organization is about to go through a significant expansion. Often, these expansions result in a much larger population base than in a small pilot or a more mature, stable project.
4. What is the organization’s human resource capacity for implementing an impact evaluation? Does the organization have the staff required to plan and oversee a complex experiment and related evaluation? If not, does the organization have the resources to bring in third-party researchers? What will be the division of labor in implementing the impact evaluation?
5. Does the organization have the logistical and financial capacity to effectively implement an impact evaluation? Once an organization decides that an impact evaluation makes sense, it must consider logistical issues to implementation.  Some questions that must be considered are: Does the intervention have sufficient resources to achieve scale and reach full coverage of all eligible beneficiaries? Are resources available from the beginning to take the pilot to scale if/when it is successful?

 

Our guest bloggers will take the next several blog posts on this site to unpack these issues and provide their own stories on the importance of each criterion in deciding whether or not to pursue RCTs or other types of impact evaluation. Stay tuned for the next series of posts where we’ll be hearing from a mix of development practitioners, from academics to NGO workers to independent evaluators on their experiences in determining when an IE should be considered.

 

In the meantime, please send us your comments directly on this site or you can email us at info@designmeasurechange.com.

We don’t often get very technical in this blog—we like to talk about big ideas when we can—but sometimes the work we do in the field lends itself so well to a quick post that we just can’t help ourselves. Our topic of conversation this time around is about in-field randomization.

 

Evaluators use randomization to ensure a representative sample of a pre-determined population when we cannot survey the entire population of interest.  Often, designing the randomization is the easy part; actually implementing it can be much trickier, especially in situations where you don’t have a comprehensive listing of the population (much less any reliable data) prior to data collection.  This can be common in small, rural communities where local population records are unavailable, outdated, or nonexistent.  Through our work and studying the work of others, though, we’ve found many ways of creatively conducting in-field randomizations.

 

One simple method that has proved particularly successful in these settings requires just two tools:  a pencil and a random number! Enumerators use the pencil to point them in the “right” (random) direction and the random number determines the number of houses to pass before the next survey.  The method goes like this:

1. Each enumerator should start in a completely different part of the area to be covered. For example, one might start at the extreme north border of the community, another at the extreme south, and the third in a central location.
2. Each enumerator starts facing a house in a location where they were dropped off. Later, the “starting point” will be the house that the enumerator just interviewed.
3. Before the enumerator starts walking, she looks at a pre-generated random number sheet, crosses off the last survey/random number/unique ID that she finished, and notes the next random number.  This is the number of the house that the enumerator will interview for the next household. For example, if the random number is 4, the enumerator will pass 3 houses and stop at the 4^th house for the interview. Be sure to count houses on both the left and right side of the road and stop at the 4^th house.

So where does the pencil come in? If the enumerator comes to an intersection, she spins the pencil and walks in the direction (on the road) that most closely resembles the direction that the pencil is pointing. She will walk the number of houses indicated in Step 3 but STOP walking if she arrives at an intersection, at which point she will spin the pencil again to see which direction to continue walking.  She should always pass the same number of houses indicated in Step 3, no matter how many intersections are passed.

 

Of course this method isn’t perfect, but it has worked particularly well for us in small, rural communities and in situations where data on the population of interest just isn’t available.  It is also important to train your enumerators on the pencil and random number method during enumerator training and in the field during the pilot stage.

 

Do you have any successful in-field randomization methods?  If so, please comment directly on this blog or email us at info@designmeasurechange.com.

We often hear development organizations say “we are mission-driven, not research-driven.” This sentiment can be seen as reflecting the tension that exists between implementing organizations, funding agencies, and evaluators (or researchers).  Mission-driven organizations are concerned with delivering services to their beneficiaries; that is, translating funding into inputs, inputs into activities, and ensuring that activities produce useful outputs.  This includes, for example, tracking the number of schools, clinics or roads built, distributing conditional cash transfers, and executing training programs, to list a few examples.  This sort of monitoring data is internal to the development project and allows the implementing organization to manage the project by regularly tracking outputs.

 

Incorporating evaluation, particularly impact evaluation (IE), tends to be viewed as more “research-driven” as it requires collecting data that is often additional and separate from that of an organization’s routine monitoring data.  This is largely because IEs incorporate a counter-factual; IEs estimate causality by determining what would have happened to project beneficiaries if they were not exposed to the project.  This requires identifying a control/comparison group (those who are similar to the project group but who do not receive the intervention) and collecting the same data on them as actual project beneficiaries.  In addition to collecting data that is not a routine part of the organization’s activities, outside evaluators are often involved to carry out the IE.

 

Although there are almost always increased costs and resources involved in IEs, such studies can be very valuable not just to research-driven organizations, but also to mission-driven organizations.   Foremost, the benefits in terms of learning what works and doesn’t is priceless if it means strengthening an organization’s mission. IEs enable an organization to say what works and what doesn’t, and, just as importantly, engage in invaluable learning that can lead to progressive steps towards adjusting program activities so that meaningful and lasting impact can be achieved.  IEs can also offer a detailed picture of what interventions have the greatest impact by leveraging small variations in program design.  For example, IEs can answer important questions such as do adolescent health programs work better if lessons are taught at home, school, or a local clinic? Similar tweaks to program design could also help an organizations determine if health lessons have a greater impact if they are taught with parents or in small groups of similarly aged girls. The bottom line is that IEs can lead to more efficient and effective programming and, in some cases, help an organization avoid poor resource allocation.

 

A case in point is a recent article in The Financial Times, Cash on delivery for the world’s poorest.  The article speaks of a community-driven development (CDD) program called Tuungane (“Let’s Unite) in the Democratic Republic of Congo.  The CDD program involved giving communities cash grants conditional on them coming up with appropriate institutions for deciding how to spend the money.  The intended outcome of CDD was that money would be spent well and accountable local institutions would be developed.  A randomized control trial (RCT) was conducted to evaluate the effectiveness of CDD on how well money was being spent in the communities.  The results of RCT failed to show any extra benefits for the CDD program as the cash was reasonably well spent whether or not a community had been exposed to CDD.

 

Despite the results, the fact that CDD was rigorously evaluated allowed the funders to realize what worked and what didn’t. Without such an evaluation, money could have kept going into a program that may not have resulted in any greater outcomes. The benefit from the IE is that it produced valuable information that would not have been realized without a rigorous evaluation. Now the program can adjust its activities to strengthen future programs instead of continuing to spend resources where they are not needed, or where the benefits will not be greatest.

 

Rigorous impact evaluations aren’t just for academics or research-focused organizations.  Mission-driven organizations have a lot to gain from implementing rigorous evaluations into their programs so that they can continue to learn from their current programming and therefore, over time, deliver the greatest benefits to the most people.  As the FT article points out, the Tuungane program should be applauded for its work. It is far too common for development projects to report nothing more than their outputs: how much is spent and on what.  The international development field needs to see more outcome-focused results, even if those results are unexpected.  The information gained provides valuable learning for not only the implementing organization but also the broader development community, especially organizations working in similar fields.

In a prior post, we featured a conversation with Joel Selanikio, creator of EpiSurveyor (soon to-be Magpi), on the challenges and future of mobile technology in international development.  Now for a different perspective, we direct our attention to George Njuguna, a star member of the EpiSurveyor support team in Kenya.  As we mentioned in our post with Joel Selanikio, we used EpiSurveyor extensively in the last year, collecting data on a number of evaluation projects from El Salvador to Vietnam and in range of fields, from gender-based violence to village savings and loans.  Throughout this work, George provided us with professional and timely support when needed, and he was an integral part of our success in the field.

 

Since George, and the majority of EpiSurveyor’s team is based in Kenya, we sent him over a few questions and here are this responses:

______________________

How long have you been with DataDyne and what convinced you to become part of the team?

I started working for DataDyne in 2009 because I saw the potential for mobile technology as a tool that would help many people collect data cheaply.

 

What is the most interesting or innovative aspect of your work right now with DataDyne?

I would say that the interesting part of my work right now is working on our new version of the system called Magpi.  I’m very excited about it since we will be improving on the overall performance, fixing issues, and adding new features that users have requested.

 

What is the best part of working with organizations like ours?

I enjoy the mutually beneficial interaction between DataDyne and organizations like evalû because it helps us to improve the system, which in turn makes the system better for all.

 

What are the biggest challenges with using this sort of technology in the field?

In most regions in developing countries, where EpiSurveyor is used most, the biggest challenge is the availability of a good internet connection. Alternatively, SMS is still expensive for most users who send bulk messages. (Editor’s note: for those who aren’t familiar, there is an SMS feature for uploading data and doing data entry in EpiSurveyor).

 

What do you look forward to with the release of Magpi?

I’m looking forward to the overall improved performance for the system. Magpi will load much faster as compared to EpiSurveyor. There will be other exciting new features such as:

- Better survey forms and logic implementation design (in other words, skips patterns);

- The ability to publish reports, maps, and data so that any user with the new Publish link can view data results without the need to log in or register an account;

- Color coding for maps based on selected options on multiple choice questions;

- The ability to organize, or group, users into teams; this way, you can share a form with a team and give the team a general privilege.

- Users will be able to run Magpi on iPads.

 

What would be your advice to organizations that are thinking about using EpiSurveyor for data collection? How can they ensure success?

Good preparation and training go a long way in ensuring both that the data collection process is easy for the data collectors and to reduce errors out in the field. I’ve found that organizations that implement proper training are the happiest with the system.

______________________

Remember, in our post on EpiSurveyor, we mention that the program and its applications are extremely user friendly.  All of our evaluators are all self-taught users.  We also train field staff and enumerators and have had great success with transferring knowledge and skills! With training this easy, you can devote more time to designing solid instruments and doing thorough field-testing.

 

Thanks, George!  We certainly look forward to the new and improved features in Magpi!

 

If your organization isn’t familiar with EpiSurveyor, or mobile data collection for that matter, now is your chance to check out Magpi.

It’s a new year and we look forward to many exciting evaluation projects in the upcoming months.  While most of our work will be done with field teams we have worked with before, we have learned never to underestimate the amount of preparation work that must be done ahead of time.  As our team prepares for another year of evaluations around the world, we have compiled some tried and trusted preparation tips for a successful trip.

 

First, it is important to get as much deskwork done as possible before your trip, including literature reviews and research.  This reduces time needed in-country and can save a lot of headaches since internet connectivity and research tools are oftentimes more reliable and easily accessible in your home country.

 

Second, make sure communication flows early and is maintained between all parties involved.  This includes your team, the client, and field staff.  At least one month before you travel start communicating with all parties. It’s a good idea to draft a schedule of activities with a tentative timeline and solicit their feedback as a means of revving up communication.  A Gantt chart is a very useful tool here.  We like to have columns for Activity, Participants, and Outcomes to ensure that it is clear what we need, from whom, and when. As we get closer to departure date, we draft a daily agenda for the field teams. Some important activities that should be included in the preparation communication include:

• Recruitment of enumerators and translators (if necessary)
• Procurement of materials and data collection tools
• Identification of pilot communities and participants

 

It’d be ideal to list a main point of contact for all activities.

 

When you’re traveling and on assignment, it’s important to maintain strong communication with your colleagues at home.  For evalû, our team members are spread out around the world so virtual communication is key at all times. Email is our most important communication tool, so we make sure we keep emails brief when possible, send weekly updates, and even use codes to alert important emails.  For example, we’ll flag very urgent emails (that is, drop everything and read this right away emails) with “!!!” in the subject line. For less important, but still important emails, we’ll code the subject with “!!” (this may mean prioritize/read first this email over the others). We all are on different time zones, so these codes are very important in ensuring timely communication and feedback. Decide with your team what codes may work best to flag urgent emails.

 

Now, for last minute packing tips:

• International adapters for your electronics
• Extra batteries
• Books (or a Kindle)
• Sunscreen and any other toiletries because they are either very hard to find or very expensive.
• A cell phone that is unlocked and works on a tri-band (GSM) network. Most times you should be able to buy a cheap cell phone in the destination country, but check first with the field team.
• Check out Chris Blattman’s nice (long) list of items to considering packing

 

Good luck in the new year!  If you have any travel-related preparation tips, please comment here or send us an email to info@designmeasurechange.com.

In our last post, we introduced our primary data collection tool, EpiSuryeyor, developed by

the social enterprise called DataDyne. EpiSurveyor is critical to our work because we rely on it for fast, reliable, and easy mobile data collection in all our evaluations. Because we work largely on international development projects, simplicity and mobility are two critical elements of our success, and EpiSurveyor has met both needs time and time again. Given EpiSurveyor’s importance to our work, we wanted to spotlight two (out of a total staff size of nine) key members of the DataDyne team. In this first post, we’ll meet Dr. Joel Selanikio, a pediatrician, former Wall Street computer consultant, epidemiologist, and creator of EpiSurveyor. Our next post will feature George Njuguna, a member of DataDyne’s support team in Kenya, and one of evalû’s favorite people!

 

Fatima, our evaluation specialist, met with Joel in DC and asked about his big ideas for the future of EpiSuryeyor (soon to be Magpi) and mobile technology.

 

__________________

DataDyne is coming up on its 10 year anniversary in 2013. What has been your greatest accomplishment thus far?

 

In the field of technology for development, technology is typically developed for one project at a time, and “scale” means expanding from one project site to three.  That approach is obviously not capable of scaling technology enough to give it to everybody who needs it.  From a scale perspective, ICT4D (Information and Communication Technology for Development) has been a huge failure.

 

At DataDyne we are showing ICT4D how to build real software that is really useful, but to do it at real scale and for very little money, and with very low total cost per implementation.  In a field where it’s common to spend hundreds of thousands of dollars per implementation on mobile technology, we’re showing people a much better, faster, and dramatically less expensive way.   That we’re making good headway, despite the inertia and vested interests in the system, is a great accomplishment and I’m very, very proud of our team.

 

 

That accomplishment, of course, was built on lots of little ones: Back in the late 90s I was the first to use mobile electronic devices (PDAs) to collect health data in developing countries — collecting nutritional data among Burmese refugees in Thailand.  In the 2000s, EpiSurveyor was the first do-it-yourself mobile electronic data collection kit for non-programmers in the development space.  EpiSurveyor was later the first cloud-based application in international development.  Finally, EpiSurveyor became the first development technology to do use a “freemium”pricing model (where 1% of users pay and 99% don’t).

 

And now DataDyne is the only tech-for-development organizations entirely supported by paying users, rather than grants or consulting revenues.  That’s not a bad list of firsts.

 

 

What is your vision for Magpi in the next 10 years?

 

I think that the technology is going to change tremendously in the next 10 years and it will scale even more computing power down to the poor.  This opens up the possibilities of moving away, at least in part, from the model where trained interviewers visit households to ask questions to a model where the people in the households are able to send information to the system; or the household itself, or the refrigerator, etc, will be sending information in.  We want Magpi to continue to be the kind of tool that lets non-programmers, non-technical people, participate in the information explosion that will come with that.

 

 

I’m going to go out on a limb and guess that you believe that mobile technology will revolutionize the data collection field (if it hasn’t already)?  Would you agree? If so, how will Magpi contribute to that revolution?

 

Right now I’d guess that more than 95% of health data in poor countries is still collected on paper. As we have focused on for ten years, we intend to make it so easy, and so inexpensive, to get better data and faster data, and to get it electronically, thatten years from now we’ll be collecting 95% electronically and in real-time. We want to move from paper as the norm to real-time electronic as the norm.

 

 

Let’s make a big prediction.  Assuming that mobile data collection is already a “big thing,” what will be the next “big thing” in data collection? 

 

As I mentioned,I think that we’ll see a lot of movement away from the “data collectors go to the field” model to one where “the field sends data to us.”  It won’t completely replace the older approach — but it will augment it, and that’s going to save a lot of effort.

 

 

Do you still meet with resistance from any parties in the field to using mobile devices for data collection? Where do you think this resistance comes from?

 

I think any new system meets resistance, sometimes just from inertia and sometimes because it threatens someone’s job.

In international development right now, there are lots of organizations paying for programmers for data collection that in the very near future will not need those programmers at all.  As you can imagine, some of those programmers are helping to lead that change, but some are not very happy about it at all.

 

Likewise, many organizations in development are incentivized to spend money, not to save it, so they are not particularly interested in low-cost solutions. They would rather pay $1 million for a perfect, custom-built solution than to pay $1,000 for a solution that provides 75% of that functionality. It’s a strange dynamic but I see it all the time.

 

 

You wrote in an article that “The question we should be asking ourselves is not  ‘how can we buy, and support, and supply electricity for, a laptop for every schoolteacher’ (much less every schoolchild), but rather  ‘what mobile software can we write that would really add value for a schoolteacher (or student, or health worker, or businessperson) and that could run on the computer they already have in their pocket?’ What do you think is the next big development problem that mobile technology will begin to address?

 

You should ask yourself “what useful software did I use this week on my computer?”  Then think about which of those software functions are currently available to a user of a $20 mobile phone.  Almost none of them, right?  Well, someday they all will be.  The next big developments will come from those that (1) understand that, and (2) are determined to build some of that technology.  This field is in its infancy.

 

 

Have you learned any lessons deploying Episurveyor in developing countries that you think data collection and evaluation professionals in rich countries should heed?

 

That’s about 5 books worth of lessons!

[In the upcoming version of Magpi, Joel mentioned that they took 40 of the top issues and improved upon them.]

 

 

If you could have any superpower in the world, what would it be?

 

The ability to function without sleeping.  :-)

__________________

 

Don’t we all wish for that! If your organization isn’t familiar with EpiSurveyor, or mobile data collection for that matter, now is your chance to check out Magpi.  Like Joel said, mobile data collection has never been easier!

 

Our next post will feature, George Njuguna, a member of DataDyne’s support team.  Stay tuned for another perspective on the future of mobile technology and ICT4D!

 

Using paper surveys and for data collection can create big hassles and inefficiencies.  In addition to spending a lot of valuable time on data entry, transferring data from paper surveys to a database leaves a lot of room for errors.  And, really, who wants to carry, organize, and track hundreds of paper surveys? Well, thanks to readily available mobile technology, there’s no reason to use paper anymore.

 

We’ve used EpiSurveyor, an open access mobile technology tool, for all our quantitative data collection over the last year.  For those who aren’t familiar with Episurveyor, it’s a mobile phone and web-based data collection system, created by Georgetown University pediatrician Dr. Joel Selanikio, developed in Kenya, and used by hundreds of organizations in over 170 countries in international development and other sectors.

 

Within the last year, we’ve learned and mastered the ins and outs of EpiSurveyor, and we’d like to share our experiences.

 

The Good

• It’s free! Well, there is a free and paid version of EpiSurveyor. The free version includes basic functions, but is limited to 20 survey forms, 100 questions per survey, and 5,000 data uploads per year. If you need more power or advanced functions such as SMS data entry and unlimited storage online, there are paid versions of Episurveyor that are still more cost-effective than other commercial programs.  Check out the EpiSurveyor pricing page.
• The program and its applications are extremely user friendly.  Our evaluators are all self-taught users.  Further, we train field staff and enumerators and have had great success with transferring knowledge and skills! With training this easy, you can devote more time to designing solid instruments and doing thorough field testing.
• Cell phones are found everywhere, even in the poorest of countries, and are relatively cheap.  This takes care of the hardware requirement.  Remember, the software is free, (see point 1 above)! In the words of Dr. Selanikio, cell phones work for data collection because they are “the most successful tech product in the developing world since the radio.”
  • Managing data collection and oversight is a cinch since data can be viewed in real time (if there is a Internet network available in the field).  Every time an enumerator completes a survey, he or she can save the completed survey and then “send” it (if there’s a working network) to the EpiSurveyor server. Data or project managers can view the submitted data in their EpiSurveyor account as it’s being sent from anywhere in the world!
  • The GPS function allows data and project managers to track geographic data, follow enumerators, and make sure the project and evaluation target area is being adequately covered.
  • EpiSurveyor support is always helpful!  When we have had issues come up, the EpiSurveyor support team is always quick to help and resolve our problems.  They are so helpful, in fact, that two members of the DataDyne team have happily agreed to be interviewed for our next post.  Stay tuned for our Q&A session with Joel Selanikio (founder) and George Njuguna (Tech Support)!

 

The (not so) Bad

• Although technology is supposed to make things easy, sometimes it can make things harder (at least in the short-term).  If Internet connection is slow, it can be a pain to download survey forms on the phone. This, of course, is not by fault of EpiSurveyor, and we hear that system improvements set for rollout in 2013 will make it even easier to download survey forms.
• Some EpiSurveyor compatible phones are easier to use than others.  We had trouble using the Nokia c3-00. However, we’ve had the most success using Android phones.
• “Missing data.” On a few occasions data that was “sent” from the phones to the server did not appear on the EpiSurveyor website. “Missing data” is put in quotations because, although the data was still stored in the phone, it did not appear in the EpiSurveyor account despite having been sent.  Tech Support was quick to resolve our issues and improvements are constantly being made to make EpiSurveyor a better product.
• With any sort of change, there are always people who will be resistant to change. Fortunately, in our experience, EpiSurveyor is so easy to use and understand that people oftentimes get more excited than hesitant for the new technology.

 

The Ugly

• There’s nothing ugly about a free user-friendly open access mobile technology tool for data collection in developing countries. We love using EpiSurveyor!

 

 

Have you used EpiSurveyor or another mobile data collection tool?  If so, tell us about it here on our blog.  If you would like to hear more about how your organization can use EpiSurveyor in your evaluation efforts, contact us directly at info@designmeasurechange.com.

In our last post, we discussed how we created and used a household listing to form the sampling frame of child laborers in India, a largely hidden and hard-to-reach population. However, household listing require significant financial resources, time, and cooperation from a local partner, all of which can be in short supply when you’re working on a development program! Below, we outline other commonly used methods for accessing hidden and hard-to-reach populations.

 

Snowball sampling

How it works:  This is a technique that relies on referrals from individuals contacted early in the process  to generate “leads” for additional respondents. For example, we identify “Jane” as a child laborer.  After completing the survey, we ask Jane to help us by identifying other children who she knows are out of school or working. Jane identifies Elizabeth and Mary.  We would then complete surveys with Elizabeth (who identifies Paula and Kelly) and Mary (who identifies Jessica and Karen).  So, in the same way that a snowball starts small but gets bigger as it rolls down a hill, your sample gets larger as your respondents help you find more people like them.

 

Check out this evaluation of a child labor project in Egypt where snowball sampling was used to locate children engaged in domestic work.

 

Strengths: Snowball sampling is cheap, simple, cost-efficient, and requires little planning and a smaller workforce than some other methods. Use this technique if you feel initial respondents would be comfortable referring you to others in the same population.

 

Respondent Driven Sampling (RDS)

How it works: This technique is similar to snowballing, but it relies on respondents—called “seeds”—to do the recruiting, rather than researchers.  Seeds receive a set number of uniquely coded coupons, which they give to their recruits to redeem at a fixed interview location within a set period of time.  Usually the number of coupons given out to seeds is fixed to allow for equal recruitment of peers per seed.  Coupons also allow researchers to link individual recruiters with their recruits, allowing for greater social network analysis. As an example, Markus and George are drug users and they were selected by researchers to initiate the recruitment process. Each man each received five uniquely coded coupons to give to their peers, who are also drug users. On each coupon, the study name, locations where they could participate, and a brief explanation was printed. Coupons linked the recruits to either Markus or George (the seeds) and ensured that no one “seed” had a greater network of peers than others, which would lead to bias.  Researchers can also use the coupons to randomly select among the sample of the recruits if they passed out more than the required sample size in coupons.

 

Check out this study of injection drug users in Tijuana and Ciudad Juarez, Mexico where researchers used RDS to rapidly recruit participants from hard-to-reach populations. More information on RDS can also be found in this methodological survey and in this FAQ from Pop Council.

 

Strengths: RDS has been used for recruiting hard-to-reach or hidden populations, such as drug users, men who have sex with men, or commercial sex workers. Coupons ensure that a sample size is met (i.e. coupons are given out until the required samples size is met).  Coupons also remove selection bias of the survey staff, and a coupon quota (number of recruits or peers that a seed can recruit into the study) reduces biases associated with the over-representation of participants with larger networks.

 

Time Space Sampling (TSS)

How it works: This technique recruits respondents in places or at times where they are likely to gather and asks them about their experiences in that place.  In a review of the time-space methodology involving young Latino men who have sex with men, researchers achieved higher participation rates of gay men at special events and gay venues compared with non-gay venues. Another study used time-space sampling to recruit 18- to 29-year-old club-going young adults into a Club Drugs and Health Project.  As an example, researchers randomly selected venues from a list of dance clubs and bars/lounges as well as special events throughout a city and each weekend recruitments teams were randomly sent to assigned venues.

 

Strengths: Most studies are successful in their use of TSS when there is a clearly defined venue for the population of interest, such as known gay venues or clubs where drug users frequent.  TSS may not be the best option for hidden or hard-to-reach populations such as child laborers, as it may be more difficult to identify a “typical” space for these groups.  An advantage of TSS is that it allows for rigorous planning of sampling events.  This allows for the recruitment of a probabilistic sample of time/space visits and results can be generalized to the studied population who attended venues included in the sample frames.

 

It can be daunting to think about collecting accurate data from populations who are tough to find and reach. But we hope our last two blog posts have given you some ideas about how you can be successful despite the challenges we all face in the field.

 

Have you had any successes, challenges, or tips on using any of these methods?  Tell us about it here or contact us directly at info@designmeasurechange.com.

When collecting data for an evaluation, we face a number of barriers in identifying and sampling hard-to-reach populations, including society’s lack of tolerance for these groups, social stigma, concerns for confidentiality, fear of exposure, and threats to security. Janice Penrod has written a great article on a number of these issues; we’ve pulled from her writings and learned some lessons of our own while working in the field with these populations.  In the next two blog posts, we describe some of our own experiences with these challenges while working on an evaluation of a child labor project in Andhra Pradesh, India. We also spend some time talking about the strategies we used to overcome the challenges we faced during the evaluation.

 

Despite the fact that India has one of the highest incidences of child laborers in the world, this subset of the population is largely hidden or hard to reach.  This presents considerable challenges for data collection when names and contact information are either non-existent or inaccessible.

 

One of the difficulties we faced seemed at first glance to be an easy step in the evaluation process: defining the population. In other words, what does it mean to be a child laborer? To ensure consistency with official figures, we used Andhra Pradesh’s definition of child laborers as any child under age 14 who is out of school. Definitions certainly vary across geographies and countries, and even within countries, but this definition made the most sense in our case.

 

One of the other challenges evaluators face in these situations is biased information from those who are asked to identify child laborers, or even from the data collectors themselves. Additionally, there may be ethical issues when working with these populations, such as unintentionally exposing respondent identity through the description of the child laborers.  For example, households or employers may not admit to having “child laborers” if asked outright.  In our case, instead of asking about child laborers directly, we asked about children living in the household and how many of them were in school. Of those who were not in school, we asked what they currently do.

 

Despite the challenges above, there are ways to successfully manage data collection for such populations.  We were fortunate enough to enlist the help of the project’s implementing partners to compile a list of households in the project target areas. As is true in many cases, the implementers had the strongest connections with the community, and we leveraged this familiarity to gather an accurate and complete listing. The listing enabled us to estimate how many households in the target area have children who are “out-of-school.” From this, we were able to form the sampling frame and estimate sample sizes, and enumerators were given a randomized subset of the listing to locate and survey households with identified child laborers.

 

Although we explored many other commonly used methodologies for accessing hidden and hard-to-reach populations, we settled on the listing option since the project had the available resources and, importantly, the cooperation of the implementing partners.  Since listings can be both expensive and time consuming (which is why this option is not always feasible), it is important to coordinate with the field teams and implementing partners well in advance of the evaluation.  Also, as a final practical tip, make sure to leave a few days for data verification checks before utilizing the listing for sampling estimates and enumeration.

 

Stay tuned for Part 2 of this blog, where we discuss less resource-intensive methodologies for sampling hidden and hard-to-reach populations.

 

Do you have a similar experience with reaching hidden or hard-to-reach populations?  We want to hear from you.  Please tell us about it by leaving a comment on this blog or contacting us directly at info@designmeasurechange.com.

As evaluators, we face two big questions. First, how do we know if an intervention is working? Second, how can we appropriately measure outcomes? In a recent project, we were tasked with designing a qualitative instrument to complement quantitative data around the sensitive topic of gender-based violence.

 

We approached this challenge by designing a focus group discussion (FGD) protocol informed by an article on the “Participatory Ranking Method” (PRM) by Lindsay Stark, Alastair Ager, Mike Wessells & Neil Boothby. PRM uses various activities to elicit local understanding of key indicators in a given topic (in this case, reintegration of child soldiers). PRM acknowledges project beneficiaries as experts and recognizes that the local community is well situated to identify metrics and measure progress.  The researchers used PRM as a way to incorporate local perspectives in constructing research instruments and identifying indicators.

 

Our team was particularly interested in using PRM in our evaluation of the gender-based violence project because it allowed for the selection of indicators that are meaningful to the project’s local beneficiaries–adolescent girls–and reflective of the concepts they find useful when tracking their own progress. Importantly, the project we were evaluating was in a community where gender-based violence pervades the culture. This meant that it may not be possible to see any kind of impact in the short or medium term, as some of the beneficiaries may not be aware of their rights or even know what gender-based violence is.

 

This is how we used PRM in our FGD design:

• We started with a participatory activity to gauge local perspectives and to understand which social practices are considered more or less acceptable in the community. In our case, we asked participants what gender-based violence meant to them.
• To facilitate ranking, we showed a series of cards labeled with different kinds of social practices (in our case: Shout; Insult, Threaten, Push, Hit, Beat, Kill) and had participants order them from the most to the least acceptable, asking them to explain their decisions.  In other cases, you may want to have participants free-list social practices that are common in their communities and then rank-order them.
• We included an open-ended discussion to understand which social practices are acceptable in different relational and social contexts.

 

By using the PRM method, you can promote the measurement of progress grounded in culturally appropriate language and actions and allow evaluators and program teams to understand which social practices are common or acceptable in a community. Having a set of participant-created indicators also allows for a much easier way of tracking progress over time for sensitive topics such as knowledge of violence.

 

 

We would love to hear your thoughts on using PRM or any other methods for creating useful, relevant indicators. Please share your stories on challenges, solutions, and experiences designing FGDs for sensitive topics by leaving a comment on our blog or contacting us directly at info@designmeasurechange.com.