Using Multiple Measurement Tools

Question: Is it better to measure your impact via surveys among your target group or by conducting a hand-full of in-depth interviews?

Answer: Both (err… trick question)!

There are multiple ways to measure your impact and relying on just one data source or type of data can prevent you from seeing all that you’ve accomplished. For example, survey data can give you a good sense of your impact on the aggregate-level. But if you only use survey data, then you may not be capturing exactly how your efforts have impacted individuals. On the other hand, in-depth interviews can provide you with more detailed contextual information that might help to explain what your data is telling you. But it can be hard (and often not feasible) to get the resources necessary to interview enough people to capture your effect on the aggregate-level.

That’s why we often draw on both quantitative data (e.g., how much, how many, how often, etc.) and qualitative data (e.g., more descriptive) whenever possible. Quantitative data can be really effective in telling you what has happened as a result of your efforts, while qualitative data can provide you with richer information providing deeper insights indicating why or how something happened. 

Consider the Source: Verifying Your Data

If you’ve been keeping up with our blog, you’ve noticed we’ve been busy presenting at various conferences over the past few months and we’ll soon be gearing up to present at the upcoming American Evaluation Association (AEA) conference in October. With all the planning and thought we’ve been putting into ensuring all our data is accurately presented and our sources properly credited… we figured we’d share some data source verification tips, as we often get asked questions about this. With the various amounts of information available to us, it can be easy to stumble upon facts and figures that aren’t very reliable. Sometimes, even the most widely cited statistics can be seriously flawed. So, how do you know that the data you’re using is accurate?
Below are some quick tips:

1)     Check for bias.  Data is only as good as its source, so it’s important to check the original source of your information to make sure that the organization collecting it isn’t biased. Government reports/statistics are typically very reliable, and you can find data on a variety of topics by searching through local, state and national government webpages.

2)     Go to the original source.  Even the most reputable non-governmental organizations can make mistakes when citing official reports. So, when you’ve found a reputable study cited, be sure to find the original report where the information came from and verify that it’s cited correctly. It’s always best to cite the original source when possible.

3)     Try to have multiple sources.  It’s generally a good idea to look for multiple studies/articles on the topic of interest. There are often several organizations that collect similar statistics, and it can be helpful to see whether the organizations found similar results. If various entities are finding the same results you can be more certain of the conclusions. If you notice that findings differ across data sources, then describe the differences, as that can be useful information as well.

4)     Remember to cite your sources!  Be sure to explain to your audience where you got your information -- always remember to cite your sources.