Happy Holidays!

Wow – 2014 sure flew by! As we wrap up the year, all of us at EVALCORP extend our warmest wishes for a joyful holiday season. And wish you a productive and successful 2015!

  -The EVALCORP team

 

Using Empowerment Evaluation to Strategize for the Coming Year

In previous blog posts we’ve talked a little about our evaluation approach and theories of evaluation we draw from. However, we haven’t yet discussed empowerment evaluation - first introduced in the early 90s. The key principle of empowerment evaluation is to enhance the capacity of primary program stakeholders to ensure their program vision/goals are achieved and to improve the overall success of the program. 

One of the components of empowerment evaluation is to reflect upon what has been accomplished, celebrate successes, learn from challenges, and set intentions for next steps. As 2014 comes to an end, this is the perfect time to reflect upon the wins and strategize for the next few months or even the next year. We frequently use this principle to help clients move forward in their projects and foster their capacity for evaluation.

In the coming year, we plan to highlight and delve deeper into the various types of evaluation theories we draw from to inform our approach to evaluation. But in the meantime, if you’d like to learn more about empowerment evaluation, check out the link below – containing some blogs written by the empowerment evaluation guru himself -- David Fetterman.
http://betterevaluation.org/resources/website/empowerment_evaluation_blog

Evaluating Thanksgiving

We can’t believe Thanksgiving is just a few days away -- we are so excited to be celebrating the holiday with friends and family! During this holiday, in particular, we spend time reflecting upon and evaluating all that has happened over the past year and appreciating all that we have to be thankful for. We also spend a considerable amount of time planning, organizing, and anticipation that our thanksgiving dinners go as expected – making Thanksgiving truly feel like an evaluators holiday. On that note…we thought it would be fun to share how you can use really basic evaluation principles and concepts to determine the extent to which Thanksgiving dinner was a success (you know we are creating logic models for our dinners as we speak!).

Every year we hope that our thanksgiving meal is a huge success. But how do we know how successful it actually was? We can start with our overarching evaluation question:  To what extent was Thanksgiving dinner a success?

Activities to ensure your dinner is successful can include:  making a list of what you’ll need to buy to prepare your meal to ensure you don’t forget anything; decorating your home for a festive atmosphere; making sure you have enough seating for all your guests; telling guests in advance if you’d like them to bring anything.

Process measures to assess the extent to which the event was successful can include:  Number of guests invited; number of guests who attended; extent to which your meal went as planned (i.e., did you follow your recipe as expected); challenges encountered in making your meal (e.g., turkey was overcooked – yikes!); and lessons learned for next year.

Sample outcome measures can include:  Number of guests who enjoyed the meal; number of guests who asked for your recipe to make one (or all of your dishes) in the future; extent to which guests had fun at your event; and the extent to which your guests felt welcome at your event. 

Did you know…

Before we end our post – we wanted to provide some fun Thanksgiving facts you can share with friends as you wait for the turkey to be ready!

·       Thanksgiving is also celebrated by Canadians  - on the second Monday in October.

·       The first Thanksgiving was observed in Plymouth in 1621.

·       In 1863, President Abraham Lincoln proclaimed the last Thursday in November as a national day   of Thanksgiving.

·       In 1939, President Franklin D. Roosevelt moved Thanksgiving Day one week earlier to boost the Christmas shopping season.

·       In 1942 Congress ruled that the fourth Thursday in November would be observed as Thanksgiving Day and a federal legal holiday.

           Source:  CNN.com

Wishing you a very Happy and Successful Thanksgiving!

Don't Be Spooked by Evaluation

Are you suddenly panicked when you hear the word evaluation? Does the thought of having to participate in an evaluation make you feel like you’re about to star in a horror movie? Well – we are here to tell you – you don’t have to get spooked by evaluation! 

One of the most common things we hear when meeting with new clients, particularly those new to evaluation, is that they have negative perceptions of the field of evaluation. Often times we are simply called in because our clients had to have an evaluator.  Much of this fear or apprehension stems from misconceptions about the purpose of evaluation and the role of an evaluator. 
Below are some common myths we’ve heard over the years.

•  Evaluators are here to cut our program(s)
•  Evaluators are evil and will make us look bad
•  Evaluators will just judge us, and the work we do
•  Evaluation won’t tell us anything we don’t already know

The fact is there are various types and phases of evaluation, and very rarely do programs ever get cut.  For instance, in addition to conducting outcome evaluations (i.e., changes that occurred as a result of the program) we often work with organizations to conduct process evaluations that help identify how the program is/was carried out including challenges, barriers, and lessons learned. At EVALCORP we take a holistic approach to evaluation meaning that we take into account multiple factors that impact a program (e.g., like political environment, level of funding, staffing changes, etc.) to help explain outcomes achieved. 

Most evaluators aren’t there to make your lives harder but rather to work with you to assess your processes, measure what changes have resulted due to your hard work, and provide you with recommendations for getting you were you’d like to go next. So don’t be spooked by evaluation – in fact, you might even be surprised at the numerous benefits it can provide.

On that note, we wish you a Spooktacular Halloween!

Feeling Inspired After the AEA Conference

Photo From AEA Website

Hi everyone! We just came back from the annual American Evaluation Association (AEA) Conference that was held in Denver, Colorado. And boy are we newly energized and filled with inspiration! The theme this year was, “Visionary Evaluation for a Sustainable, Equitable Future” accomplished through integrating systems thinking, building relationships, and understanding sustainable and equitable living. This theme really resonated with us, as so much of what we do requires taking a systems approach to evaluation, building relationships for effective data collection and participant buy-in, and ensuring the framework will be sustainable and equitable to the often various parties involved.

We were fortunate enough to present the great work some of our clients are doing and how we’ve been able to build an evaluation framework to help support their efforts and enhance evaluation capacity for sustainability during two roundtable sessions. During the first session, we presented on Developing Comprehensive Evaluations of Complex Programs:  Prevention and Early Intervention for Mental Health, and focused on providing the audience with challenges experienced and lessons learned around developing a comprehensive large-scale evaluation plan with multiple stakeholders. Much of this included strategies for systems-level thinking, establishing a baseline of what is, how to elevate data collection to ensure all necessary metrics are captured, and ways to include key stakeholders in the process.  

Similarly, the second round table, Using Evaluation to Help Build and Sustain a HIV/AIDS and Substance Abuse Prevention Program for High Risk Young Adults, provided lessons learned around developing an evaluation framework to ensure the necessary data are collected even amid challenging circumstances. Through the development of the evaluation and the implementation of a data feedback loop with key partners, relationships were able to be developed and fostered leading to a more sustainable long-term evaluation.

Check out the link to conference at:  http://www.eval.org/p/cm/ld/fid=226

We always find the AEA conference a great way to stay current on evaluation practices and to catch up with colleagues and friends.

Did anyone else attend the conference? What did you think and what insights did you gain from the presentation(s) you attended? 

Using Multiple Measurement Tools

Question: Is it better to measure your impact via surveys among your target group or by conducting a hand-full of in-depth interviews?

Answer: Both (err… trick question)!

There are multiple ways to measure your impact and relying on just one data source or type of data can prevent you from seeing all that you’ve accomplished. For example, survey data can give you a good sense of your impact on the aggregate-level. But if you only use survey data, then you may not be capturing exactly how your efforts have impacted individuals. On the other hand, in-depth interviews can provide you with more detailed contextual information that might help to explain what your data is telling you. But it can be hard (and often not feasible) to get the resources necessary to interview enough people to capture your effect on the aggregate-level.

That’s why we often draw on both quantitative data (e.g., how much, how many, how often, etc.) and qualitative data (e.g., more descriptive) whenever possible. Quantitative data can be really effective in telling you what has happened as a result of your efforts, while qualitative data can provide you with richer information providing deeper insights indicating why or how something happened. 

Consider the Source: Verifying Your Data

If you’ve been keeping up with our blog, you’ve noticed we’ve been busy presenting at various conferences over the past few months and we’ll soon be gearing up to present at the upcoming American Evaluation Association (AEA) conference in October. With all the planning and thought we’ve been putting into ensuring all our data is accurately presented and our sources properly credited… we figured we’d share some data source verification tips, as we often get asked questions about this. With the various amounts of information available to us, it can be easy to stumble upon facts and figures that aren’t very reliable. Sometimes, even the most widely cited statistics can be seriously flawed. So, how do you know that the data you’re using is accurate?
Below are some quick tips:

1)     Check for bias.  Data is only as good as its source, so it’s important to check the original source of your information to make sure that the organization collecting it isn’t biased. Government reports/statistics are typically very reliable, and you can find data on a variety of topics by searching through local, state and national government webpages.

2)     Go to the original source.  Even the most reputable non-governmental organizations can make mistakes when citing official reports. So, when you’ve found a reputable study cited, be sure to find the original report where the information came from and verify that it’s cited correctly. It’s always best to cite the original source when possible.

3)     Try to have multiple sources.  It’s generally a good idea to look for multiple studies/articles on the topic of interest. There are often several organizations that collect similar statistics, and it can be helpful to see whether the organizations found similar results. If various entities are finding the same results you can be more certain of the conclusions. If you notice that findings differ across data sources, then describe the differences, as that can be useful information as well.

4)     Remember to cite your sources!  Be sure to explain to your audience where you got your information -- always remember to cite your sources.

SUD Conference 2014

EVALCORP staff was fortunate to present at the Statewide Substance Use Disorders Conference that was held last week. The theme for the conference was “Integration through Collaboration and Innovation” with workshops focusing on ideas for integrating addiction prevention, treatment and recovery support services with primary care and mental health. EVALCORP staff presented on (1) how Ventura County’s Place of Last Drink (POLD) Survey is used as an effective tool for prevention and (2) ways to collaborate with project staff in evaluating prevention and early intervention (PEI) programs.

In the POLD presentation, we briefly outlined the history of the POLD survey in Ventura County, described the quality assurance measures built in to the data collection and analysis of the POLD surveys collected, shared current selected findings, and illustrated how the data are used to inform and design prevention efforts throughout Ventura County.  This presentation was a collaboration with Dan Hicks, Ventura County Behavioral Health, Alcohol and Drug Programs, Prevention Services Manager who shared extensive knowledge about the ways POLD data have been used in the development of innovative county initiatives to combat impaired driving. 

During the PEI workshop, EVALCORP staff shared their experiences in collaborating with various stakeholders involved in the evaluation of PEI projects. Topics discussed included how to engage service providers in data collection efforts, how to measure existing collaboration efforts and ways to enhance collaboration among partner agencies, and ways in which to overcome challenges associated with diverse programs.

More information about the conference and access to the workshop content can be found here:  http://www.cce.csus.edu/conferences/aod/14/sessions.cfm?pid=936

What Next Week’s Conference Participants Will be Thinking…

As we’re gearing up to present at the CA Substance Use Disorders Conference in Costa Mesa, taking place next week -- we thought the image above was especially fitting. Hope to see you there!  

GO ELIMINATORS!

We hope everyone has been out enjoying the summer so far! In keeping with our sports theme this month (we talked about the World Cup a little in our previous post) – we wanted to give a shout out to the EVALCORP sponsored softball team, The EVALCORP Eliminators. The Eliminators are currently playing the summer season (June 23 to September 8). When you’re ever in the Oxnard area, stop by for a game and cheer them on!

What in the World Does the World Cup have to do With Evaluation?

Was anybody else as excited about the 2014 World Cup as we were? We’ve pretty much spent the last month doing lots of cheering every time our favorite teams scored a goal or advanced into the next round! Watching all those goals scored got us thinking about goal setting (shocking, right?) and since it’s coincidentally the beginning of a new fiscal year – we thought it would be the perfect time to talk a little about this subject.

Every year at this time, the EVALCORP team is busy assisting our clients with fiscal year planning, work plan development, and kicking-off new initiatives.  As we strategize on how to best set and accomplish new goals, we adhere to the “SMART” concept (i.e., setting goals that are specific, measurable, attainable, relevant, and time-sensitive) and the following key “plays” for goal achievement:  

(1) developing tools and tracking mechanisms to ensure our clients are headed down the field in the most strategic manner;  

(2) reinforcing  the need for ongoing communication with the client team and within the EVALCORP team; and

(3) like the winning team of a soccer match, making sure to celebrate the wins!

And, similar to teams studying game footage in order to adjust their strategies and formations for ongoing improvement, we also measure progress throughout the year – noting what has been accomplished, identifying challenges encountered and determining ways to overcome roadblocks and to get the ball in the net, so to speak!

Send us an email at info@evalcorp.com if you’d like to talk through goal setting/planning – we are happy to discuss how we can set you up to accomplish your goals. 

Lastly because we LOVE data at EVALCORP, we wanted to wrap up this post with some fun facts about the 2014 World Cup:  

·      A total of 64 matches were played.

·      Soccer matches were held in stadiums across 12 cities throughout Brazil.

·      Brazil has won the World Cup title 5 times.

·      The Maracana Stadium, where the final game took place, seats nearly 75,000 people. 

·      USA goalkeeper, Tim Howard, set a record for most saves in a single World Cup game (i.e., total of 16 saves). 

·      This is the 4^th World Cup Title for the 2014 World Cup Champions, Germany 

·      Germany is the first European country to win the title in a Latin American host country.

Congratulations to the 2014 World Cup Champions – Germany!

Managing the Strategic Planning Process

There are many considerations that go into the development of an effective strategic plan – which can make the process seem daunting at first. At EVALCORP, we've found that if you break up the overall project into smaller, more manageable components, the process can be quite enlightening and the final product will serve as a valuable guidepost you can refer to as you move your organization forward.

At the most fundamental level, strategic planning is assessing where your organization is now, where you want/need it to be, and what you need to do to get it there. Often times, starting with a high-level outline of what your strategic plan should be is probably the best first step you can take. This way you can gain a clearer sense of which components need to be included in your plan and can “grease the slide” as they say, in developing a well-written and useful strategic plan. 

We recently came across an interesting post on the American Evaluation Association’s AEA 365 blog, related to simplifying strategic planning, that breaks the process down into 3 steps:  (1) taking stock of the situation; (2) picking targets and setting objectives; and (3) formulating plans. The diagram pertaining to this post is provided below -- and to read more about this approach, please click on the following link:

http://aea365.org/blog/fred-nickols-on-simplified-strategic-planning/