Monday, December 21, 2015
Tuesday, November 17, 2015
Evaluation in a Multicultural Context
Hi everyone! We have just returned from the annual American
Evaluation Association Conference in Chicago – and what a great conference it
was. The theme of this year’s conference, “Exemplary Evaluations in a
Multicultural World,” may have been one of the most relevant themes to our
work in recent years.
In addition to participating in numerous sessions and
workshops, we had the opportunity to present our experiences related to
evaluating Prevention and Early Intervention (PEI) Programs across three counties in Southern California. During the presentation we shared
how we ensure that our evaluations capture the voices of the various
populations participating in PEI Programs. We also discussed lessons learned
and challenges experienced. If you’re interested in the full presentation,
please click on the following link:
Key take-aways that emerged across the various
sessions included:
- The importance of being inclusive of varying perspectives and populations as we design and implement measurement strategies.
- Evaluators should be aware of their own limitations in terms of cultural competence and be mindful of seeking external expertise if working on evaluations that require obtaining the perspective of cultures the evaluator might not be familiar with.
- The term "culture" goes beyond just race but also includes sub-populations and sub-cultures such as the LGBTQ community and marginalized groups.
- Clear standards for identifying "cultural competence" are still lacking in the field; however, work is being conducted to fill this gap.
More information about the American Evaluation Association can be found at the following link: http://www.eval.org/
Friday, August 14, 2015
Empowerment Evaluation: Key Facets of Empowerment Evaluation
David Fetterman, one of empowerment evaluation’s founders,
offers five facets of empowerment evaluation that turn the theory of
empowerment evaluation into the practice of empowerment
evaluation:
1) Training: This one should be pretty obvious. There is a lot of planning
and knowledge-expertise that goes into evaluating, so the evaluators first need
to teach their clients the principles and methodology of evaluation.
2) Facilitation: After training, evaluators should step back from conducting the
evaluation themselves to take on the role of a guide or coach, monitoring the
client’s processes and ensuring they can use the evaluation to improve their
program.
3) Advocacy: A large part of a
successful and effective empowerment evaluation comes from the support of the
evaluator. An evaluator might see several different options for making a
program succeed and can recommend which might be the best. But rather than
telling the client what to do, the evaluator advocates for client’s own
abilities to put into practice the steps that the client believes will make the
program successful.
4) Illumination: This facet may be one of
the most important to consider when collaborating with clients. Evaluators must
take the time to share the processes, techniques, and the theory of empowerment
evaluation with their clients so that clients can build confidence in their
evaluative skills.
5) Liberation: When clients gain the
skills necessary to evaluate their own programs, they become liberated from the
need to have someone else to tell them what they’re doing right and wrong. The
evaluator instead supports clients when opportunities to grow arise, when it’s
time to consider redefining the program, and when clients want to change
themselves and the program for the better.
Empowerment evaluation is a highly participatory and
collaborative approach to evaluating programs. Following these five facets
allows a program to be evaluated, transformed, and effectively improved to
address community and society problems.
Want more information? Check out the links below:
An overview of the principles of empowerment evaluation and
details an example of the process in action: http://people.ucsc.edu/~ktellez/266Readings/Empowerment%20Evaluation.PDF
A short article about the history, the current state, and the
future plans for empowerment evaluation written by Fetterman and Wandersman: http://davidfetterman.com/documents/EEyesterday.pdf
Wednesday, July 29, 2015
Empowerment Evaluation: Focus on Client Control
As you know from our last post, we’re in the midst of a 3-part series on empowerment evaluation. In our prior installment we introduced the basics of empowerment evaluation providing you with some of the key concepts and a big picture overview. For this post we’d like to focus a little more on one of the key components of the theory, which is client control over the evaluation. The role of the evaluator in an empowerment evaluation is to encourage clients to collaborate, communicate, and be involved in the process of program improvement. Evaluators are the facilitators or coaches to the program improvement process where the client is in control. This a key difference from other theories or approaches to evaluation where the evaluator is in control of the process.
What? The client is in control, can that be right? --- Yes!
However, evaluators would never just hand over the reins to
clients without providing them with the necessary components for an effective and credible evaluation. The most traditional concept of evaluation typically means placing a value
judgment of the merit and worth of a program, but in empowerment evaluation the
context for the evaluation is different, as the clients are heavily involved.
Rather than assessing the value or impact of a program, empowerment evaluation focuses
on program improvement.
Where should one start with empowerment evaluation?
Because empowerment evaluation calls for huge investments of
time and energy from the clients, it can be difficult to know how to begin.
David Fetterman, the father of the empowerment evaluation, suggests several
steps to ensure that clients are prepared for the extensive process of
illumination and understanding that leads to the program improvement brought by
empowerment evaluation. The four steps for clients that can help facilitate the
learning process are as follows:
1) Taking Stock: With the help of the evaluator,
begin to determine where the program currently stands. Discussions should
include requisite program resources and the ultimate impact of the program.
2) Establish Goals: Discuss program
improvement with the coach (evaluator) and community. Define exactly what needs
to be better and where program staff and others in the community want the
program to be in the future.
3) Develop Strategies: Converse with staff
members about the potential challenges that could arise during the evaluation process
and establish clear strategies to ensure a successful evaluation is implemented
and desired outcomes are achieved.
4) Select Evidence: Determine outcomes and
indicators to reach the program improvement goals established in step 2. Ask
yourself “How will I reach my goals and objectives? How will I measure success
of accomplishing goals?”
Alright, we’ve highlighted some important aspects of
empowerment evaluation and steps to begin the process but there is so much more
information about empowerment evaluation out there.
Are you further intrigued about empowerment evaluation? Then
be sure to look for our last blog post in this empowerment evaluation series
coming soon, where we will discuss in which contexts and for whom empowerment
evaluation is a good fit.
You can also find more information about empowerment
evaluation by clicking on these links:
David Fetterman’s Empowerment Evaluation blog: http://eevaluation.blogspot.com.au/?view=magazine
Better Evaluation website with a plethora of resources on
empowerment evaluation: http://betterevaluation.org/plan/approach/empowerment_evaluation
Sources:
Fetterman, D. M.
(1996). Empowerment evaluation in theory. The
Evaluation Exchange, 2(2), n.p. Retrieved from website: http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/family-resource-centers/empowerment-evaluation-in-theory
Fetterman, D. M.,
Rodriquez-Campos, L., Wandersman, A., & Goldfarb O’Sullivan, R. (2014).
Collaborative, participatory, and empowerment evaluation: Building a strong
conceptual foundation for stakeholder involvement approaches to evaluation (a
response to Cousins, Whitmore, and Shulha, 2013). American Journal of Evaluation, 35(1), 144-148. doi:
10.1177/1098214013509875.
Sherriff, B.,
& Porter, S. (n.d.). An introduction to empowerment evaluation: Teaching
materials. Retrieved from website: http://www.mrc.ac.za/crime/evaluation.pdf
Friday, July 17, 2015
Empowerment Evaluation: The Basics and the Evaluator’s Role
In our previous blog post, we began a brief introduction to the
theories of evaluation. We touched on some of the key theories used today:
utilization-focused, developmental, empowerment, and participatory evaluation. In this post, we’d like to delve a little
deeper into the concepts and background of empowerment evaluation.
As a quick recap, empowerment evaluation is a
type of evaluation where the evaluator takes on a coaching role and provides
the client with greater control over the evaluation. The evaluator gives the
client the tools and abilities to meet and evaluate the program goals, while
the evaluator helps keep the evaluation process on track by addressing the
client’s needs.
Empowerment evaluation originated in the 1990s and stemmed from
the need for a more iterative approach to assessing the strengths and
weaknesses of a program to improve it. David Fetterman, one of the pioneers of
empowerment evaluation, stresses that empowerment evaluation empowers clients by
providing them with the skills to evaluate their own program(s). When
clients can evaluate their program(s) (with the guidance of an evaluator),
evaluation becomes part of the program planning and management process.
Empowerment evaluation aims to ensure that the program will
succeed by identifying approaches that will increase the program’s success.
This is one key difference from other theories of evaluation that assess
program outcomes and success without influencing/altering the program in any
way.
Next we’ll talk about how
empowerment evaluation works in practice. Stay tuned.
Sources:
Patton, M.
Q. (2005). Toward distinguishing empowerment evaluation and placing it into a
larger context: Take two. American Journal of Evaluation, 26(3),
408-414. doi: 10.1177/1098214005277353
Sherriff,
B., & Porter, S. (n.d.). An introduction to empowerment evaluation:
Teaching materials. Retrieved from website: http://www.mrc.ac.za/crime/evaluation.pdf
Tuesday, June 30, 2015
Theories of Evaluation
As the evaluation field has developed over the past several
decades, evaluators have adopted different approaches to guide their work. Although
it can be daunting to learn about all the different approaches to evaluation
research, it is helpful to be aware of the various types of evaluation
approaches, as it could help when selecting an evaluator or when designing an
evaluation.
At Evalcorp, we predominantly take a utilization-focused
approach to our work, meaning that we first identify how clients want to use
the information before we design our evaluation plan. We want to ensure our
evaluation design, methods, and reports address our clients’ most important
questions and provide relevant information to help them make informed decisions.
Other types of evaluation approaches include:
1) Developmental Evaluation: By practicing
developmental evaluation, evaluators become part of their clients’ program
design or organizational management team. Rather than just collecting data and
analyzing results, they work directly with clients to help them develop
interventions to key problems.
2) Empowerment Evaluation: In this type of
evaluation, the evaluator acts as a coach or facilitator who helps clients
develop the knowledge and skills to monitor and evaluate their own programs.
The goal is to help clients develop their own self-determination so that they
don’t become dependent on consultants to inform them of their performance.
3) Participatory Evaluation: Evaluators
practicing participatory evaluation help inform and improve the ways clients
implement their programs. They become partners with their clients and help them
learn how to coordinate their own evaluation efforts. This leaves the evaluator
to the task of completing technical activities, like designing surveys,
analyzing results, and reporting findings.
Over the next few months we’ll dig deeper into some of these
theories, so be sure to come back to learn more about the different
types of evaluation theories and how they influence evaluation approaches.
For more information on other approaches to evaluation
research, check out Evaluation Roots by Mark Alvin or
betterevaluation.org.
Friday, May 29, 2015
Top 10 Time Management Tips
With
the end of the fiscal year quickly approaching, we often encounter folks who
are stretched too thin facing numerous conflicting deadlines. So, we thought
this would be the perfect opportunity to share a few time-management skills.
Some of the following tips are strategies we’ve implemented and found to be
effective, while others are frequently mentioned in time management articles.
We know there are numerous ways to manage your time and re-prioritize, so let us know if you have any time management strategies by leaving a note in the comment section below.
Time Management Tips
1.
Make
a to-do list that prioritizes upcoming tasks and estimate how long each task will
take. Work on the most time-sensitive tasks first and set a plan for
accomplishing the other tasks. It’s important to be realistic about the time
things will take, and as a general rule it’s better to overestimate the time
you think it will take to complete each activity.
2.
Determine
the time(s) of day you are most productive and/or do your best thinking. Try
to work on more demanding/important tasks during those times to maximize
efficiency.
3.
Break
down big projects into smaller tasks that are more manageable to
accomplish within limited time.
4.
If
you find that you have a spare half-hour or small period of time, don’t waste
it. Use it to accomplish one of the shorter tasks on your to-do list.
5.
If
you’re spending a lot of time on one task and aren’t getting very far, then
take a short break. You’ll be more productive once you’re re-energized.
6.
Try
to stay off of emails and turn your phone on silent when you are in the middle
of a big report or task. Reading and responding to emails, calls, or text
messages will distract your focus.
7.
It’s
OK to say “No.” (when appropriate). Your time is valuable, and it’s important
to set priorities and boundaries.
8.
Make
use of idle time—waiting in line at the grocery store, being in traffic,
sitting on the train—by reflecting on what you’ve accomplished and your goals
moving forward.
9.
Unless
you need to use social media for work, set short limits (10 - 15 minutes at the beginning or end of the day) on the amount of time you spend
on social media sites.
10. Remember not to skimp on sleep. Being well-rested allows for a clearer mind, and will
make you more efficient so that you can accomplish everything that needs to be
done.
Sources:
Refine The Mind: http://www.refinethemind.com/21-time-management-tips/
Thursday, April 23, 2015
Overcoming Data Collection/Management Challenges
Challenges are an inevitable part of life. No matter what
field or profession you work in, you are bound to encounter some sort of roadblocks.
Same thing goes for conducting applied evaluations or program implementation. Over
the years, we’ve found that many organizations experience similar problems in
regards to data collection and measurement. We’ve highlighted the two most common
problems we encounter specific to data collection and survey design along with
some recommendations.
Survey Design Pitfalls
Surveys are often a topic of much consternation and can
become problematic if not crafted with the target audience in mind. We often
see surveys that include too much content, are not tailored to the appropriate
audience, include items that don’t address the evaluation questions, or have
double-barreled questions that limit the interpretability.
To solve survey design issues: look at the survey in a more
objective manner and ask yourself whether the questions are “need to know” vs.
“nice to know.” Only keep the items that are truly necessary and will be used
in the analysis – so as not to waste the survey respondent’s time. Consider removing
any unnecessary identifying or demographic information that you don’t intend to
us. Don’t just collect data because you think you’re supposed to. Also, keep
your survey items simple and to the point so the respondent can focus on
selecting the most appropriate response, and doesn’t spend time trying to
interpret/understand the question. Additionally, simple, small tweaks to
phrasing or rating scales can instantly transform a problem survey to a successful
one! For example, try using a five-star rating scale if collecting data from
youth audiences. Using more visually interesting content can make the survey
process more appealing.
Staffing Issues/Limited Time for Data Management
We often hear agencies say they don’t have enough staff to
do the work that is required. Specifically, data collection and implementation
often fall to wayside with the other daily demands. However, often times the
data monitoring and collection is the most critical part of the job to ensure
all the hard work is being documented.
We have found that streamlining data collection tracking
tools into one centralized database can help save time in the long run, as all
data can be found in one location. Similarly, as discussed above, re-evaluating
the data that is being collected might help to identify pieces that are not
relevant and can stopped being tracked. While re-structuring existing practices might be time consuming at the beginning, coming up with a better system can save your staff time and energy when entering, analyzing, and
reporting out data. We also find it helpful to keep a codebook or example entry
in your data collection system to ensure your data is entered accurately and
efficiently even when time is limited or when there is staff turnover. Need assistance
streamlining your existing tools? Talk to you evaluator to brainstorm ways to
set up a system that works best for your organization.
Friday, March 6, 2015
California Problem Gambling Summit
This past month we attended the 2015 Problem
Gambling Training Summit, organized by the California Department of Public
Health, Office of Problem Gambling (OPG) – and boy are we glad we had the
opportunity to go. The theme of the summit was: “Don’t Ignore the Signs” and included
presentations from leading experts in the field who discussed current trends in
problem gambling, co-occurring disorders, impacts and negative consequences, and prevention
and treatment focus areas.
OPG contracts with the University of California,
Los Angeles Gambling Studies Program to develop and coordinate a broad spectrum
of treatment services using a multi-pronged approach, including: problem gambling telephone intervention;
outpatient treatment; intensive outpatient treatment; residential treatment
program; and clinical innovations. Additionally, OPG funds and oversees a
prevention component that includes a toll-free helpline, training and
education, and outreach and public awareness.
Awareness and educational materials are made available in a variety of
different languages to ensure a broader reach.
The two-day intensive summit was a way to
bring together the expansive group of providers offering problem gambling
treatment services and to present/discuss current research and trends. We were truly
impressed by the comprehensive services available and vast network of
providers.
To learn more about the program, treatment
outcomes, and to access fact sheets and other resource materials click on the
following link: http://problemgambling.securespsites.com/ccpgwebsite/default.aspx
Subscribe to:
Posts (Atom)