15.1 Evaluation research

Learning Objectives

  • Describe how to conduct evaluation research
  • Define inputs, outputs, and outcomes
  • Identify the three goals of process assessment

 

As you may recall from the definition provided in Chapter 1, evaluation research is research conducted to assess the effects of specific programs or policies. Evaluation research is often used when some form of policy intervention is planned, such as welfare reform or school curriculum change. The focus on interventions and social problems makes it natural fit for social work researchers. Evaluation research might be used to assess the extent to which intervention is necessary by attempting to define and diagnose social problems in a social worker’s service area. It might also be used to understand whether their agency’s interventions have produced their intended outcomes.

I often remind my students they will eventually have bright ideas about what programs or interventions their agency should try. Moreover, they will eventually be so good at their job they will take on additional administrative and supervisory responsibilities. As a result, you will need to prove to your agency and the people who fund your agency that your interventions are successful. Government and private grants normally require outcomes to be measured and reported to maintain funding.

 

scrabble tiles spelling out the word "assess"

An outcomes assessment is an evaluation designed to discover if a program achieved its intended outcomes. Much like all of research, it comes with its own peculiar terminology that resembles an assembly line at a factory (Engel & Schutt, 2016). [1] Inputs are the resources needed for the program to operate. These include physical location, any equipment needed, staff (and experience/knowledge of those staff), monetary funding, and most importantly, the clients. Program administrators pull together the necessary resources to run an intervention or program. The program is the intervention your clients receive—perhaps giving them access to housing vouchers or enrolling them in a smoking cessation class.

The outputs of programs are tangible results of the program process—i.e., the boring things that come out of your program. Outputs in a program might include the number of clients served, staff members trained to implement the intervention, mobility assistance devices distributed, nicotine patches distributed, etc. By contrast, outcomes speak to the purpose of the program itself.

Outcomes are the observed changes, whether intended or unintended, that occurred due to the program or intervention. By looking at each of these domains, evaluation researchers can obtain a comprehensive view of the program.

Let’s run through an example from my wife’s social work practice. She runs an after-school bicycling club called Pedal Up for children with mental health issues. She has a lot of inputs in her program. First, there are the children who enroll, the volunteer and paid staff members who supervise the kids (and their knowledge about bicycles and children’s mental health), the bicycles and equipment that all clients and staff use, the community center room they use as a home base, the paths of our city where they ride their bikes, and the public and private grants they use to fund the program. Next, the program itself is a twice weekly after-school program in which children learn about bicycle maintenance and bicycle safety for about 30 minutes each day and then spend at least an hour riding around the city on bicycle trails.

There are many ways to measure the outputs of this program. She would probably include the number of children and staff participating in the program or the number of bike rides or lessons given. Other outputs might include the number of miles logged by the children over the school year, the number of bicycle helmets or spare tires distributed, etc. Finally, the outcomes of the program might include providing surveys to family members or teachers assess whether each child’s mental health symptoms have improved. Outcomes may also be measured by counting the number of behavioral issues at school or by conducting a child-friendly survey with the children themselves.

Outcomes assessments are performed at the end of a program or at specific points during the grant reporting process. What if a social worker wants to see if their program is on target to achieve its outcomes during the early stages of implementation? In this case, a process assessment is recommended, which evaluates a program in its earlier stages. Faulkner and Faulkner (2016) [2] describe three main goals for conducting a process evaluation.

The first is program description, in which the researcher simply tries to understand what the program looks like in everyday life for clients and staff members. To assess program description in our Pedal Up example, we might measure the number of hours children spent riding their bikes or the number of children and staff in attendance during the first few weeks of the program’s implementation. This data will allow the individuals who are in charge of the program to see how their ideas have translated from the grant proposal to the real world. If child attendance is low or if they are only able to ride their bikes for ten minutes each day, then the assessment may indicate that something is wrong.

Another important goal of process assessment is program monitoring. If you have some social work practice experience already, it’s likely you’ve encountered program monitoring. Agency administrators may look at sign-in sheets for groups, hours billed by clinicians, or other metrics to track how services are utilized over time. They may also assess whether clinicians are following the program correctly or if they are deviating from how the program was designed. This can be an issue when conducting program evaluations of specific treatment models, as any differences between what the administrators conceptualized and what the clinicians implemented jeopardize the internal validity of the evaluation. In our Pedal Up example, we could utilize program monitoring to catch staff members who do not review bike safety each week or staff who do not consistently enforce helmet laws.

The final goal of process assessments is quality assurance. At its most simple level, quality assurance may involve sending out satisfaction questionnaires to clients and staff members. If there are serious issues, it’s better to know them early on so the program can be adapted to meet the needs of clients and staff. Not only is it important to solicit consumer feedback, but it is also imperative to ask for staff feedback. The staff have valuable insight into how the program works in practice and they can identify areas in which the may be falling short. In our example, we could spend some time talking with parents when they pick their children up from the program or hold a staff meeting to provide opportunities for those most involved in the program to provide feedback.

Evaluation research is a part of all social workers’ toolkits. It ensures that social work interventions achieve their intended effects. This protects our clients and ensures that money and other resources are not wasted on programs that do not work. Evaluation research uses the skills of quantitative and qualitative research to ensure clients receive interventions that have been shown to be successful.

 

Key Takeaways

  • Evaluation research is a common task for social workers.
  • An outcomes assessment evaluates the degree to which a program achieves its intended outcomes.
  • Outputs differ from outcomes.
  • Process assessments evaluate a program in its early stages so changes can be made.

 

Glossary

Inputs- resources needed for the program to operate

Outcomes- the issues you are trying to change in your clients

Outcomes assessment- an evaluation designed to discover if a program achieved its intended outcomes

Outputs- tangible results of the program process

Process assessment- an evaluation conducted during the earlier stages of a program or on an ongoing basis

Program- the intervention clients receive

 

Image attributions

assess by Wokandapix CC-0

 


  1. Engel, R. J. & Schutt, R. K. (2016). The practice of research in social work (4th ed.). Washington, DC: SAGE Publishing.
  2. Faulkner, S. S. & Faulkner, C. A. (2016). Research methods for social workers: A practice-based approach. New York, NY: Oxford University Press.

License

Icon for the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Scientific Inquiry in Social Work Copyright © 2018 by Matthew DeCarlo is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted.

Share This Book