Program Evaluation

Program Evaluation Process

The Racine County Human Services Department utilizes three basic formats to review, assess and evaluate services which it purchases: 

• Site Assessments

The Site Assessment is designed to review three primary areas of the contracted service. The first is the Administrative Operation of the agency or program. An overall assessment of the agency's administrative operation and policies occurs to determine an agency's use of sound and efficient management techniques. Areas specifically targeted for review include: the role of the Board of Directors and their involvement is establishing program direction; mechanisms in place for program and budget planning; financial, operational and personnel procedeures; compliance with EEO/Affirmative Action, Accessibility and program reporting.

The second area of assessment reviews the fiscal operation of the agency or program to determine if the appropriate financial records are being maintained and whether the agency is complying with accounting principles established in the contract and per the State Allowable Cost Policy Manual. A review occurs of indirect versus direct cost allocation, bank reconciliation, accounting systems, audit compliance, leases, and fiscal reporting.

The Program Management and Service/Treatment process are also reviewed during the site assessment. Program outcomes, data collection, consumer surveys, and verifiable program information are researched to determine how the agency meets its program goals and objectives, and the processes utilized to do so.

• Program Evaluation Contracted Outcomes

Evaluation of a program's contracted outcomes occurs annually, or based on the services provided more frequently. Each agency's contracted outcomes are specified in each contract. Actual outcome data is requested at either at mid-year, or within one month after their contract ends. Outcome reports are generated annually and submitted to the Human Services Boad for their review.

• Program Evaluation 

Each year the Human Services Department selects certain program areas and agencies for program evaluations and client satisfaction surveys. The evaluation reviews the contracted service's overall effectiveness and the benefit to the client population being served. The evaluation process is designed to determine the effectiveness and client satisfaction with the services being contracted. Case Manager feedback is also solicited to determine how responsive the program is to HSD, and how the agency is meeting the objectives as envisioned by department staff. On occasion, organizations besides the Human Services Department, such as the University of Wisconsin-Parkside perform formal and extensive evaluations of Racine County programs. Ordinarily, these types of evaluations are required by national funding sources and provide longitudinal data regarding a program's effectiveness over a course of several years. 

Site Review Format

Agencies are notified several weeks in advance of the formal site review. A description of the items required for the review, as well as pertinent staff are identified to conduct the review. Ordinarily, the Contract Monitor, a program staff person familiar with the program, and a fiscal staff person conduct the review. Once site review scoring sheets are tabulated, the agency is given an assessment of their results, based on the percentage score in each of the folllowing areas: 

1. Operational Management - Administration
2. Operational Management - Personnel
3. Fiscal Management
4. Program Management
5. Service/Treatment Process/Documentation
6. Compliance with Contracted Outcomes

These are the forms that will be used during this process:

• Site Review Notification Form
• Site Review Assessment
• Site Review Scoring Sheet

Program Outcomes

All agencies contracting with Racine County have targeted outcomes which are identified by the department requesting the service, or by the particular funding source. Outcomes are well-defined and are measureable. In general all agencies contracting for a particular service, have the same outcomes, although agencies may include additional outcomes that they are attempting to achieve. The following are sample outcomes for particular service delivery areas:

• Outcomes for Adult Programs

Outcome Objectives for Community-Based Residential Facilities, Adult Family Homes, and Supportive Home Care:

• 75% of clients will not be discharged to a more restrictive living arrangement
• 80% of clients will maintain their skill level (i.e. self care, daily living skills, self-management, etc.)

Other outcomes for individuals in long-term care could include:

• Residents are safe
• Residents make choices about their lives
• Residents receive the services and supports that they need
• Residents are satisified with the services and supports they receive
• Residents maintain their personal dignity

Agencies contracting with Racine County may add outcomes that are specific to their programming and that will enhance the quality of services that they can provide to the residents within their facility.

• Outcomes for Youth Programs

Youth Program Outcomes vary depending on the type of program the youth is participating in. For youth residing in treatment homes the following outcomes are standard:

• 80% of youth will meet the goals and objectives specified in their treatment plans
• 75% of youth will not have contact with the Juvenile Justice System while participating and six month following release from program
• 90% of youth will not be placed in a more restrictive arrangement

For youth participating in various County prevention/intervention programs, the following are some of the outcomes:

• 80% of teens invovled in pregnancy prevention programming will remain pregancy free
• 80% of children involved in Birth-to-Three programs will meet their individual objectives
• 65% of youth participating in AODA treatment programs will successfully complete the treatment
• 95% of youth involved in intensive supervision will be maintained in a non-secure setting
• 90% of youth involved will complete their high school education

Again, outcomes are always measureable and must relate to the end result desired for participants involved in the programming.

Some outcomes may be related to community wide initiatives, and have longer-term desired results, and focus on a specific issue the community is facing. For instance, outcomes that may not be measureable in the short-term, but can be measured long-term could be:

• Teen pregnancy rates in Racine will be reduced by 5% within the next five years
• High school graduation rates will increase by 3% within the next five years
• Juvenile recidivism rates will decrease by 5% within the next year
• 80% of youth entering kindergarten in Racine County will have received an assessment prior to their entering school to identify any physical/academic needs

For more information on how to determine your program's outcome objectives and ways to measure your program effectivenes, view the following website:

United Way Logic Model  www.http://national.unitedway.org/outcomes/resources/mpo/model.cfm

• Outcomes for Workforce Development Center Programs

The majority of programming for the Workforce Development Center focuses on training, employment and retention of individuals accessing services. The following are sample outcomes for WDC Adult Programs:

• 72% of adults participating will enter unsubsidized employment
• 90% of adults will remain in unsubsidized employment six months after employed
• 65% of individuals will attain an educational or occupational skills credential
• Minimum earnings for adults after six months of employment will be $3,000

The following are WDC youth program outcomes:

• 73% of youth participating will attain a basic work readiness or occupational skills credential
• 55% of younger youth, and 72% of older youth will enter unsubsidized employment
• 80% of individuals receiving job retention support will retain employment six months after the start of employment
• Older youth average earnings after six months on-the-job will be $2,430

The Workforce Development Center also utilizes customer satisfaction surveys to all individuals participating in the programs offered at the Center. A sample outcome objective for the WDC workshops is:

• Customer surveys are compiled and reported on a quarterly basis with an overall 95% satisfaction rating

Customer Satisfaction

Some agencies focus heavily on customer satisfaction surveys to determine the effectiveness of their programming. Whether individuals feel their needs are being met, and/or if individual's needs are actually being met are two distinct types of measurement. In addition, the results of customer satisfaction surveys can be biased by who does the questionnaire. If you are using customer satisfaction surveys to evaluate your program, it is preferable that individuals other than those administering or running the program conduct the survey. 

• Sample Survey for Adult Family Homes/Community Based Residential Facilities
• Sample Survey for Workforce Development Center Programs

Follow-up Action Plans

Site reviews are utilized by Racine County to insure that the services provided are cost effective and quality-driven. Occasionally an agency may have areas that are identified and noted for improvement. The agency is given a written description of the concern(s), along with a specific target date with individual responsible for improving that program area. If the organization fails to take corrective action in a timely manner, or is cited again for the same issue, future funding through Racine County is at jeopardy.