CS 8 – Demonstrating Program Effectiveness
The Scenario: A service agency is committed to large program expenses, but is unable to identify specific outcomes.
Organizational Profile: 500+ staff members plus 50,000 volunteers.
Outcomes: Renewed commitment and legitmacy for the program.
Proving the Program
This membership organization habitually spent hundreds of thousands of dollars for the training of their staff and volunteers. However, asside from the occasional “How do you feel….?” Questionaires, no attempt was ever made to determine whether the expenditure was producing the desired results.
I was the training director, and had incorporated many changes in the training. Now questions were being asked about the wisdom of further increased expenditures. Kirkpatrick’s 4-phase approach to training evaluation was implemented to answer these questions.
The following data collection steps were taken:
a – pre and post-tests were developed and tested relative to validity and reliability and used to measure changes in learning during the training
b – post training surveys assessing how specific skills taught in the training were applied in the months after training, and rubrics for verification from colleagues using a 360 degree approach were developed and applied at the zero, six, twelve, eighteen, and twenty-four month marks after training completion.
c – organizational leaders agreed to rubrics that would identify desired organizational change, and they were applied at the zero, six, twelve, eighteen, and twenty-four month marks after training completion.
d – “I feel…” affective surveys were standardized.
The data was analyzed seeking indications regarding the effectiveness or lack of effectiveness.
The following are the results of the analysis.
1. The affective measure covaried with the difference between pre and post-test measures. This was not expected, but may be due to the nature of adults in educational and training activities in which it is generally accepted that adults are more accepting of activities that are directly applicable to their lives.
2. There was a high level of application of the new skills immediately upon returning home, but a significant drop in application from the sixth through eighteenth month after training. There was not a recovery in the applicatioin of the new skills until the eighteenth and twenty-fourth month after training. This result was surprising, and additional research was conducted to illuminate the causes. The early drop was apparently due to the resistance to new skills from co-workers. This was modified after a few months when the training participants confronted challenges requiring the new skills.
3. There was a uniform increase in the measures of organizational effectiveness in the months following the training.
 |