Evaluating the success of your training: Process vs Outcome evaluation

1. Evaluation of Training B&T Chapter 9

2. Evaluation = 2 Types Process Evaluation Outcome Evaluation

3. Outcome Evaluation The KEY outcomes achieved by your training Trainee Reactions Learning Transfer to the job Organizational Results The same as your objectives

5. What kind of evaluation will you be using for your workshop?

6. Reaction Questionnaires Reaction Questionnaires Training reaction questionnaires do not assess learning, but DO assess the trainees attitudes about and perception of the training.

7. Reaction Questionnaires continued At a minimum, the following categories should be assessed in a reaction questionnaire: Relevance/value of training (to trainees) Training content and materials Trainers behavior/effectiveness Facilities, activities and procedures

8. Steps to Consider in Developing a Reaction Questionnaire Part 1 of 2 Determine what you want to find out (consider training objectives). Develop a written set of questions to obtain the information. Develop a scale to quantify respondents data. Make forms anonymous so participants will feel free to respond honestly. Ask for subgroup information (e.g., young vs. old; minority vs. non- minority). This could be valuable in determining effectiveness of training for different cultures, for example, which may be lost in an overall assessment.

9. Steps to Consider in Developing a Reaction Questionnaire Part 2 of 2 Allow space for Additional Comments in order to allow participants the opportunity to mention things you might not have considered. Decide the best time to give the questionnaire to get the information you want. If right after training, ask someone other than the instructor to administer and pick up the information. If some time later, develop a mechanism to obtain a high response rate (e.g., encourage the supervisor to allow trainees to complete the questionnaire on company time).

10. Evaluations at UWP Exercise 1 Text page 345

11. Instructions In groups Examine the course evaluation we use here in CBEC at UWP. Is it designed to rate the course content and/or to rate instructors? Does it meet the requirements of a sound reaction questionnaire? Why or why not? Explain how you would improve it (if possible).

12. Debriefing Is the CBEC evaluation designed to rate the course content and/or to rate instructors? Does it meet the requirements of a sound reaction questionnaire? Why or why not? Explain how you would improve it (if possible).

13. Fabrics Inc. Part 1

14. Instructions Go to the role-play for active listening in the Fabrics Inc. example (pages 339-340). In groups of 5-6, choose someone to be the initiator and someone to be the trainee. Have them go through the role play while the rest evaluate the trainees response on a scale of 1 to 7 1 being poor 7 being excellent

15. You have up to 15 minutes to complete the 3 scenarios

16. Debriefing Now observers, share your scores. Were they all exactly the same? If not, how could you make the instrument more reliable? If they were all the same, why was that? Is there anything you would suggest to make the evaluation process easier?

17. Fabrics Inc. Part 2

18. Critique the two evaluation instruments developed for the Fabrics case trainee reaction page 338 evaluator report form page 344 E.g., how easy are these forms for people to use? What useful information is gathered? How could they be improved?

19. Westcan Revisited Case in Chapter 4 Exercise 4 Text page 345

20. Assume you were the training manager in the Westcan case (in chapter four). How would you suggest evaluating the training, assuming they were about to conduct it as suggested in the case? Be as specific as you can.

21. Tricky Nickys Carpet Cleaning Co Blanchard pages 345-346

22. Tips: This is a relatively straight forward assessment of the financial value of the training provided to the carpet cleaners. Questions 1-4 take the student step by step through the assessment. Question 5 gets the students to focus on the factors that affect performance, other than KSA deficiencies.

23. Question 1 How much does the re-cleaning cost Nicky per year? Show all mathematical calculations.

24. Question 1 How much does the re-cleaning cost Nicky per year? Show all mathematical calculations. Number of offices cleaned per year = 6/day X 100 people X 250 days = 150,000 Bad cleanings = 1/6, so 150,000/6 = 25,000 bad cleanings/year Cost per bad cleaning = $20.00 Yearly cost = 25,000 X $20.00 = $ 500,000 Since the training program will reduce bad cleanings to 1 in 12, it will save Nicky $250,000 a year if everyone is trained.

25. If you wanted to figure in opportunity costs as well, the total cost of bad cleanings is $40.00. The additional $20 in cost is because time spent re-cleaning means time not spent cleaning and earning a $20.00 profit. If opportunity cost is built into subsequent calculations, it will be based on a $40 cost rather than the $20 figure we use.

26. Question 2 If everyone is trained, how much will the training cost? How much will it cost if only the group with the most errors is trained? Show all mathematical calculations.

27. The value of this question is to determine if the students are able to see that some costs will be the same (development), some will be reduced by half (e.g., cost of employee wages while at training), and some will only be reduced by 40 percent (per session costs)

28. Question 2 If everyone is trained, how much will the training cost? Costs add up to $17,900.00. This is the cost if everyone is trained (i.e., five sessions of 20 employees each).

29. Question 2 How much will it cost if only the group with the most errors is trained? Show all mathematical calculations. If only the 50 with the most errors are trained, only three training sessions will be needed (no more than 20 employees/session)

30. $4,000 in development costs remain the same. Trainer cost would be reduced to $240.00 3 sessions instead of 5. Also reduces cost of facility & equipment to $300.00. For factors associated with number of employees trained, the cost is cut in half. cost of materials is $1,000.00, refreshments are $300.00, and employee costs would be $4,000.00. Indirect costs (evaluation) also cut by 50 % to $1,200. Total costs for training only 50 employees = $11,040

31. If only half of the cleaners were trained (those averaging two bad cleanings out of every nine), the $4,000 in development costs would remain the same. Trainer cost would be reduced to $240.00 because three sessions would be required instead of five. The reduction in sessions also reduces the cost of the facility and equipment to $300.00. For factors associated with number of employees trained, the cost is cut in half. The rationale is that, for the most part, these costs are based on the number of people being evaluated. Since most of the evaluation measures are developed in the TNA stage, the vast majority of trainer time is spent in collecting and analyzing post training data. Thus, the cost of materials is $1,000.00, refreshments are $300.00, and employee costs would be $4,000.00. Indirect costs (evaluation) would also be cut by 50 percent to $1,200. Thus, total costs for training only 50 employees would be $11,040.00.

32. Question 3 If everyone is trained, what is the cost savings for the first year? If only the group with the highest recleaning requirements is trained, what is the cost savings? Show all mathematical calculations.

33. Question 3 If everyone is trained, what is the cost savings for the first year? Training will reduce the number of bad cleanings by 50 percent; the yearly cost of bad cleanings after training = cost prior to training ($500,000) = $250,000. Cost savings is $250,000 - $17,900 (cost of training) = $232,100.00.

34. Question 3 If only the group with the highest recleaning requirements is trained, what is the cost savings? Show all mathematical calculations. The group averaging 2/9 re-cleanings generates 2/3 of the re-cleanings, costing the company $333,333/yr. If this group reduces its re-cleaning to 1/12 it will still cost the company $125,000/yr. in bad cleanings. The gross savings will be $333,333-$125,000= $208,333/yr. The cost savings is $208,333-$11,040 (the cost of training) = $197,293.

35. Question 4 What is your recommendation based on the expected return on investment? Should both groups be trained or just the group with the 2/9 recleaning ratio? Provide a rationale that includes both financial and other factors. Show any mathematical calculations.

36. Since the gross savings if everyone is trained is $250,000, this means that training the group averaging 1/9 recleanings will result in a gross savings of $41,667.00 (i.e., $250,000- $208,333=$41,667).

37. The cost of training only the 2/9 group is $11,040.00. The additional cost to train the 1/9 group is $6,860. The percent return on investment for the 2/9 group is [($208,333-$11,040)/$11,040] x 100, or a 1787.07 percent. For the 1/9 group the return on investment is [($41,667 - $6,860)/$6860] x 100, or 507.39 percent.

38. From a financial perspective, both groups should be trained, even though ones ROI is considerably higher than the other.

39. From an employee relations standpoint, training both groups has the advantage of giving both groups the increased KSAs and equal treatment.

40. Question 5 Lets back up and assume were still at the needs analysis stage. Assume that employees had the KSAs needed to clean the offices effectively. What other factors might you look at as potential causes of the recleaning problem?

41. Motivational factors such as pay, working conditions, group norms, etc. Opportunity factors such as equipment, procedures, etc.