Evaluation planning for psychological health and safety

The following is adapted from Guarding Minds at Work.

Consider these evaluation questions:

“How will we know whether these actions achieved our intended outcomes?” 

“How will we know whether the time, effort and money we invested was worthwhile?”

Effective evaluation is:

• Practical
  • Relevant to your intended outcomes
    
  • Straightforward
  • Measurable
  • Easy to put in place
  • Cost-effective
• Flexible – adaptable to your organization and its available resources
• Continuous – including ongoing collection of employee feedback to improve intervention(s)

Step 1: Decide the purposes of the evaluation

There are many reasons for doing an evaluation, including a commitment to:

• Accountability – demonstrates the promised results and how resources were used. 
• Quality improvement – enhances the excellence of services or products on an ongoing basis.
• Examination of specific outcomes – tracks the progress of specific changes identified as priorities.
• Cost-effectiveness – a cost analysis of the resources needed and the outcomes they delivered. 
• Uptake – determines the extent to which employees or other target groups got involved.
• Fidelity – determines whether implemented actions were consistent with the plan.
• Sustainability – predicts the degree to which the employer can maintain the actions over time. 

Identify the purposes and methods of evaluation before initiating the action so you will have a clear and useful answer to the question of whether it worked as planned.

Step 2: Identify key informants

Key informants are those who will provide evaluation data for each action. Participants of the planned action will always be key informants. Others might include:

• Senior leaders
• Human resources
• Benefit providers
• Supervisory staff with responsibility for carrying out the planned action
• Union representatives
• Occupational health and safety staff

Step 3: Make a list of short-term and long-term outcomes to measure

Change can take time. If you focus only on long-term outcomes, you may become discouraged. Identify desired short-term outcomes on top of setting your long-term goals. Short-term outcomes provide rapid feedback and are typically easier to measure.

Short-term measurements and approaches could include:

• Participation – measure how many people take part in your action.
• Behaviour changes – track how many times participants do things differently.
  • Be specific about the change you’re looking for beforehand. Explain how you’ll have participants or their leaders report this activity to you
• Absenteeism – track the number of days off before and after the action
  • Note: if mental health was the focus, an increase in absenteeism may represent people seeking out treatment. 
  • Look for ways to maintain confidentiality while tracking the reason for absence
• Complaints, conflict, or grievances – track the number reported before and after the action
  • Note that in some cases, objections may increase while employees learn what appropriate behaviour is. This short-term increase could be positive. 
• Awareness – determine the level of awareness for your concern by surveying employees.
• Policy or process change – identify any changes made to existing policy or processes.
• Participant evaluation – ask specific questions to get data about the experience. Use this data to set future intentions for implementing change.
• Supervisor evaluation – ask supervisors direct questions about participant behaviour before and after implementation.
• Focus groups – get feedback from participants on the impact the action had on them. 
• Stress satisfaction scan – administer this 6-question survey before and after the action to measure how employees feel about work-related stress

Long-term measurements provide enough time for sustainable change to happen and could include any of the above or either of the following:

• Re-survey – do a follow-up survey to track how employees rank each of the psychosocial statements
• Data analysis – analyze what’s relevant from the sources of data for each psychosocial factor:
  • Before taking the action
  • 13 or more months later

This allows enough time to compare data from the same period as implementation. 

Step 4: Use short-term evaluation results to modify

You can use the short-term outcome evaluation results to adjust the action. You may even want to launch a pilot to determine what’s working well and what should be enhanced or expanded. And you can change or drop those elements that don’t seem to be reaching their objectives. Identifying challenges to success early on allows you to take corrective action. This can positively influence your long-term results.

If the short-term outcomes look positive, communicating this can improve morale and commitment. 

Step 5: Collect long-term outcome results

After a few cycles of short-term outcome evaluation it’s a good idea to review your long-term evaluation strategy. Are you still on track to measure what’s important to the success of this action or do you need to make adjustments to the data collected?

Ensure that the data collection process is working.

Step 6: Analyze results

Think critically about the evaluation results you have collected and develop a report. This report can be in the form of a podcast, video, slide presentation or publication, but make it easy for all to access and understand. Be transparent about whether your intended outcomes were achieved and include both challenges and successes.

Step 7: Present your results and decide next steps

The purpose of evaluation is continual improvement. Present your results to both employer and employees for feedback about what went well and what could be done differently. Make your decision about whether to refine, continue or discontinue the action in collaboration with others.

Change takes time and effort. Good evaluation processes help you measure and improve effectiveness. Remember to celebrate successes along the way!

Share this with anyone responsible for psychological health and safety at work.