Evaluation Summary, December 2022
The Office of the Auditor General of Canada's (OAG) 2021 Follow-up Audit on Rail Safety found that Transport Canada (TC) could not demonstrate the impact of its rail oversight activities on regulatory compliance or on rail safety overall.
In response, TC's Evaluation and Advisory Services conducted the Evaluation of Rail Safety Oversight Activities in 2021-22. We looked at the effect of oversight activities on compliance and on safety, as well as how rail regulators in other countries approach this question.
The evaluation emphasized quantitative evidence through two statistical analyses:
- A case study of Track inspection results from 2015-16 to 2021-22 to assess the performance of different inspection types and risk over time
- An analysis of the relationship between regulation and rail occurrences over time
To complement and enhance these analyses, we also conducted 58 key informant interviews with internal and external stakeholders and a detailed document and literature review.
On this page
What we found about rail safety's oversight effectiveness
Key metrics show that intended outcomes are being achieved. Non-compliance rates have mostly remained stable or decreased since 2015. Occurrence data shows that rail safety has improved overall in the last two decades.
Interview evidence suggests that oversight does contribute to compliance and through that impacts safety.
Interviews also found that education activities for industry stakeholders, though ad hoc, are a valued part of oversight.
There is evidence that in recent years risk-based inspections are achieving their goal of identifying higher risk non-compliances in comparison to random inspections. Identifying higher risk non-compliances should enable Rail Safety to help reduce more risk in the rail system. However, additional years of data are needed to confirm the trend, as the results are not statistically significant.
The average relative risk score per inspection for the Track functional group increased from 2016 to 2018 and decreased from 2018 to 2021. A decrease in risk scores over time would imply an improvement in safety. The decrease since 2018 could suggest a recent improvement in safety, but there are not enough years of data to identify a trend. As more years of data become available tracking risk over time could be used as an additional measure of safety.
There is evidence of a correlation between regulatory compulsion (e.g., words such as 'must' and 'shall') and improvements in safety. Beginning in 2015, as compulsory language in regulations increased, incidents decreased. This suggests that regulations are having an impact, and oversight — which checks for compliance to those regulations — should play a role in that relationship. While accidents have also decreased over time, this change was not shown to be statistically related to changes in regulatory compulsion.
How other countries measure oversight effectiveness
We compared TC to the Federal Railroad Administration (USA), the Department for Transport and Office for Rail and Road (UK), and the Office of the National Rail Safety Regulator (Australia).
We found that directly measuring the extent to which oversight activities contribute to safety outcomes is a shared challenge: While common metrics (e.g., accidents, injuries, fatalities) are used to track progress, none of these regulators have been able to quantitatively attribute safety outcomes to oversight activities. That said, TC can learn from these other countries in a number of ways, particularly in terms of data analytics, risk assessment and modelling, and safety management systems.
Recommendations
To enable the creation of appropriate metrics, Rail Safety should internally define "oversight effectiveness" and the ways by which its oversight activities contribute to this.
Rail Safety should consider expanding data collection to account for variables other than oversight that can impact intended outcomes. This would ensure relevant and good quality data is available for future quantitative analysis of oversight effectiveness.
Actions
- Rail Safety will work to define oversight effectiveness and the related activities.
- Rail Safety will develop a list of oversight effectiveness indicators to be considered for further development and implementation. The draft proposal will go to Program Business Committee for approval. (March 31, 2023)
- Final approval of the indicators by PBC will be requested in June 2023. The indicators will be adjusted over time as needed. (June 2023)
- Rail Safety will assess the various sources of our data and the methods in which they are collected, with the objective of improving the data available for use. (October 2023)
- Rail Safety will have dedicated staff in each region to review the quality of oversight data. (December 2023)
- Once the definition of oversight effectiveness is approved, Rail Safety will start collecting the data, and will build this data over time and review on a semi-annual basis. (April 2024)
- Rail Safety will work with Digital Services to have data from all sources included in the data lake so it can be reviewed in an integrated manner and enable a more holistic oversight story. (April 2024)