2021 Reports of the Auditor General of Canada to the Parliament of Canada
Independent Auditor’s Report Report 5—Follow-up Audit on Rail Safety—Transport Canada
Table of Contents
- Introduction
- Findings, Recommendations, and Responses
- Risk-based planning for oversight
- Conducting audits and inspections
- Transport Canada made progress in assessing and following up on railway companies’ corrective actions but did not measure the effectiveness of their safety management systems
- Good progress on assessing railway companies’ corrective actions
- No standard time frame for assessing whether railway companies completed corrective actions
- Improved information management for inspections but not for safety management system audits
- Effectiveness of safety management systems not assessed
- Transport Canada made progress in assessing and following up on railway companies’ corrective actions but did not measure the effectiveness of their safety management systems
- Conclusion
- About the Audit
- List of Recommendations
- Exhibits:
- 5.1—Percentage of rail accidents by type in 2019
- 5.2—Fatalities by occurrence type in 2019
- 5.3—Transport Canada’s oversight activities
- 5.4—Overview of how Transport Canada conducts rail safety inspections and safety management system audits
- 5.5—Over the past 14 years, several reports recommended that Transport Canada assess the effectiveness of railway companies’ safety management systems
Introduction
Background
Rail safety in Canada
5.1 Canada has a large and well-developed railway network that transports more than 88 million passengers and more than $328 billion worth of goods across the country each year.
5.2 Of all the railway networks in the world, Canada’s is one of the longest and transports the fifth largest volume of goods. Each year, Canadian railways move 70% of the country’s surface goods (including 40% of its exports). Railway tracks converge with motorists and pedestrians at thousands of public and private crossings.
5.3 Safety risks are inherent to all modes of transportation, and rail transportation is no exception. In 2019, there were 1,245 rail accidents (Exhibit 5.1), up from 1,169 in 2018. Total accidents in 2019 were 17% higher than the previous 10-year average of 1,064. The trend in accident occurrences over the past 10 years should be considered in the context of increasing rail traffic, which would suggest some improvement in the rate of accidents relative to rail traffic volume. Rail accidents include derailments and collisions, with the majority happening in train yards at low speeds and not on main-line railway tracks. Accidents may result in the unintended release of dangerous goods, serious injuries, or even deaths (Exhibit 5.2).
Exhibit 5.1—Percentage of rail accidents by type in 2019
Exhibit 5.1—text version
This pie chart shows the rail accidents by type in 2019.
In 2019, there were 1,245 accidents. Of those, from most to least common, there were
- 694 derailments (56% of accidents)
- 175 accidents that occurred at train crossings (14% of accidents)
- 115 collisions (9% of accidents)
- 56 trespasser accidents on tracks other than at railway crossings (5% of accidents): Trespasser accidents involve persons who are struck by trains on tracks (not at train crossings).
- 14 employee/passenger accidents (1% of accidents)
- 191 accidents that fell into other categories (15% of accidents)
Source: Based on data from Table 1—Railway occurrences and casualties, 2009–2019, Transportation Safety Board of Canada
Exhibit 5.2—Fatalities by occurrence type in 2019
Exhibit 5.2—text version
This pie chart shows the fatalities by type of railway accident in 2019.
In 2019, there were 72 fatalities. Of those,
- 38 were fatalities from trespasser accidents on tracks (53% of accidents)
- 28 were fatalities that occurred at train crossings (39% of accidents)
- 6 were fatalities caused by other accidents, including derailments and accidents involving employees or passengers (8% of accidents)
In 2019, there were no passenger fatalities.
Source: Based on data from Table 1—Railway occurrences and casualties, 2009-2019, Transportation Safety Board of Canada
5.4 Rail traffic has been increasing. Freight tonnage increased to more than 328 million tonnes in 2018, up from 312 million tonnes in 2017. In terms of goods transported, fuel oils and crude petroleum recorded a significant increase by weight from 2017 to 2018 of more than 45%. This surge in train traffic means more wear and tear on tracks, which can pose additional safety risks.
5.5 Safety risks are greater as more land is developed close to railway operations and as rail tracks expand into urban areas with road and pedestrian traffic. Rail safety can also be affected by extreme weather conditions and climate change—that can result in coastal storm surges, landslides and avalanches, and extremes in temperatures and precipitation.
5.6 All these factors make it even more important that Transport Canada, railway companies, and stakeholders work together to make rail transport as safe as possible for all Canadians.
Roles and responsibilities
5.7 Railways. Railway companies are responsible for the safety of their own operations pursuant to the Railway Safety Act and the accompanying Railway Safety Management System Regulations adopted in 2001. These were repealed and replaced in 2015. Each railway company is required to develop and follow its own safety management system in its day-to-day operations. These systems generally include a safety policy, safety targets, a risk-assessment process, and processes for continuous improvement.
5.8 Safety management systems complement and do not eliminate or replace other regulatory requirements. They are intended to enable railway companies to manage risks before Transport Canada intervenes and before major safety issues emerge.
5.9 Transport Canada. Under the Railway Safety Act, Transport Canada is responsible for overseeing safety requirements for federally regulated railway companies. The department’s oversight role includes monitoring compliance and enforcement activities.
5.10 In all, 75 railway companies and more than 45,000 kilometres of railway tracks in Canada are subject to Transport Canada’s oversight. These railway companies include both federal companies operating across provincial or international borders and local railway companies operating on federally regulated tracks.
Transport Canada rail inspectors examine the railway infrastructure to determine if it complies with safety regulations.
Photo: Transport Canada
5.11 To mitigate safety threats, Transport Canada oversees rail safety from its headquarters in Ottawa and its 5 regional offices through activities such as
- inspections
- safety management system audits
- data analytics
5.12 Stakeholders. While railway companies are responsible for rail safety, municipal road authorities are responsible for maintaining the approaches to public crossings for motorists and pedestrians. Stakeholders include provinces and territories, rail safety associations, labour unions, and the general public.
What we found in our 2013 audit
5.13 This audit followed up on selected recommendations from our 2013 Fall Report of the Auditor General of Canada, Chapter 7—Oversight of Rail Safety—Transport Canada. In that audit, we concluded that Transport Canada needed to address significant weaknesses in its oversight of safety management systems implemented by federal railway companies to manage safety risks on a day-to-day basis. We recommended that the department
- improve its risk-based planning of oversight of railway companies, including collecting relevant risk and safety information from railways
- conduct timely follow-up on deficiencies affecting the safety of railway companies’ operations, including assessing the effectiveness of railway companies’ safety management systems
Focus of the audit
5.14 This audit focused on whether Transport Canada implemented selected recommendations from our 2013 audit regarding the department’s oversight of the safe transportation of people and goods on federally regulated tracks. In particular, we examined the risk-based planning process for rail safety oversight and whether the department confirmed that railway companies corrected deficiencies affecting the safety of their operations.
5.15 This audit is important because rail safety accidents can have serious consequences, including the loss of life and damage to property, goods, and the environment.
5.16 We did not examine the inspections of facilities and sites that handle, offer for transport, transport, or import dangerous goods. We also did not examine the emergency response assistance plans for companies regulated by the Transportation of Dangerous Goods Act, 1992. On these topics, we published a separate follow-up audit report—see our 2020 Fall Reports of the Commissioner of the Environment and Sustainable Development to the Parliament of Canada, Report 1—Follow- up Audit on the Transportation of Dangerous Goods. In addition, we did not examine the department’s enforcement activities, the Transportation Safety Board of Canada, railway companies, provinces, territories, or municipalities. Finally, our audit was not designed to conclude on whether individual railways in Canada are safe.
5.17 More details about the audit objective, scope, approach, and criteria are in About the Audit at the end of this report.
Findings, Recommendations, and Responses
Overall message
5.18 Overall, Transport Canada made progress in addressing recommendations from our 2013 audit in the areas we followed up on; however, we also found that the department still had important improvements to make. The department strengthened its approach to planning and prioritizing activities to oversee rail safety and in following up on companies’ plans and actions to address deficiencies noted in inspections.
5.19 However, Transport Canada was unable to show whether departmental oversight activities have contributed to improved rail safety. In addition, the department did not assess the effectiveness of the railways’ safety management systems—despite the many reports over the last 14 years recommending that Transport Canada audit and assess these systems.
Risk-based planning for oversight
Transport Canada could not demonstrate that its oversight planning activities improved railway companies’ regulatory compliance
What we found
5.20 We found that Transport Canada improved its risk-based planning for oversight. In particular, the department significantly increased the number of its planned risk-based inspections. We also found that Transport Canada made progress in conducting more audits of railway companies’ safety management systems.
5.21 However, we found that the department did not measure the overall effectiveness of its oversight activities. Properly assessing the effectiveness of the department’s oversight activities is complex because other factors, such as new technology, can also improve rail safety. Nonetheless, it remains crucial to determine whether the department’s oversight is having a positive impact. We found that the department could not demonstrate the extent to which its oversight activities have improved the railway companies’ compliance with regulations that mitigate key safety risks. This is an important element of measuring its oversight effectiveness.
5.22 We also found that although the department began to implement our recommendation to collect more information from railway companies to support risk-based decision making, it did not provide sufficiently detailed guidance to the railway companies on how to report their safety data.
5.23 We found that although the department completed inspections and collected data from multiple sources, including the Transportation Safety Board of Canada, it did not adjust the selection of its planned inspections to take into account the results from safety management system audits.
5.24 The analysis supporting this finding discusses the following topics:
- Improved risk-based planning process
- Overall effectiveness of oversight activities not measured
- Inadequate information to plan for inspections
- Audit results of safety management systems not considered
Why this finding matters
Context
5.25 This finding matters because incorporating better data into its planning activities will help Transport Canada address the areas of highest risk and better support the safe transportation of people and goods.
5.26 Transport Canada’s rail oversight program is planned annually and adjusted as needed. The department monitors compliance through inspections and audits (Exhibit 5.3) and takes necessary enforcement action.
Exhibit 5.3—Transport Canada’s oversight activities
Transport Canada’s Rail Safety Oversight Program
- Full-time equivalent staff: 149
- Budget: $24.2 million
|
Inspections |
Audits of safety management systems |
|
Requirements for railway company compliance are set out in the Railway Safety Act and in the associated regulations and standards. |
According to the Railway Safety Management System Regulations, 2015, railway companies are required to implement a system for managing the safety of their day-to-day operations. This system should include policies, plans, and procedures to manage and reduce safety risks. |
|
Inspections are technical examinations of equipment, operations, crossings, signals, tracks, bridges, and natural hazards. |
Auditors review selected elements of the safety management system (such as accountability and risk assessment processes). Transport Canada audits these systems to determine regulatory compliance and whether the railway companies have developed and implemented a system to manage safety. |
|
In the 2019–20 fiscal year, 40,530 inspections were completed. |
In the 2019–20 fiscal year, 19 safety management system audits were completed. |
Source: Based on information provided by Transport Canada
5.27 Because the department cannot oversee each railway location, kilometre of track, piece of equipment, or train crew, it relies on risk- based oversight planning. The department develops these plans to determine the audits and inspections to conduct in order to assess whether railway companies manage safety risks appropriately and comply with regulations.
5.28 Transport Canada’s risk-based planning process identifies and prioritizes where inspections and audits may be required. The department’s annual planning process starts with data collection from a variety of sources:
- Transportation Safety Board of Canada incident reports and accident reports
- Transportation Information Regulations data provided by the railway companies, which includes information on the condition and maintenance of locomotives and train cars, and information relevant to railway infrastructure, including bridges, tracks, culverts, signalling, and grade crossings
- railway companies’ past inspection results, enforcement actions, and public complaints and inquiries
5.29 Our 2013 Fall Report of the Auditor General of Canada, Chapter 7—Oversight of Rail Safety—Transport Canada recommended that Transport Canada improve its planning for risk-based oversight activities to
- collect relevant risk and safety performance information from federal railways and assess its completeness and reliability (recommendation 7.42)
- reassess the number of its planned railway audits and inspections to take into account the safety management system environment (recommendation 7.49)
Recommendations
5.30 Our recommendations in this area of examination appear at paragraphs 5.37, 5.41, and 5.45.
Improved risk-based planning process
Analysis to support this finding
5.31 We found that since our 2013 audit, Transport Canada made progress in improving its risk-based approach to identifying and overseeing safety issues. In the 2016–17 fiscal year, 40% of inspections were risk-based, and this proportion increased to 72% in the 2019–20 fiscal year. The department also conducted random inspections (15%) to measure general levels of compliance and reactive inspections (13%) to assess emerging issues. The proportion of risk-based inspections relative to random inspections was determined by the department.
5.32 We also found that each regional office followed different planning practices to prioritize their oversight activities. Although Transport Canada identified risks for rail safety at both the national and regional levels, the department’s risk assessment was inconsistent across regions and was not always adequately documented, even though the department provided guidance to regional offices on how to identify areas of highest risk. The department was aware of this situation and acknowledged that it needed to work with the regional offices to assess risk consistently.
Overall effectiveness of oversight activities not measured
5.33 We found that Transport Canada did not measure the overall effectiveness of its rail safety oversight. It had not identified whether rail safety has improved as a result of its inspections and audits of safety management systems.
5.34 The department reported publicly on a 10-year rail accident rate and a 10-year rail fatality rate. Although the department measured the number of accidents and fatalities, these measures did not specifically assess the impact of the department’s oversight on the safety of rail transportation. Moreover, reductions in accidents and fatality rates depend on collaboration among many parties, including railway companies and municipal road authorities, which share responsibility to help reduce these rates. Transport Canada also developed internal performance indicators, such as 5-year average compliance rate trends in key risk areas, but it did not consistently measure performance against them.
Transport Canada’s risk-based planning process identifies and prioritizes where inspections and audits may be required.
Photo: Transportation Safety Board of Canada, Investigation R16W0059
5.35 We also found that Transport Canada senior management received only partial information on the results of oversight activities to make informed risk-based planning decisions. Although the department’s senior management received year-end reports, the following information to support decision making was not included:
- sufficient data on railways’ compliance with regulations
- results of departmental audits of railways’ safety management systems
- inspection results—departmental data focused instead on the number of inspections performed
5.36 Without clear information about the outcomes and impacts of the department’s safety oversight activities, the department has difficulty making risk-based planning decisions so that inspectors and auditors focus on the highest-risk areas that can improve rail safety.
5.37 Recommendation. Transport Canada should determine the extent to which its inspections and audits have improved the railway companies’ compliance with regulations that mitigate key safety risks. The department should integrate these results into its risk-based planning decisions, including determining which and how many inspections to perform.
The department’s response. Agreed. Transport Canada will significantly improve the ongoing performance measurement of the rail safety program by improving the Rail Safety Oversight Performance Information Profile. This effort will be informed and supplemented by other work in the department, including the development of an outcome measurement system and a safety and security data strategy (February 2022).
Transport Canada will also conduct an evaluation to assess the impact of oversight activities on the railway companies’ compliance with key regulations (December 2022).
These results will be used to improve risk-based planning (June 2023).
Inadequate information to plan for inspections
5.38 We found that Transport Canada began to implement our 2013 recommendation to collect relevant risk and safety performance information from federal railways. The department amended the Transportation Information Regulations requiring railway companies to submit specific safety data so the department could proactively identify risk areas—and incorporate this information into its planning.
5.39 However, we also found that the department did not provide sufficiently detailed guidance to the railway companies on how to report their safety data. As a result, the information was sometimes late, incomplete, or of varying quality.
5.40 In particular, the department did not obtain all the required information needed to plan inspections for areas of highest risk. Departmental staff identified several concerns, such as limited information on how a signal malfunction was resolved, and varying interpretations of the content required for some indicators. Staff also noted that the annual deadline, as specified in the regulations for railway companies’ data submissions, was not always respected and that some submissions were received later than the due date.
5.41 Recommendation. Transport Canada should improve guidance to railway companies so that safety data submissions are complete, reliable, and timely.
The department’s response. Agreed. Transport Canada will develop an information request template (September 2021) and issue guidelines to railway companies and stakeholders (October 2021) to facilitate data analysis and streamline reporting.
Audit results of safety management systems not considered
5.42 In the 8 years since we recommended that Transport Canada reassess the number of its planned audits, we found that the department made progress in conducting more audits of railways’ safety management systems. According to the department, it audited all the railway companies in operation at least once between 1 April 2016 and 31 March 2020.
5.43 However, we also found that Transport Canada’s oversight planning process did not consider findings from departmental audits of the railways’ safety management systems. Our examination of these audits identified results that the department could have used to focus its inspection planning on areas of highest risk. For example, one audit revealed that a rail company did not collaborate with its employees when developing a hazard management program and a policy on whistleblowing. Another audit discovered that a railway company did not have procedures or documentation to show that it provided employees with the required skills, knowledge, and qualifications to conduct their work.
5.44 If safety management system audit findings had been included in Transport Canada’s planning oversight process, the department’s annual inspection focus may have shifted. In our view, it is important that the department integrate the results of its audit activities into the risk-based planning for rail safety oversight, so that it considers all available information.
5.45 Recommendation. Transport Canada should integrate the findings from audits of safety management systems when planning its oversight activities and adjust its selection of planned inspections to address areas of highest risk.
The department’s response. Agreed. Transport Canada will identify and adapt best practices to integrate safety management system audits into the selection of oversight activities for risk-based planning purposes. The risk-based business planning procedure will be updated to incorporate a review of audit findings by April 2021.
Auditors and inspectors who take part in safety management system audits will also participate in risk-based business planning activities for the various functions to highlight and explain audit findings.
These measures will be completed by May 2021.
Conducting audits and inspections
Transport Canada made progress in assessing and following up on railway companies’ corrective actions but did not measure the effectiveness of their safety management systems
What we found
5.46 We found that since our 2013 audit, Transport Canada made progress in assessing whether railway companies corrected safety issues identified in departmental inspections and audits of safety management systems.
5.47 However, we also found that Transport Canada did not establish standard time frames for how long the department should take to assess whether railway companies corrected the identified safety issues.
5.48 We also found that the department did not assess the effectiveness of the safety management systems that each railway implemented. Departmental audits of these systems determined only whether railway companies complied with specific regulations and not whether the overall systems themselves resulted in greater safety.
5.49 The analysis supporting this finding discusses the following topics:
- Good progress on assessing railway companies’ corrective actions
- No standard time frame for assessing whether railway companies completed corrective actions
- Improved information management for inspections but not for safety management system audits
- Effectiveness of safety management systems not assessed
Why this finding matters
5.50 This finding matters because safety management systems are designed to achieve the highest level of safety in day-to-day rail operations. Unless Transport Canada assesses the effectiveness of these systems and follows up within a standard time frame to determine whether railway companies have corrected the identified safety issues, it cannot demonstrate that the likelihood of accidents is minimized.
Context
5.51 As a key component of its oversight activities, Transport Canada assesses whether the railway companies’ corrective actions resolve safety issues. It is critical that these assessments be timely so that high-risk issues are prioritized and resolved before accidents can occur.
5.52 Measuring the effectiveness of safety management systems is another important component of the department’s oversight responsibilities. Auditing effectiveness gives Transport Canada assurance that railway companies are integrating safety into their day-to-day operations.
5.53 This audit included follow-up on recommendation 7.58 from our 2013 audit of Transport Canada to
- require railway companies to make the necessary changes to correct deficiencies that affect the safety of their operations
- conduct timely follow-up on deficiencies affecting the safety of railway companies’ operations to assess whether they were corrected
- •improve departmental oversight of railway companies’ safety management systems by assessing their quality and effectiveness
Recommendations
5.54 Our recommendations in this area of examination appear at paragraphs 5.61, 5.64, and 5.68.
Good progress on assessing railway companies’ corrective actions
Analysis to support this finding
5.55 We found that for most of the inspections and audits we examined, Transport Canada assessed whether railway companies took corrective actions when non-compliances were identified. In our view, Transport Canada’s process to validate the railway companies’ plans to address problems was reasonable.
5.56 We used representative sampling to examine 66 inspections conducted between September 2018 and August 2020 that required a railway to take follow-up action. We found that the department confirmed that railway companies took corrective actions in 61 (92%) of those inspections. For the remaining 5 (8%), Transport Canada determined that the railway companies’ plans to address minor issues, such as trimming grasses and shrubs near crossings, would satisfy regulatory requirements.
5.57 We also randomly selected 24 of Transport Canada’s safety management system audits requiring a railway company to correct safety issues between September 2018 and August 2020. As of December 2020, the department was still in the process of assessing corrective action plans for 7 of these audits.
5.58 For the remaining 17 audits, the department assessed that railways had corrected all identified issues for 10, and it was in the process of confirming that corrective actions were implemented for the remaining 7. In our opinion, Transport Canada’s assessments of
corrective actions taken were reasonable. Compared with inspections, safety management system audits are broader in scope, and the identified issues may be more complex. Exhibit 5.4 presents an overview of how Transport Canada conducts rail safety inspections and safety management system audits.
Exhibit 5.4—Overview of how Transport Canada conducts rail safety inspections and safety management system audits
|
How does Transport Canada conduct a rail safety inspection? |
How does Transport Canada audit a railway company’s safety management system? |
|
1. Transport Canada inspectors conduct on-site inspections to verify compliance with safety requirements. |
1. Transport Canada auditors conduct on-site visits to verify that processes and procedures support safety. |
|
2. Transport Canada identifies issues in railway company operations, such as
|
2. Transport Canada identifies weaknesses in a railway company’s safety management systems, such as
|
|
3. The railway company is responsible for making the necessary changes and providing Transport Canada with corrective actions, such as
|
3. The railway company is responsible for making the necessary changes and providing Transport Canada with corrective action plans, such as
|
|
4. Transport Canada reviews the railway company’s corrective actions, which detail actions taken or planned. Transport Canada then communicates whether it is satisfied with the actions taken or if the railway company needs to do more. |
4. Transport Canada reviews the railway company’s corrective actions, which detail actions taken or planned. Transport Canada then communicates whether it is satisfied with the actions taken or if the railway company needs to do more. |
|
Time frame * Average: 45 days Median: 25 days |
Time frame * Average 169 days Median 135 days |
|
5. Transport Canada follows up with the railway company to confirm that corrective action has been taken. The time required for the railway company to take corrective action varies depending on the safety risk and the complexity of the issues identified. |
5. Transport Canada follows up with the railway company to confirm that corrective action has been taken. The time required for the railway company to take corrective action varies depending on the safety risk and the complexity of the issues identified. |
|
* Results of our file examination Source: Based on information provided by Transport Canada |
|
No standard time frame for assessing whether railway companies completed corrective actions
5.59 To determine whether Transport Canada fulfilled its requirement to provide timely follow-up on safety issues, we examined samples of the department’s inspections and audits conducted between September 2018 and August 2020. Although Transport Canada performed follow-up activities, we found that the department had no standard time frame for how long it should take to assess whether railway companies completed corrective actions.
5.60 The department’s standards for on-site audits and inspections focused primarily on how quickly the railway companies needed to provide a plan to correct the issues—and not on confirming that railway companies made timely corrections to the deficiencies affecting safety.
5.61 Recommendation. Transport Canada should set standards for the time it should take to assess whether railway companies corrected deficiencies identified in inspections and audits of safety management systems.
The department’s response. Agreed. By June 2021, Transport Canada will identify best practices, including those in other modes of transportation, to establish standards for following up on inspections and safety management system audits. The department will establish a standard by December 2021 on how long it should take to assess whether railway companies corrected deficiencies resulting from inspections and audits of safety management systems.
Improved information management for inspections but not for safety management system audits
5.62 We found that Transport Canada updated its information management system to better manage inspections. Refinements included tracking railway companies’ corrective actions, inspection results, and departmental communication with railway companies. The system also generated reports for management.
5.63 However, we found that the department did not integrate information and results from its safety management system audits into its information database, nor did it have another way to track this data. Furthermore, Transport Canada did not adhere consistently to its documentation standards, which made it difficult to determine whether the department uniformly assessed the railway companies’ safety management systems. For example, the department used a standard checklist in only 6 of the 24 audits we examined.
5.64 Recommendation. Transport Canada should improve data management for its safety management system audits by
- adhering to its documentation standards
- integrating the results of its safety management system audits into a data management system
The department’s response. Agreed. Transport Canada has confirmed that the audit teams in every region are using the recently implemented electronic working papers system to manage audit documentation (December 2020). In addition, the department will conduct a review of data management practices for audits by April 2021 to ensure consistent use across regions. A data management system will be developed to facilitate the tracking of audit findings and identify compliance trends. This system will be in place by December 2021.
Effectiveness of safety management systems not assessed
5.65 Safety management systems emerged in the 1990s as a proactive way to manage safety. Starting in 2001, the Railway Safety Management System Regulations required that railway companies implement and
maintain a safety management system that included, among other things, goals, risk assessments, and monitoring and evaluation processes. The regulations also required that railway companies collect and provide the department with data annually for assessing their safety performance.
In 2015, the department repealed and replaced the regulations. Collectively, these legislative provisions supported our view that Transport Canada’s oversight role included assessment of the effectiveness of these safety management systems.
5.66 Over the past 14 years, several reports advised Transport Canada to assess the effectiveness of railway companies’ safety management systems (Exhibit 5.5). In particular, the 2007 review of the Railway Safety Act recommended that the department integrate the assessment of safety management systems into its oversight activities. Furthermore, the department agreed to audit safety management system effectiveness in response to the Transportation Safety Board of Canada’s Lac-Mégantic investigation report in 2014 on runaway trains and main track derailments.
Exhibit 5.5—Over the past 14 years, several reports recommended that Transport Canada assess the effectiveness of railway companies’ safety management systems
|
2001 |
The Railway Safety Management System Regulations came into force. |
|
2007 |
The Railway Safety Act review Footnote 1 recommended that Transport Canada integrate assessments of safety management systems into its oversight activities. |
|
2008 |
The Standing Committee on Transport, Infrastructure and Communities recommended that Transport Canada and the railway companies develop a concrete assessment tool to continually monitor the implementation of safety management systems. |
|
2013 |
We audited rail safety oversight, and Transport Canada committed to planning an increased number of audits and more fully integrating the assessment of the safety management system environment into its risk-based oversight activities. Amendments to the Railway Safety Act included provisions to expand the use of safety management systems. |
|
2014 |
The Standing Committee on Transport, Infrastructure and Communities recommended in its Interim Report on Rail Safety Review that Transport Canada better define audit methodology for safety management systems. In response to the Lac-Mégantic accident investigation, Transport Canada agreed to fully implement the Transportation Safety Board of Canada’s recommendations, including providing guidance to inspectors on conducting audits of safety management systems and on the elements of an effective safety management system. |
|
2015 |
As a result of an internal audit report, Transport Canada committed to measuring the effectiveness of safety management systems against defined performance indicators and objectives. The Railway Safety Management System Regulations were updated. |
|
2016 |
The Standing Committee on Transport, Infrastructure and Communities recommended that Transport Canada prioritize inspections for rail operations with a record of poor performance in terms of developing and implementing effective safety management systems. |
|
2018 |
The Railway Safety Act review Footnote 2 recommended strengthening safety management systems to ensure a greater focus on effectiveness and safety outcomes. |
|
Source: Standing Committee on Transport, Infrastructure and Communities; Transport Canada; and Transportation Safety Board of Canada |
|
5.67 We found that the department’s audits continued to focus on whether safety management systems complied with regulatory requirements and not whether these systems were effective and actually improved rail safety. Determining whether the supporting processes are effective is an important next step for the department in assessing the overall effectiveness of railway companies’ safety management systems.
5.68 Recommendation. Transport Canada should make it an immediate priority to regularly assess the effectiveness of railway companies’ safety management system processes.
The department’s response. Agreed. Transport Canada will undertake program and regulatory action to assess the effectiveness of safety management system processes:
- Develop and launch a targeted audit framework and supporting implementation methodology, which will be used as a guide to identify issues that may be a result of ineffectiveness of a particular railway company’s safety management system processes (April 2021).
- Finalize a risk-based process for determining when targeted audits are required (June 2021).
- Develop tools to measure effectiveness and identify baseline and targets for each safety management system process (November 2021).
- Undertake the audits of effectiveness that leverage the tools developed (December 2021).
- Launch consultations in April 2022 on potential amendments to the safety management system regulations to strengthen rail safety requirements, including measures to improve the overall performance of safety management systems.
Conclusion
5.69 We concluded that Transport Canada had not fully implemented all of the selected recommendations from our 2013 audit report on rail safety regarding the department’s oversight of the safe transportation of people and goods.
5.70 The department made important improvements to both its safety oversight planning and inspection follow-up, but it did not assess whether its oversight activities improved rail safety. Also, the department did not measure the effectiveness of the railway companies’ safety management systems or integrate the results of those audits into its rail safety oversight planning.
About the Audit
This independent assurance report was prepared by the Office of the Auditor General of Canada (OAG) on the oversight of rail safety. Our responsibility was to provide objective information, advice, and assurance to assist Parliament in its scrutiny of the government’s management of resources and programs, and to conclude on whether the oversight of rail safety complied in all significant respects with the applicable criteria.
All work in this audit was performed to a reasonable level of assurance in accordance with the Canadian Standard on Assurance Engagements (CSAE) 3001—Direct Engagements, set out by the Chartered Professional Accountants of Canada (CPA Canada) in the CPA Canada Handbook—Assurance.
The Office of the Auditor General of Canada applies the Canadian Standard on Quality Control 1 and, accordingly, maintains a comprehensive system of quality control, including documented policies and procedures regarding compliance with ethical requirements, professional standards, and applicable legal and regulatory requirements.
In conducting the audit work, we complied with the independence and other ethical requirements of the relevant rules of professional conduct applicable to the practice of public accounting in Canada, which are founded on fundamental principles of integrity, objectivity, professional competence and due care, confidentiality, and professional behaviour.
In accordance with our regular audit process, we obtained the following from entity management:
- confirmation of management’s responsibility for the subject under audit
- acknowledgement of the suitability of the criteria used in the audit
- confirmation that all known information that has been requested, or that could affect the findings or audit conclusion, has been provided
- confirmation that the audit report is factually accurate
Audit objective
The objective of this audit was to determine whether Transport Canada had implemented selected recommendations from our 2013 audit report on the oversight of rail safety regarding the department’s oversight of the safe transportation of people and goods.
Scope and approach
Audit scope. We examined whether Transport Canada made progress on its oversight of the safety of railway companies operating on federally regulated tracks. Specifically, we addressed recommendations from the 2013 audit report to
- review the risk-based business planning methodology to identify key safety risk and performance indicators
- collect the relevant risk and safety performance information from railway companies and assess its completeness and reliability
- develop an approach to make better use of this information when preparing annual oversight plans
- reassess the number of planned audits and inspections so that the department takes into account the new safety management system environment and conducts the minimum level of oversight on the basis of this assessment
- improve its oversight of federal railways’ safety management systems by assessing their quality and effectiveness
- require railway companies to make the necessary changes to correct deficiencies affecting the safety of their operations
- conduct timely follow-up on deficiencies affecting the safety of railways’ operations to assess whether they were corrected
Finally, we examined whether Transport Canada measured its oversight performance and used the resulting information to make informed risk-based planning decisions.
Audit approach. We examined the methodology, process, and information the department collected to plan its oversight activities for the 2019–20 fiscal year. We reviewed the documentation supporting the development of the national oversight plan, including the information used by the 5 regional offices in regards to the following functions: crossing, equipment, operation, signal, and track.
We also examined Transport Canada’s oversight of railways through its inspections and audits conducted between 1 September 2018 and 31 August 2020, especially how the department followed up to verify that the railway companies made necessary changes to correct the issues identified. For inspections, we used representative sampling to examine 66 inspection files that identified instances of non-compliance and concerns. These samples were sufficient in size to project to the sampled population with a confidence level of 90% and a margin of error (confidence interval) of +10%.
For Transport Canada’s audits of railways’ safety management systems, we used random sampling to examine 24 audit files that identified instances of non-compliance and deficiencies. We examined whether follow-up compliance activities were conducted according to regulatory requirements and Transport Canada’s procedures to confirm that the railway companies took corrective actions. We reviewed inspection reports and audit reports that indicated problems.
We also examined the department’s communication with the railway companies, Transport Canada’s assessment of the railway companies’ corrective actions, and documentation signed by the departmental staff indicating their satisfaction or dissatisfaction with the railway companies’ progress.
We did not examine the inspections of facilities and sites that handle, offer for transport, transport, or import dangerous goods. We also did not examine the emergency response assistance plans for companies regulated by the Transportation of Dangerous Goods Act, 1992. On these topics, we published a separate follow-up audit report—see our 2020 Fall Reports of the Commissioner of the Environment and Sustainable Development to the Parliament of Canada, Report 1—Follow-up Audit on the Transportation of Dangerous Goods.
In addition, we did not examine the Transportation Safety Board of Canada, railway companies, provinces, or municipalities. Finally, our audit was not designed to conclude on whether individual railways in Canada are safe.
Criteria
We used the following criteria to determine whether Transport Canada had implemented selected recommendations from our 2013 audit report on the oversight of rail safety regarding the department’s oversight of the safe transportation of people and goods:
|
Criteria |
Sources |
|
Transport Canada’s annual oversight plan is risk-based, and the plan is actioned accordingly. |
|
|
Transport Canada measures the effectiveness of its oversight activities against pre- established targets to make informed risk- based planning decisions. |
|
|
Transport Canada assesses whether railway companies have addressed non-compliances and threats to safety identified through the department’s inspections and audits. |
|
|
Transport Canada audits railway companies to assess the effectiveness of their safety management systems. |
|
Period covered by the audit
The audit covered the period from 1 September 2018 to 31 August 2020. This is the period to which the audit conclusion applies. However, to gain a more complete understanding of the subject matter of the audit, we also examined certain matters that preceded the start date of this period.
Date of the report
We obtained sufficient and appropriate audit evidence on which to base our conclusion on 13 January 2021, in Ottawa, Canada.
Audit team
Principal: Dawn Campbell
Director: Carey Agnew
Younes Boukhaffa Alina Dan Raylene MacLellan Isabelle Marsolais Catherine Martin
List of Recommendations
The following table lists the recommendations and responses found in this report. The paragraph number preceding the recommendation indicates the location of the recommendation in the report, and the numbers in parentheses indicate the location of the related discussion.
|
Recommendation |
Response |
|
5.37 Transport Canada should determine the extent to which its inspections and audits have improved the railway companies’ compliance with regulations that mitigate key safety risks. The department should integrate these results into its risk-based planning decisions, including determining which and how many inspections to perform. (5.31 to 5.36) |
The department’s response. Agreed. Transport Canada will significantly improve the ongoing performance measurement of the rail safety program by improving the Rail Safety Oversight Performance Information Profile. This effort will be informed and supplemented by other work in the department, including the development of an outcome measurement system and a safety and security data strategy (February 2022). Transport Canada will also conduct an evaluation to assess the impact of oversight activities on the railway companies’ compliance with key regulations (December 2022). These results will be used to improve risk-based planning (June 2023). |
|
5.41 Transport Canada should improve guidance to railway companies so that safety data submissions are complete, reliable, and timely. (5.38 to 5.40) |
The department’s response. Agreed. Transport Canada will develop an information request template (September 2021) and issue guidelines to railway companies and stakeholders (October 2021) to facilitate data analysis and streamline reporting. |
|
5.45 Transport Canada should integrate the findings from audits of safety management systems when planning its oversight activities and adjust its selection of planned inspections to address areas of highest risk. (5.42 to 5.44) |
The department’s response. Agreed. Transport Canada will identify and adapt best practices to integrate safety management system audits into the selection of oversight activities for risk-based planning purposes. The risk-based business planning procedure will be updated to incorporate a review of audit findings by April 2021. Auditors and inspectors who take part in safety management system audits will also participate in risk-based business planning activities for the various functions to highlight and explain audit findings. These measures will be completed by May 2021. |
|
Recommendation |
Response |
|---|---|
|
5.61 Transport Canada should set standards for the time it should take to assess whether railway companies corrected deficiencies identified in inspections and audits of safety management systems. (5.59 to 5.60) |
The department’s response. Agreed. By June 2021, Transport Canada will identify best practices, including those in other modes of transportation, to establish standards for following up on inspections and safety management system audits. The department will establish a standard by December 2021 on how long it should take to assess whether railway companies corrected deficiencies resulting from inspections and audits of safety management systems. |
|
5.64 Transport Canada should improve data management for its safety management system audits by
(5.62 to 5.63) |
The department’s response. Agreed. Transport Canada has confirmed that the audit teams in every region are using the recently implemented electronic working papers system to manage audit documentation (December 2020). In addition, the department will conduct a review of data management practices for audits by April 2021 to ensure consistent use across regions. A data management system will be developed to facilitate the tracking of audit findings and identify compliance trends. This system will be in place by December 2021. |
|
5.68 Transport Canada should make it an immediate priority to regularly assess the effectiveness of railway companies’ safety management system processes. (5.65 to 5.67) |
The department’s response. Agreed. Transport Canada will undertake program and regulatory action to assess the effectiveness of safety management system processes:
|