Audit of Risk-Based Business Planning in Safety and Security

Audit and Advisory Services

Integrity, Innovation and Quality

February 2017

Table of Contents

EXECUTIVE SUMMARY

• Statement of Conformance

1. INTRODUCTION

• 1.1. Purpose
• 1.2. Background
• 1.3. Audit Objective, Scope and Approach
  • 1.3.1. Audit Objective
  • 1.3.2. Audit Scope
  • 1.3.3. Audit Approach
  • 1.4. Report Structure

2. FINDINGS AND RECOMMENDATIONS

• 2.1. Risk Assessment, Cost-Efffective Intervention and Oversight Strategies
• 2.2. Activity Costing and Resource Allocation
• 2.3. Governance

3. MANAGEMENT RESPONSE AND ACTION PLAN

MODAL PROFILES

EXECUTIVE SUMMARY

The Audit of Risk-Based Business Planning in Safety and Security focuses on Civil Aviation, Marine Safety and Security, Rail Safety, and the Transportation of Dangerous Goods for the planning cycles of 2015-16 and 2016-17. Our work builds on both external and internal audit findings over the past five years and examines the extent to which risk assessment directly supports the selection of appropriate interventions and the associated oversight activities as well as the capability to cost these activities and report on results.

We recognize that advances have been made over the past few years in the area of risk-based business planning in Safety and Security. Most notably, the introduction and continued refinement of an annual National Oversight Plan has created a vehicle through which programs are expected to identify risks, consider how these risks can best be mitigated through oversight activities, and track progress throughout each fiscal year. These changes are starting to provide senior management with a tool to manage and oversee program and regional business decisions in a manner that was not possible before.

There are also several incremental and longer-term initiatives currently underway. In particular, Safety and Security is working to introduce more rigour into environmental scanning, identification of risks to the transportation system, and costing of activities. As a longer-term measure, an oversight modernization initiative will be launched to improve risk identification and analysis, reform oversight policies and procedures to better tailor responses to risk, and better evaluate the efficiency and effectiveness of oversight regimes.

The audit findings highlight existing gaps and opportunities to help inform this work to modernize the regulatory and oversight regime.

Overall we found that risk assessment methodologies and processes are at different stages of maturity across the examined programs. Issues remain in ensuring consistency nationally and modally. Approaches to risk assessment methodologies and processes differ fundamentally such that they do not generate information that supports modal, cross-modal or national comparisons. Moreover, the programs have not clearly defined risk tolerance levels and are choosing interventions based mainly on available resources rather than on those that could be used to reduce risks to an acceptable tolerance level.

Because there is no standard approach to costing activities and given the lack of data quality and analytics, we found that the programs are not able to accurately cost their various activities. While we acknowledge that many programs have started work in this area, it is not being done consistently across modes and is not yet ready to be used for comparison between regions or to fully support resource allocation decisions.

We have made five recommendations stemming from these findings and management has developed a comprehensive plan to make the necessary changes to improve the maturity of risk-based planning in Safety and Security.

STATEMENT OF CONFORMANCE

This Audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of an external assessment of Internal Audit’s Quality Assurance and Improvement Program.

Dave Leach (CIA, MPA) Director, Audit and Advisory Services

Martin Rubenstein (CPA, CIA, CFE) Chief Audit and Evaluation Executive

1. INTRODUCTION

1.1. PURPOSE

The Audit of Risk-Based Business Planning in Safety and Security was included in Transport Canada’s Risk-Based Audit Plan for 2015-16 with the objective of assessing if the planning and reporting process adequately defines the activities, resources and progress needed to effectively deliver modal programs in Safety and Security.

Within Safety and Security, risk-based business planning enables modal programs to ensure that resources are available and directed to the areas of highest risk and priority and that progress can be actively monitored and reported to senior management in support of timely decision making. Given that Safety and Security represents 60% of the Department’s human resources, risk-based business planning is critical to sound budgeting and resource allocation decisions in the department.

To determine the adequacy of the business planning process the audit examined:

1. the risk-assessment process i.e., how risks are identified and assessed, how intervention options (for example education and outreach vs. regulations) are selected and how oversight activities align with the risk strategy; and
2. how the modes cost oversight and allocate resources.

Changes introduced to the risk-based business planning process during the conduct of the audit for the 2016-17 planning cycle are taken into account. The audit also builds on findings from previous internal and external audits which assessed how the department plans and allocates its resources, particularly in conjunction with its oversight functions.

1.2. BACKGROUND

Over the course of the last five years, there has been extensive external-audit focus on Transport Canada’s Safety and Security programs. A few of the overarching audit findings from these audits include the following:

• To determine the sites of highest priority for inspection, the department requires a national risk-based process (see CESD Audit of the Transportation of Dangerous Products 2011).
• To provide assurance that industry complies with the existing regulatory framework, the department requires an effective risk-based surveillance regime (see OAG Audit of Civil Aviation Oversight 2012).
• To facilitate a consistent approach to oversight, the department requires a methodology that uses uniform risk and performance indicators (see OAG Audit of Rail Safety Oversight 2013).

These external audit findings are consistent with Transport Canada’s recent internal Audit of Quality Management Systems for Oversight Programs (2015) which likewise concluded that “an effective risk-based inspection/audit planning and reporting process that integrates business needs with operational needs is required across all oversight programs.”

In response to these external and internal audit findings, a variety of initiatives have been put in place at both the individual modal program level and Safety and Security group level, including, in April 2014, the issuing of the Transport Canada’s Directive on Safety and Security Oversight (DOSSO). The Directive sets out the minimum standards and expectations for the design, development and delivery of oversight programs across all the modal programs and regions required to help ensure consistency, coherence and accountability.

In addition, in 2016-17 National Inspection Plans were replaced by National Oversight Plans (NOPs). While the NOPs do not yet include all oversight activities, they do include oversight activities other than inspections and work is underway to include all activities in future NOPs.

1.3. AUDIT OBJECTIVE, SCOPE AND APPROACH

1.3.1. Audit Objective

This audit assessed the effectiveness of risk-based planning to develop comprehensive plans that clearly define the activities and resources needed to deliver Safety and Security’s modal programs. Specifically, the audit assessed:

• the governance of the program to determine how decisions are made;
• the risk-based planning process to determine what activities are required; and
• the process used to cost activities, including consideration of efficiencies to complete the planned activities.

 

Figure 1: Risk-based Planning and Reporting Process

In assessing the overall effectiveness of the risk-based planning process, the audit examined the extent to which risk assessment directly supports the selection of appropriate interventions and the associated oversight activities, as well as the capability to cost these activities and report results.

1.3.2. Audit Scope

The audit examined the risk-based business planning process used for 2015-16 and 2016-17 while focusing primarily on the two largest programs, Civil Aviation and Marine Safety and Security and two programs that have evolved rapidly in the past few years, Rail Safety and Transportation of Dangerous Goods.

1.3.3. Audit Approach

The audit approach included:

• reviewing program governance;
• reviewing relevant functional direction documents;
• conducting interviews with Program Accountable Executives, Regional Directors and other staff involved in risk-based business planning and costing;
• interviewing staff in Pacific, Prairies and Northern, Ontario and Atlantic regions;
• analyzing the National Inspection/Oversight Plans to identify costs and activities and various risk-based planning activities, methodologies and processes used; and
• comparing documents within and between SO3 programs for consistency and comparability.

In addition, the audit approach included reviewing relevant Government of Canada guidance such as the Cabinet Directive on Regulatory Management, the Treasury Board Framework for the Management of Risk as well as relevant academic research in the areas of risk-based planning for oversight programs such as the Organization for Economic Cooperation and Development work on best practice principles for regulatory enforcement and inspections, the Institute on Governance’s review of safety and security risk governance at Transport Canada, and, finally, research from the Penn Law Best in Class Regulator Initiative which focused on identifying core attributes of regulatory excellence.

A senior applied-statistics consultant provided guidance on sampling strategies and sampling evaluation criteria.

1.4. REPORT STRUCTURE

The report findings are described in four parts:

• Risk Assessment, Cost-Effective Intervention and Oversight Strategies
  • What We Expected
  • Conclusions
  • Recommendations
• Activity Costing and Resource Allocation
  • What We Expected
  • Conclusions
  • Recommendations
• Governance
  • What We Expected
  • Conclusions
  • Recommendations
• Individual Modal Observations
  • Transportation of Dangerous Goods
  • Rail Safety
  • Civil Aviation
  • Marine Safety and Security

The report’s conclusions are based on an overall assessment of Safety and Security and the recommendations are addressed to the Safety and Security group. For ease of reference, management’s response and action plan are included in a separate section.

2. FINDINGS AND RECOMMENDATIONS

2.1. RISK ASSESSMENT, COST-EFFFECTIVE INTERVENTION AND OVERSIGHT STRATEGIES

Context:

In its capacity as a regulator, Transport Canada develops strategies including developing regulations, enforcing regulations and education and awareness outreach to help ensure a safe and secure transportation system. Each of the modes, air, marine, rail, and surface, develop separate safety standards for the transportation of dangerous goods.

Safety & Security’s “Directive on Safety and Security Oversight (DOSSO)” includes a requirement that each modal program develop and implement an integrated risk management framework, including practices and procedures that must be followed to ensure transportation risks are identified, monitored and mitigated. Effective risk management is a pre-requisite for ensuring resources are available and directed to the areas of highest priority and risk.

What We Expected:

Internal Audit expected that each modal program would have the following components as part of their integrated risk management framework:

• a clear, consistent and reliable risk identification and assessment methodology and process to identify the key risks (internal and external) a mode needs to mitigate to help ensure a safe and secure transportation system (the universe);
• a strategy for determining cost effective interventions for mitigating identified risks (e.g., legislative and regulatory instruments, education-outreach program, design adjustments, etc.); and
• an oversight strategy (i.e. choice and combination of oversight tools such as regulatory authorizations, inspections, audits, enforcement) to seek compliance based on the management of risks to an acceptable tolerance level.

Conclusions:

Although each of the modes has been improving their individual risk assessment methodologies and processes, they all fundamentally differ in their basic approaches. The value placed on certain risk factors, the extent to which quantitative and qualitative data is used, including risk ranking scales, differ between the modes and, in some cases, within the modes. As well, each of the modal programs face different challenges in collecting and analyzing basic data such as the number of operators/entities subject to regulation and the lack of consolidated non-compliance data to identify systemic system-wide issues.

Safety and Security has developed the Planning Environment Document (PED) that captures an environmental scan, a Strength/Weaknesses/Opportunities/Threats analysis, and a risk analysis of internal risks to the Programs and external safety/security risks to the transportation system. This document aims at providing multi-modal guidance to the modes regarding risk identification and assessment for the annual planning exercise, albeit the PED has been designed and used primarily for the development of the Corporate Risk Profile. The modes have not used it to focus on mitigating factors resulting from external safety/security risks (e.g. causes of transportation related deaths and accidents) to plan their intervention strategies, nor was the PED linked to the National Oversight Plans.

In addition, the current maturity of the risk assessment processes in Safety and Security does not generate information that supports national, within modes or even cross-modal comparisons. If cross-modal comparisons are required the Department could consider the following:

• defining the technical knowledge and skills needed to design and employ a risk assessment methodology and process that facilitates cross-modal comparisons; and
• identifying and employing cost-effective data collection and analysis methods required to further strengthen its risk assessment process.

As well, a decision would be required about where to build the capacity, i.e., at the modal or Safety and Security group level, or an appropriate combination.

Even if cross-modal comparisons are not required, intervention options need to be identified and assessed so that the most cost-effective risk intervention strategy is selected; this includes identifying benefits, anticipated impacts, and their respective costs. Currently, the practices and the robustness of these assessments varies between the modes.

Finally, while risk tolerances and the corresponding planned level of oversight are not explicitly defined by the modes, they are implicit in each modes’ resource allocations. Specifically, some modes invest major resources in random sampling to measure general levels of compliance, while other modes invest no resources in measuring general levels of compliance. While all modes plan to carry-out some inspections of high, medium and low ranked areas of risk, in any given year, the percentage of each varies between modes and in some cases within a mode. The choice is also based mostly on available resources rather than on what interventions should be used to reduce risks to an acceptable tolerance level. As a result, the modes have not clearly defined the risk tolerance levels required to identify, achieve, monitor and report on its oversight strategies.

Recommendations:

Transport Canada should:

1. Develop a consistent approach to modal program level risk assessments, i.e., a common modal risk assessment methodology and process to identify, assess and rank transportation safety and security risks.
2. Ensure assessments of alternative risk intervention/mitigation strategies (e.g., education versus regulation) consider costs and benefits before selecting a preferred strategy.
3. Define risk tolerance levels to provide the basis for the choice of oversight interventions that will reduce risks to an acceptable level for each mode and provide the rationale for differences between the modes.

2.2. ACTIVITY COSTING AND RESOURCE ALLOCATION

Context:

In 2016-2017, Safety and Security introduced National Oversight Plans (NOPs) to plan and report on a range of oversight activities. While the NOPs describe the overall operating context and process followed to identify oversight priorities, they do not include all oversight activities, namely, enforcement and investigations or education, outreach and awareness. The modes have stated that they plan to expand their NOPs in the future to include all oversight activities.

The approach to planning the 2016-2017 NOPs categorized activities as follows:

• Mandatory/Legislated Services: includes statutory requirements to conduct an activity (certification, litigation, treaty obligations, etc.) and the scope of an activity is scaled based on the statutory requirement.
• Risk-Based Oversight Activities: includes activities in which Transport Canada decides to conduct oversight activity based on the NOP where risks are mapped and activities scaled to level of resources available. It also includes follow-up inspections and responsive oversight activities to address risks emerging during the year.
• Other Program Activities: includes activities that do not fall into the two above categories.

What We Expected:

We expected that each mode would have a process in place to track the cost of their oversight activities, to help identify efficiency opportunities and support resource allocation decisions including:

• a) clearly defined activities;
• b) time and activity data;
• c) controls to ensure data integrity;
• d) analysis of data to identify opportunities for efficiencies and improved risk mitigation; and
• e) use the information to determine resource requirements and help allocate resources.

Conclusions:

All the modal programs have defined activities and collect time and activity data to varying degrees. While most of the modal programs have, at minimum, limited controls to ensure some level of data integrity, all of the modal programs have identified the need to improve data quality and analytics. Despite the availability of time and activity data, within each mode, there has been little analysis to help determine whether there are significant variances in the cost of doing an inspection between regions or within a region. This type of analysis could help indicate either greater standardization is needed or provide the rational for reallocating resources.

Although time and activity processes and data exist for the various modal programs, there are fundamental differences that make intra-modal and cross-modal comparisons difficult. Each mode defines their inspection process differently, including pre-site, on-site and post-site inspection activities. Each modal program uses a different time and activity system to capture activities. There is no standard approach between the modal programs and, in some cases, within a modal program to account for non-inspector staff time expended on supporting “oversight activities”. Without activity costing data, it is not possible to properly define resource requirements or make informed decisions about how to allocate resources.

Recommendation:

• 4. In the shorter term, modal programs should use existing data to analyze their current activities in order to identify areas of inconsistency and opportunities for improvement.

In the longer term, Safety and Security should design and implement a consistent standard activity costing methodology for oversight activities that supports risk-based business planning.

2.3. GOVERNANCE

Context:

At Transport Canada, nine programs fall within Strategic Outcome 3 (SO3) A Safe and Secure Transportation System. Each program has a Program Business Committee (PBC) that is comprised of functional leaders and regional directors and is chaired by a Program Accountable Executive (PAE).

Collectively, the PBC members are accountable to the SO3 Management Board (MB) chaired by the ADM, Safety and Security. The two largest programs, Civil Aviation and Marine Safety and Security and two programs that have evolved rapidly in the past few years, Rail Safety and Transportation of Dangerous Goods, were examined.

What We Expected:

PAEs supported by PBCs and corporate advisors conduct planning and reporting that supports timely and effective decision making by the SO3 MB and TMX.

Conclusions:

The Program Business Committees meet regularly and follow established protocols; however, as reported in Transport Canada’s recent Internal Audit of Recruitment Strategies and the Staffing Process, PAEs have had difficulty receiving the financial data and analysis needed to effectively manage their national programs. Because Financial Management Advisors (FMAs) in the regions support the different programs within their region and FMAs in headquarters support program activities within headquarters, separate discussions on the same issues for PAEs take place in both headquarters and the regions. This can lead to contradictory direction and actions or duplication of efforts and is exacerbated when the PAE is responsible for two or more programs. In addition, even though the Department’s Integrated Decision Making Framework clearly defines the roles and responsibilities of PAEs, there continues to be examples of decisions made at a regional level that impact the national program’s budget.

Recommendation:

• 5. Corporate Services Financial Management should be made responsible to 1) gather Program financial information (HQ and Regional inputs) and 2) for providing PAEs with a complete financial picture that can be used for decision making purposes.

3. MANAGEMENT RESPONSE AND ACTION PLAN

OVERVIEW

• Transport Canada’s Safety and Security Group (TC) agrees with the recommendations of the Internal Audit on Risk-based Business Planning. Our proposed response includes taking short- and longer-term measures to achieve a stronger risk-based approach across all modes.
• In particular, TC will increase its ability to:
  • Identify and evaluate risks to the safety and security of the Canadian transportation system;
  • Act in a way best suited to address these risks; and
  • Evaluate whether its actions have had their desired impact on the safety and security of the transportation system.

BACKGROUND: HOW SAFETY AND SECURITY GROUP MANAGES RISKS

• As stated by the Transportation Safety Board in its 2015-16 annual report to Parliament, Canada is “generally considered to have a safe transportation system, though accidents continue to occur and unmitigated risks in the transportation system persist.”
• Statistics ^{Footnote 1} on transportation accidents and fatalities in Canada show a decline across the modes (i.e. aviation, marine, rail, road, transportation of dangerous goods) despite increasing vehicle traffic and freight volumes. For instance, there were 28 per cent fewer deaths in 2015, all modes combined, compared to the 2010-2014 average.
• TC’s main means of addressing risks to the transportation system has historically been through its regulatory regime: developing regulations, conducting inspections to ensure industry meets regulatory requirements, and taking enforcement actions (e.g. penalties) when they do not meet these requirements. TC continues to work on ensuring it carries out these actions fairly, predictably and effectively.
• Several years ago, TC began using risk factors as a basis for setting its inspection priorities. Today, we have made progress and each mode develops annual inspection plans, which:
  • cover all of its monitoring, inspection and enforcement activities; and
  • explain the way risk is measured.

This approach facilitates each mode identifying the most at-risk operators, operations and/or segments of the transportation system and taking steps to reduce these risks. Periodic reports throughout the year help each mode monitor the implementation of these actions and allow them to make adjustments as necessary, based on emerging risks. At the end of the year, each mode considers any risks that remain, during the next year’s inspection planning. There have been continuing efforts at TC to improve the effectiveness of this approach and supporting processes, as well as strive to achieve greater consistency across the modes, to help senior managers oversee their particular programs and make risk-informed decisions.

• However, we recognize that the maturity of our risk-based approach varies across modes and further improvements are required to fully address the audit findings.

TRANSPORT CANADA’S SAFETY AND SECURITY GROUP RESPONSE TO THE AUDIT

• Applying a strong, risk-based approach consistently across all modes involves strengthening and reforming a range of elements in how Safety and Security Group at Transport Canada conducts its business: from its inspection and enforcement activities to how it makes fundamental decisions about when and what to regulate. Regulation will certainly remain an important way for TC to intervene in the Canadian transportation system. However, other powerful tools exist that may achieve the desired results more effectively or efficiently in certain situations.
• For instance, an educational campaign aiming to change a behavior that will lead to a potential risk may achieve the same outcome in the overall safety of the system without the administrative and financial burden of creating a regulation and conducting inspections.
• A stronger risk-based approach will allow TC to pursue greater safety performance across the Canadian transportation system. Our accident and incident rates are already relatively low, which means that lowering them even further requires a more sophisticated and targeted approach with robust risk analysis. A more risk-based approach will also help us to allocate resources where they can make the biggest difference in safety and security results.
• TC’s response to the audit includes both short-term and longer-term measures:

Short-term: Incremental Improvements through Existing Tools

TC’s Safety and Security Programs will strengthen their efforts to:

• Identify, assess, evaluate, and monitor risks in the transportation system throughout each fiscal year;
• Link its actions to the identified transportation risks
• Adopt a common costing methodology to better measure cost efficiency; and
• Monitor and report on progress made over time in reducing risks to and in the transportation system through its actions.

Longer-term: Transformational Initiatives

TC’s Safety and Security will launch modernization initiatives designed to:

• Increase the use of evidence and intelligence to better identify issues, trends and emerging risks to the transportation system;
• Compare risks and their relative potential impacts;
• Ensure the actions TC takes to control, reduce or manage these risks are fair, predictable and appropriate; and
• Better evaluate the effectiveness and efficiency of TC actions to inform future decision making.

Recommendation	Management Response	Timeframe (for each action)	OPI direct report for each specific action
1. Develop a consistent approach to modal program level risk assessments, i.e., a common modal risk assessment methodology and process to identify, assess and rank transportation safety and security risks.	Incremental improvements through existing processes and policies
	TC’s Safety & Security (S&S) will revise existing tools to increase its focus on: systematically identifying risks to the transportation system; and explaining how we choose actions to manage, reduce, control or monitor risks.	September 2017	Director General (DG), Multimodal Strategies and Programs Integration (MSPI)
	Transformational Initiatives
	Modernize the S&S intelligence and risk analysis functions to be more proactive at identifying risks and adopting a common risk assessment model that supports the ranking of risks.	Starting in 2017-2018 until 2021-2022	Oversight transformation team
2. Ensure assessments of alternative risk intervention/mitigation strategies (e.g., education versus regulation) consider costs and benefits before selecting a preferred strategy.	Incremental improvements through existing processes and policies
	Enhance existing reporting tools to provide more information on: ongoing changes to the risks to the transportation system; and whether the chosen actions (i.e. intervention/mitigation strategies) are successfully addressing these risks.	March 2018	DG, MSPI
	Transformational Initiatives
	Define expected impact or change in use of a range of actions Measure effectiveness of these actions against defined performance indicators and objectives. Modernize S&S internal policies to increase predictability and provide guidelines for selecting appropriate actions based on new trends or emerging risks. Introduce greater flexibility into S&S internal directions and operations to provide more diversified responses to the level and nature of transportation risks.	Starting in 2017-2018 until 2021-2022	Oversight transformation team
3. Define risk tolerance levels to provide the basis for choosing oversight interventions that will reduce risks to an acceptable level for each mode; and provide the rationale for differences between the modes.	Incremental improvements through existing processes and policies
	Amend existing processes and policies to clearly define risk tolerance levels that will ensure what actions should be taken to manage, reduce, control or monitor risks.	April 2018	DG, MSPI
	Transformational Initiatives
	Develop common criteria to define risk tolerance levels based on several factors (e.g. fatalities, serious injuries, property/environmental damage, legal liability, economic impact) Develop processes and policies that support decisions to address risks that are above the set risk tolerance levels.	Starting in 2017-2018 until 2021-2022	Oversight transformation team
4.A. In the shorter term, modal programs should use existing data to analyze their current activities to identify areas of inconsistency and opportunities for improvement B. In the longer term, Safety and Security should design and adopt a consistent standard activity costing methodology for oversight activities that supports risk-based business planning.	Incremental improvements through existing processes and policies
	A. S&S Programs will continue to improve their tracking systems for activity costing.     B. Since completion of the audit, a common inspection and enforcement costing methodology was developed to support 2017-18 business planning. S&S will apply this approach for the upcoming year and will refine it in future years.	Ongoing       April 2017	S&S Programs       S&S MSPI & Programs
5. Corporate Services – Corporate Financial Management should be made responsible for: 1. Gathering Program financial information (HQ and Regional inputs); and 2. Providing PAEs with a complete financial picture they can use for decision making purposes.	Incremental improvements through existing processes and policies
	Corporate Services agrees with this recommendation and will address it in the following manner: Within three working days after month end, Corporate Financial Management (CFM) will provide Programs Accountable Executives (PAEs) with a report that details each Program’s month end operating financial picture.  The report will include a breakdown of the financials by Group Head (HQ and Regions).  Modal Programs will have a week after receiving the report, to meet and report back to CFM, any decisions/adjustments of a financial nature CFM should report against their Program’s financial position for the period in question.  CFM will then reflect these off-book modifications in the Program financial position reported to Transport Canada’s Executive Management Committee for the month.	Complete as part of the P4 review (July 2016)	Corporate Services, Resource Management     S&S Programs     Corporate Services, Resource Management

Modal Profiles

As previously indicated, the objective of this audit was to assess the effectiveness of Safety and Security’s risk-based planning processes. To do this, we assessed Safety and Security’s overall process as well as specific modes’ processes. As illustrated in figure 1 below, we expected to see the following:

 

Image Description

Risk Identification

• identify external issues & hazards
• precisely define problem
• risk assess
• identify key indicators to measure & monitor performance

Intervention Strategy

• identify tools to manage risk (legislation, regulation, incentives, education, etc.)
• implement strategy (certification, oversight activities, enforcement, etc.)
• monitor, research and analysis

Oversight Strategy

• develop oversight tools (inspections, audits, compliance estimation programs)
• calibrate scope and depth of oversight activities using appropriate risk factors
• carry-out oversight to manage risks and collect data to allow performance to be evaluated

Figure 2: Regulatory Process

Risk Identification and Intervention Strategies

From a risk identification perspective we expected to see a clear, consistent and reliable risk identification and assessment methodology and processes to identify the key risks a mode needs to mitigate to ensure a safe and secure transportation system. As part of this, we expected modes to clearly define what “risk-based” means in their modal context; including what role (if any) non-risk-based approaches play and why they are appropriate.

We expected to see an intervention strategy designed to mitigate the identified risks to the system and produce the desired results - being a safe and secure transportation system. As part of this strategy, we expected to see an assessment of which intervention tools would be most cost-effective in producing the desired and lasting results. In addition, we expected to see a plan to implement the strategy.

Finally, we expected that each mode monitors how effective their interventions are in ensuring a safe and secure transportation system. Included in this is the expectation that interventions would be continuously improved.

Oversight Strategy

We expected that each mode showed evidence of a well-organized strategy to carry out selected oversight activities driven by clear objectives and clear accountability for results. In doing this, we expected to see an oversight strategy that supports the day-to-day analysis and decision-making process. Within this strategy we expected to see: clear objectives which delineate the scope and depth of oversight required to monitor intervention strategies and carry out oversight activities (such as inspections and audits) to gather the appropriate level of information to ensure risks are being managed (including those previously identified and those that may be emerging). Finally we expected to see the data gathered at this level be fed back into the system-wide risk identification assessment.

Activity Costing

We expected that each mode would have a process in place to track the cost of their regulatory process to assist in selecting the most cost effective intervention tool, to identify efficiency opportunities, and to support resource allocation decisions.

Modal Profiles – Limitation

We attempted to portray the current progress of each mode toward building an effective risk-based planning process. Within each area we examined (i.e., Risk Identification, Intervention Strategy, Oversight Strategy and Activity Costing), each mode is at a different stage of development making it a challenge to render individual portraits that allow for a simple comparison between the modes. Overall, the profiles reinforce our summary findings and the need for the Department to address the recommendations in a comprehensive, strategic and sustainable manner.

Transportation of Dangerous Goods (TDG)

In Canada, there are over 25,000 known commercial sites that handle, offer for transport, transport, or import dangerous goods (HOTI sites) and over 3,000 registered Means of Containment (MOC) facilities.

Based on the authority provided by the Transportation of Dangerous Goods Act, 1992, (TDG Act) the Transportation of Dangerous Goods (TDG) Program develops safety standards and regulations, conducts oversight activities and provides expert advice (e.g. Canadian Transport Emergency Centre (CANUTEC)) on dangerous goods incidents.

To achieve its mandate, TDG identifies threats to the public, promotes public safety, enforces the TDG Act and its regulations, guides emergency response, intervenes to limit the impact of incidents involving the transportation of dangerous goods, and develops policy based on research.

Risk Identification & Intervention Strategies:

To ensure it focuses on areas of highest risk, namely, harmful consequences to the public, infrastructure and/or the environment, TDG prioritizes, identifies, assesses and monitors risks and emerging issues at three fundamental levels:

• strategic level (system-wide risks);
• operational level (national and regional level priorities); and
• operator level (issues specific to individual HOTI sites or MOC facilities).

Strategic Risk Management

The operational and operator level risk assessments are discussed in the Oversight Strategy section below.

To perform the annual strategic level risk assessment, TDG follows a six step process:

• establish the context;
• identify and assess the risks;
• identify and assess possible methods to treat the risk;
• assess costs and benefits of risk treatment options;
• take action/develop action plan; and
• monitor impact/conduct follow-up.

Once risks are assessed, TDG identifies the most cost-effective intervention or combination of interventions to mitigate the identified risks. Intervention tools may include amended/new regulations or education/awareness campaigns. For example, during the 2015-16 risk assessment process, TDG identified the volatility of lithium-ion batteries as an emerging issue that could have a significant impact on the safety of the transportation system. As a result, effective April 1, 2016, TC issued a new protective direction banning the shipment of lithium-ion batteries on passenger aircraft in Canada. This direction will remain in effect until new permanent rules are added to the TDG regulations.

Although not identified as a specific step in the process, TDG consults with external stakeholders in deciding the intervention strategy.

At the time of our audit, TDG was developing a TDG Integrated Risk Management Framework to formalize the strategic level risk assessment process and integrate all aspects including its risk registry, risk corporate profile and risk ranking capabilities. The framework is expected to be completed during fiscal year 2016-17 and fully implemented for 2017-18.

Operational and Operator Level Risk Management

To assess operational and operator level risks, TDG separates its population of operators into the following categories:

• sites that handle, offer for transport, or import dangerous goods (HOTI);
• Means of Containment (MOC) facilities; and
• Emergency Response Assistance Plan (ERAP) holders.

As noted above, there are over 25,000 known HOTI sites that are listed in the TDG Inspection Information System (IIS) database which is used by inspectors to log inspection results. However, TDG estimates approximately 15,000 additional HOTI sites are not in the IIS and, as a result, are not subject to its oversight regime. ^{Footnote 2} The Federal Budget 2016-17 approved funding for the implementation of a registry system to identify the total population of HOTI sites. Until the registration system is implemented, TDG has implemented a New Sites Strategy for the identification, validation, input and maintenance of potential new HOTI sites.

Oversight Strategy:

To oversee its program, TDG has developed the following oversight activities:

• compliance inspections (risk-based inspections, randomly selected inspections and reactive inspections);
• regulatory authorizations (certification);
• enforcement (tickets and investigations); and
• outreach and awareness.

Internal Audit Observation

TDG does not have a clear vision statement with respect to expected outcomes of its oversight strategy. In addition, it does not appear that TDG measures the performance of its oversight strategy and, as a result, it is not clear to the audit team whether the strategy is achieving its objective(s).

National Oversight Plan

At the start of each planning cycle TDG creates a “National Oversight Plan” identifying all compliance inspection activities and regulatory authorizations for the coming year. It does not, however, plan outreach and awareness activities. TDG plans to develop an approach to account for these activities in future years’ plans.

Compliance Inspection Activities

Compliance inspection activities include risk-based inspections, randomly selected inspections and reactive inspections.

Risk-Based Inspections

Risk-based inspections assess a HOTI site’s or MOC facility’s compliance with the required regulations. They include the following:

• risk-ranked general compliance inspections (of HOTI sites);
• risk-ranked MOC facility inspections; and
• inspections of potential new HOTI sites.

Risk-ranked inspections account for 51% of all compliance inspection activities for 2016-17. Risk ranking is based on pre-established factors that are reviewed on a regular basis (see section below).

Randomly selected Inspections

TDG has created a program called the Compliance Estimation Program (CEP) to determine a statistically reliable and unbiased estimate of the industry’s general compliance rate. Operators are selected randomly and are subjected to a general compliance inspection. In the selection process, the random sample includes low-, medium- and high-risk operators. These inspections are mandatory (per TDG’s oversight strategy) and account for 34% of all compliance inspection activities for 2016-17. Another benefit of the CEP is that it updates the risk rankings of medium- and low-risk sites, which otherwise would not be inspected, and provides information on the TDG industry which is useful for risk-analysis within the TDG oversight program.

Reactive Inspections

Reactive inspections enable TDG to target HOTI sites and MOC facilities to address issues (strategic, operational and/or operator) that occur during any given fiscal year. They include the following:

• follow-up inspections;
• follow-up confirmations of compliance;
• unplanned MOC facility inspections;
• triggered inspections;
• opportunity inspections; and
• consignment inspections.

Reactive inspections account for 15% of all inspection activities in 2016-17.

Risk Ranking of HOTI Sites and MOC Facilities

In order to plan the risk-ranked general compliance inspections for the HOTI sites, TDG carries out an annual risk assessment process using its Inspection Prioritization Model. First, TDG considers operator level risk through the assignment of a risk score based on the assessment of each HOTI site using seven pre-established risk factors including history of site inspections, compliance and accidents, and population density. Second, TDG considers operational level risk through the assignment of risk scores based on the prioritization of national and regional priorities identified from the annual strategic risk assessment. The combination of these scores equals the overall rating for each TDG site. Emerging issues that are identified as high risk between planning cycles will be inspected and resources may be reallocated.

TDG also uses the Inspection Prioritization Model to plan the risk-ranked inspections for the MOC facilities. Each facility is risk ranked and assigned a risk score based on 15 pre-established risk factors that assess the compliance to standards including manufacture, repair, building, periodic re-testing or inspection of containers.

Compliance Estimation Program (CEP)

Through the CEP, random sites are selected and regions are provided a list of the HOTI sites and MOC facilities that must be inspected. It should be noted that the risk scores for many medium- and low-risk HOTI sites are calculated based on historical information recorded in IIS given that there may not have been any recent interaction with the HOTI site.

Regulatory Authorizations (Certifications)

Regulatory authorizations are regulatory certifications and form part of the oversight strategy. Regulated parties submit an application to TC seeking permission either to conduct a regulated activity or be exempt from it. The party may be granted a permit, a license or a certification. TDG does not control the number of requests that may be received in any given year.

Risk Prioritization & Risk Thresholds

In 2016-17, TDG developed the following thresholds for prioritizing inspection activities:

1. Mandatory Inspections (100% completion):

• Compliance Estimation Program (CEP);
• High-risk HOTI sites with a risk score of 30 or greater; and
• MOC facilities with a risk score of 23 or greater.

2. Prioritization of remaining HOTI sites:

• High-risk HOTI sites with a risk score of less than 30 and greater than or equal to 20.1; and
• Medium- and low-risk HOTI sites.

The scheduling of inspection activities is based on the prioritization thresholds. TDG provides the risk ranking of operators to each regional director, who in turn with the inspectorate, is required to select certain operators for inspection. For 2016-17, regions were only guided to concentrate on high risk sites and facilities. Such guidance did not exist for fiscal year 2015-16.

Activity Costing:

To cost its oversight activities, TDG has established activity time standards for each type of oversight activity. With respect to inspections (i.e. risk-based and randomly selected) activity standards include time for the preparation, on-site fieldwork, post-field work reporting, and travel times to sites. Time associated with inspection training, inspector administrative work, management review/oversight and administrative support is not included in this standard. The activity time standard is used to cost its NOP and to request additional resources from Treasury Board.

On a quarterly basis, TDG compares this standard against the actual time recorded by inspectors to complete planned inspections. The objective of this review is to confirm the reasonableness and accuracy of their time standards. They also follow up on any significant variations amongst the inspectorate. Through this process, TDG has identified a few issues with the accuracy and comparability of their time standards: inspectors are inconsistently tracking their time amongst the various activities and there are discrepancies between the time it takes for inspectors to execute an inspection (most likely due to the varying experience levels of inspectors).

For 2015-16 planning purposes, TDG estimated that any given inspector would be able to perform 85 inspections each year. Based on this standard, the following table illustrates each region’s actual performance against the established standard.

2015-16 Year-to-date Cost ($)

	Planned for the Year (only includes sites that were identified for inspection in the work plans)			Actual YTD (includes planned and unplanned inspections)
	FTE Salary	Travel Cost	Total	FTE Salary	Travel Cost	Total
Atlantic	$224,100	$45,448	$269,548	$234,600	$47,995	$282,595
Quebec	$183,600	$42,456	$226,056	$324,900	$50,995	$375,895
Ontario	$237,300	$107,171	$344,471	$302,400	$99,478	$401,878
PNR	$260,640	$110,480	$371,120	$324,660	$157,068	$481,728
Pacific	$262,440	$69,688	$332,128	$392,880	$75,523	$468,403
National	$100,962	$25,200	$126,162	$65,692	$42,251	$107,943
Total	$1,269,042	$400,443	$1,669,485	$1,645,132	$473,310	$2,118,442

Figure 3: Comparison of Annual Planned and Actual Regional Inspection Costs (2015-16)

TDG did not report actual time spent on planned and unplanned inspections separately. As a result, it is not possible to identify whether their actual performance is consistent with their performance standards nor is it possible to accurately cost their overall oversight strategy.

Internal Audit Observations:

Ideally, a formal strategy should guide the selection process and be based on a risk tolerance that appropriately “accepts” a level of risk commensurate with achieving an overall objective of ensuring compliance with TDG regulations.

As illustrated in the table below, for fiscal year 2016-17, all regions planned to inspect 100% of their high-risk HOTI sites with risk scores greater than or equal to 30, while only 64% of the other high-risk sites (with a score between 20.1 and 30) were planned to be inspected. In addition, 569 medium- to low-risk HOTI sites were planned to be inspected. It was explained to the audit team that in certain cases medium- to low-risk sites are planned for inspection due to their proximity to high-risk sites that are scheduled for inspection. Notwithstanding this, it is not clear why medium- to low-risk sites would take precedence over high-risk sites. As can be inferred by the data, and all things considered equal (i.e. the cost of performing a high-, medium- and low-risk inspections is the same), if the Atlantic, Quebec and Ontario regions dedicated their inspections of medium- to low-risk to those of high-risk (per the established threshold strategy), each of these regions would be able to complete all of their high- risk inspections. On the other hand, the Prairie and Northern, and Pacific regions would not be able to complete inspections for their high-risk operators based on the resources allocated; that is, 575 and 10 high-risk inspections would still not be conducted, respectively. Furthermore, based on the data presented, if all resources were dedicated to high-risk inspections (i.e. based on a national perspective versus a regional one), approximately 5% of the high-risk operators would not be inspected.

HOTI Sites to be inspected for 2016-17 (not including CEP activities)
Region	High-Risk Sites with score >= 30	Planned Inspections of High-Risk Sites with score >= 30	High-Risk Sites with score < 30 and >= 20.1	Planned Inspections of High-Risk Sites with score < 30 and >= 20.1	% of High-Risk Sites with score < 30 and >= 20.1 included in inspection plans	Medium- and Low-Risk Inspections Planned
Atlantic	1	1	93	80	86%	271
Quebec	44	44	249	288	113%Footnote 3	4
Ontario	22	22	394	369	94%	178
PNR	52	52	905	322	39%	8
Pacific	28	28	323	205	66%	108
Total	147	147	1,964	1,264	64%	569

Figure 4: Number of Regional HOTI Sites and Planned Inspections Based on Risk (2016-17)

It is also not clear to the audit team why the CEP takes precedence over the risk-based inspection activities. The Organization for Economic Co-operation and Development (OECD) provides the following guidance with respect to regulatory enforcement and inspections: “Governments… should consider the use of random statistically representative surveys every few years so as to get a “reality check” of the situation with business operators’ compliance in critical areas.” ^{Footnote 4} Not only is the CEP program for TDG completed on an annual basis but it is also taking precedence (and resources) over high-risk operator inspectors. For fiscal year 2016-17, 1,510 CEP inspections are planned; which includes 130 high-, 565 medium- and 951 low- risk operators. If the CEP were conducted after all high -risk-ranked inspections were conducted, the overall program for risk-ranked inspections would be achieved.

Rail Safety

TC’s Rail Safety Program is responsible for safety oversight of railways that fall under the Railway Safety Act. This includes railway companies who hold a Certificate of Fitness from the Canadian Transportation Agency, as well as local railway companies, which include provincially-regulated shortlines, light rail transit, and tourist trains and a variety of industrial operations that operate on federally-regulated tracks. The Railway Safety Act also applies to road authorities.

Rail companies are expected to make safety a top priority and commit to maintaining an effective safety culture throughout their organization. Under the authority of the Railway Safety Act, companies operating on federally-regulated railway lines are directly responsible for complying with the federal regulations. The Railway Safety Act provides a specific set of authorities and corresponding tools which allow TC to address both compliance (with legal requirements) and safety (of rail operations). This includes developing regulations and approving or establishing rules on a variety of matters related to safe operation of railways.

In managing its safety oversight of railways, Rail Safety approaches its responsibility by:

• promoting compliance and safety;
• monitoring compliance and safety; and,
• enforcing compliance / mitigating threats to safety.

Risk Identification and Intervention Strategies:

Disciplines

Unlike other modes, Rail Safety segments railway operations into functional areas known as disciplines, as follows:

• Equipment (including Rail Freight and Passenger Cars and Locomotives);
• Engineering- Grade Crossings (includes 33,150 crossings);
• Engineering- Signals;
• Engineering- Track (includes 45,700 miles of track);
• Engineering- Bridges (includes 6,800 bridges);
• Engineering- Natural Hazards;
• Operations;
• Occupational Health and Safety; and,
• Safety Management Systems.

Operators can have functions that cross some or all disciplines.

Risk identification & Assessment process

Rail Safety’s integrated risk management framework, Rail Safety’s Risk-Based Business Planning Process, provides the process to identify, assess, monitor and mitigate safety issues (e.g. vulnerability of track in winter) and program management issues (e.g. road authorities insufficient knowledge of regulatory requirements). This process is designed to identify issues for which a possible intervention may be required as well as to aid in the prioritization of Rail Safety’s inspection regime (as discussed in the next section). It also allows Rail Safety to gather information on certain issues (e.g., TSB occurrences data) prior to selecting an appropriate intervention.

The annual process begins with the collection of data from external and internal sources. External sources include information on commodities, TSB incidents and accident reports, operational information from the Railway Association of Canada and leading indicator data provide under the Transportation Information Regulation, such as track geometry data, trouble ticket information, etc. Internal data sources include past inspections results, enforcement actions, inspector observations, and complaints and inquiries. All issues are assessed and risk-ranked based on the criteria of probability and severity of adverse consequences using either the safety risk matrix or the program management risk matrix.

Once safety and program management issues are identified and assessed, Rail Safety develops a plan which monitors or mitigates the identified issues. In those cases where Rail Safety requires further information on the nature and extent of the issue, research and analysis may be conducted or additional lines of inquiry or focus will be integrated into the inspection regime (as discussed in the next section). Throughout the year, Rail Safety reviews the results of its mitigation and monitoring activities and at year end, determines if additional or continued action is required. In some cases the issue is carried forward to the next fiscal year for monitoring purposes, while others are “closed” as they have either been deemed to be a non-issue or have been dealt with by way of the selected intervention.

Intervention Strategies

Once an issue is identified as a risk to be addressed, an intervention strategy is required to address the risk. Intervention strategies could include legislation/regulations, voluntary agreements, education/outreach or research and analysis. The primary intervention tool for Rail Safety is the creation of a new regulation or new rules, but others tools such as outreach and guidance material may also be used. However, this process of identifying intervention strategies does not consider the costs of the various alternatives. Only after a new regulation is the chosen intervention, does Rail Safety consider the costs of this option. This analysis is required as part of the Cabinet Directive on Regulatory Management and is reported in the Regulatory Impact Analysis Statement when proposed regulations are published in the Canada Gazette.

Oversight Strategy:

To oversee the safety of railways, Rail Safety has developed the following oversight activities:

• inspections (planned randomly selected inspections, planned risk-based inspections and reactive inspections);
• safety management system audits (comprehensive audits, targeted audits); and
• regulatory authorizations.

Inspections

Rail Safety’s National Inspection System includes the following types of inspections:

• Planned randomly selected inspections, referred to as Component “A” inspections, verify the regulatory compliance within the railway industry and may detect emerging safety issues and immediate safety threats. They are determined by using a sampling methodology that considers several factors such as previous year’s defect rates, population size, margin of error, and desired level of confidence. These inspections are broken down by region and functional discipline and HQ provides guidance to the regions on sample size. They account for approximately 60% of all inspections^{Footnote 5}.
• Planned risk-based inspections, referred to as Component “B” inspections, address known safety and program management issues identified from the annual risk assessment (as discussed in the previous section). These inspections can also be a result of required follow up to a Component A inspection. Each discipline determines whether the issues are national in scope or whether they are regionally based, and through discussions with HQ and the regions, decisions are made as to where and what to inspect. This selection is based on HQ and regional knowledge of the sites and operators. They account for approximately 28% of all inspections.
• Reactive inspections, referred to as Component “C” inspections, are used to address emerging issues, (e.g. derailments, complaints) for follow-up to verify appropriate corrective actions have been taken, or as “opportunity inspections”. When a Component A or B inspection is conducted in a remote location, additional “opportunity” inspections, with a narrow scope, may be included in order to maximize efficiency of travel costs. Component C inspections account for approximately 12% of all inspections.

As previously explained, once priority areas are identified (per the integrated risk assessment process), regions are responsible for planning inspections (i.e. random, risk-based and reactive) to ensure delivery of the oversight plan and proper monitoring of issues within their region, as well as national issues for which the region is contributing. Each discipline (within each region) has its own approach to planning and prioritizing the inspections deemed necessary. With the exception of the discipline of Grade Crossings, there is no nationally consistent approach to assess risk. Disciplines within the regions employ their own unique approaches to assess risk and to target their oversight activities to the operators/sites/within their region.

During fiscal year 2015-16, Rail Safety conducted 37,540 inspections, of which 22,195 were Component A inspections, 10,507 were Component B and 4,838 were Component C.

Safety Management System Audits

Safety Management System (SMS) audits assess operators’ compliance with the Rail Safety Management Systems regulation which requires every operator to develop and implement an SMS.

The new SMS Regulations came into effect April 1 2015. Fiscal year 2015-16 was considered a transition year and no SMS audits were completed. Rail Safety focused efforts on education and awareness to make sure that rail companies understood the requirements of the new regulation. Rail safety plans to complete 30 audits in 2016-17.

• Comprehensive Audits will assess whether railways are compliant with the regulatory requirements and if their SMS processes are working as intended. They are determined by the size and complexity of a railway and may be staggered over multiple visits. All railways will be audited over a three to five year cycle depending on the assessed risk level of a railway. Each region independently assesses the level of risk and determines the order of priority for audits of regional railways.
• Targeted Audits focus on safety concerns identified by regions and/or headquarters based on an analysis of compliance rates, TSB occurrence data, TSB recommendations and industry data. These audits consist of verifying the implementation of core components of a railway’s SMS and provide TC with an appraisal of the effectiveness of the railway’s SMS with respect to specific safety issues or program management issues.

Regulatory Authorizations

When the Railway Safety Act was amended on May 1, 2013, the Minister of Transport was granted the authority to develop Railway Operating Certificate (ROC) regulations that pertain to the issuance, variance, suspension, and cancellation of a RROC for all companies subject to the Railway Safety Act. This includes 73 railway companies (federally regulated companies) and local railway companies (companies, other than railway companies, that operate on federal tracks). They do not expire and, providing the rail company remains in compliance with regulations and rules, the ROC does not have to be re-issued.

Activity Costing:

To cost its oversight activities, Rail Safety has developed inspection time standards for each type of inspection within each discipline within each region. It includes time spent on planning, travel time, conduct and reporting. Time associated with inspection training, inspector administrative work, management review/oversight and administrative support is not included in this standard, however this information is nonetheless captured within Rail Safety’s Resource Management System.

Rail Safety has started to compare standards against the actual time recorded by inspectors to complete planned inspections. The objective of this review is to confirm the reasonableness and accuracy of their time standards. Rail Safety used these time standards to develop their NOP for 2016-17 for each individual disciple and region. In the long term they would like to have national time standards.

Internal Audit Observations:

Rail Safety does consider different intervention options but the costs of these options are not compared when deciding the most cost-effective option. While risks are identified at the national level they are managed at the regional level. While oversight activities are defined, there is no clear vision describing the expected outcomes of these activities and, as a result, Rail Safety cannot assess the effectiveness of its strategy.

While Rail Safety’s approach to prioritization of its inspection activity focuses on monitoring identified risks at the “system” level (i.e. horizontal issues that cut across all railways, but may not apply to all railways) there is no national risk assessment at the railway (or operator) or activity level with the exception of Grade Crossings. The delegation of prioritization of inspection activities to the regions does not allow for a national consistent approach to assess risks and target resources to inspect the highest risk areas prior to medium or low risk areas. Each discipline and each region follow different practices in prioritizing their activities. As a result, it is not clear if resources are being effectively deployed to mitigate risks. Furthermore, since Rail Safety uses its inspection regime to monitor its safety and program management issues, it may not be possible to fully ascertain if the level of risk associated with these issues is necessarily mitigated. The Organization for Economic Co-operation and Development (OECD) provides the following guidance with respect to regulatory enforcement and inspections: “Each activity and business should have their level of risk assessed.”^{Footnote 6} This approach would ensure that Rail Safety mitigates the highest risks associated with a safe and secure modal transportation system.

Rail Safety completes its random selection inspection program (i.e., Component A inspections) on an annual basis. As explained above, it is not clear to Internal Audit if this is the most effective use of resources to mitigate the identified risks, since the planning process does not prioritize Component A versus Component B inspections. Rail Safety is also struggling with this question and in the 2016-17 budget exercise looked for direction from senior management with respect to what is the right mix of these two activities. A decision was made to go forward with both activities. As TC looks for consistency between modes, the balance between risk-based inspections and inspections based on a random sample is going to have to be addressed. The OECD has written a paper called Regulatory Enforcement and Inspections and in the paper it is recommended, “Governments… should consider the use of random statistically representative surveys every few years [not annually] so as to get a ‘reality check’ of the situation with business operators’ compliance in critical areas.”^{Footnote 7}

Finally, Rail Safety should continue to improve its costing practices to ensure complete, accurate and transparent cost data is available to support program level resource allocation decisions.

Civil Aviation

TC’s Civil Aviation Program promotes the safety of the national air transportation system through its regulatory framework and aviation industry oversight activities. As part of the regulatory framework, Civil Aviation develops policies, guidelines, regulations, standards, and educational materials to advance civil aviation safety in Canada. As part of the oversight activities, Civil Aviation verifies the aviation industry’s compliance with the regulatory framework through surveillance activities (i.e. assessments, validations, inspections, and enforcement) and service activities to the aviation industry (i.e. licenses, certifications).

The Aeronautics Act and the Canadian Aviation Regulations (CARs) provide a specific set of authorities, compliance provisions and fees for services provided by TC.

Risk Identification and Intervention Strategies:

Civil Aviation’s vision is to promote a “proactive safety culture” within the civil aviation system. Using a “systems approach”^{Footnote 8} to managing risks, Civil Aviation develops and administers policies and regulations to ensure the safest civil aviation system for Canadians. To ascertain whether it is meeting its vision, Civil Aviation has developed performance indicators with targets for a three-year period that are included in the SO3 Performance Measurement Framework. The performance indicators are a work in progress and are reviewed and reported annually in the Departmental Results Report. They are based on data collected from various sources (e.g. accident rate from Transportation Safety Board). As part of the recent “Follow-up Audit of Civil Aviation”, TC’s Evaluation function reviewed the performance indicators and noted that they could be strengthened. Civil Aviation is currently developing an “Aviation Safety Program Management Framework” with improved performance indicators to better measure and report on performance.

To ensure it focuses on areas of highest risk to the system, Civil Aviation prioritizes, identifies, assesses and monitors issues at two fundamental levels:

• strategic level (system-wide risks); and
• enterprise level (individual enterprise risks).

Strategic Level Risks

As part of the Department’s annual integrated planning process, Civil Aviation completes an environmental scan that includes an analysis of the identified internal and external system-wide level risks.

To support this strategic level risk assessment, Civil Aviation collects safety data from both external and internal sources. External sources include Transportation Safety Board information on incidents and accidents, as well as safety information from its international partners (i.e., International Civil Aviation Organization (ICAO), Federal Aviation Administration (FAA) in the U.S., Civil Aviation Safety Authority (CASA) in Australia, Safety Assessment of Foreign Aircraft (SAFA) database), industry, and other federal government departments. Internal sources include various Civil Aviation databases such as the Civil Aviation Daily Occurrence Reporting System (CADORS), Aviation Incident Reporting and Admin (AIRA), and the National Aviation Company Information System (NACIS).

Building on this work, for the first time in 2015-16, Civil Aviation carried out an analysis that identified, assessed and ranked strategic system-wide safety risks. The following four highest risk system-wide safety issues were identified:

• approach and landing phases of flight;
• Unmanned Aerial Vehicles (UAVs);
• loss of control in-flight; and
• human performance factors.

These issues were considered during the 2016-17 Civil Aviation business planning process and included in the branch level work plans.

To integrate this strategic risk assessment into the planning process, Civil Aviation had plans to establish a “strategic safety risk assessment framework” and develop corresponding risk management tools to assist in identifying system-wide safety risks; however, as of the date of this report, the project was on hold given TC’s fiscal environment.

Interventions

Over the years, Civil Aviation has used different approaches to assess alternative interventions to mitigate safety risks including new regulations and education and awareness. For example, in determining the intervention strategy for Unmanned Aerial Vehicles (UAVs), Civil Aviation performed a comparative analysis of three related data sets (Special Flight Operating Certificates, incidents, inquiries). As there was a steady increase in all three categories, Civil Aviation decided that an awareness safety campaign would increase compliance to safety regulations. The campaign included the creation of a website and a partnership with Best Buy to provide information to UAV purchasers. This campaign has positively resulted a greater number of UAV registrations.

In 2016-17, Civil Aviation has begun to implement a formal process to assess alternative interventions to mitigate safety risks and to select the most cost-effective approach or combination of approaches through cost-benefit analysis and consultation with stakeholders. Technical Program Evaluation and Coordination (TPEC) within the Standards Branch recently developed a Safety Initiative Business Model, based on Treasury Board Secretariat’s Cabinet Directive on Regulatory Management. Civil Aviation has not yet taken a safety risk through the entire process.

When the selected intervention involves developing and/or amending regulations, the Canadian Aviation Regulation Advisory Council (CARAC) processes are followed. The issue is assessed against factors including the cost or savings to businesses, Canadians or government and the potential impact on the Canadian economy and its internal and international competitiveness.

Enterprise Level Risks

To assess enterprise level risks, Civil Aviation separates its population of regulated enterprises into the following categories based on certificate type:

• Operating Certificate holder: Flight Training Units, Private Operators, Domestic Air Operators, Aerial Work, Air Taxi Operators, Commuter Operators, Airline Operators;
• Non-Operating Certificate Holder: Manufacture of Aeronautical Products, Distribution of Aeronautical Products, Aircraft Maintenance Engineer (AME) Licensing & Training, Approved Maintenance Organizations, Air Navigation Services;
• Airports; and
• Heliports.

There are approximately 2,253 regulated enterprises recorded in the National Aviation Safety Information Management System (NASIMS). An exercise began in 2015-16 to update the population and remove those enterprises that are no longer operating.

Oversight Strategy:

In October 2015, the National Oversight Office (NOO) was created and given the responsibility for developing and documenting Civil Aviation’s oversight planning and reporting practices, including the development of the National Oversight Plan (NOP).

To oversee its program, Civil Aviation has developed the following oversight activities to verify that the aviation industry complies with its regulatory framework:

1. Certification:
   
   Certification activities relate to granting and maintaining permission for people (i.e. pilots), organizations (i.e. airlines) and products and design (i.e. aircraft) to operate within the Canadian civil aviation system. This involves inspection of the enterprise or other verification activities and the issuance of certificates and licenses.
2. Surveillance (Inspections):
   
   Surveillance activities involve examining regulated enterprises to determine compliance with regulatory requirements. There are three main types of surveillance activities:
   
   i. Assessments: Evaluate effectiveness and level of compliance with regulations. It looks at all of the Safety Management System (SMS) components required in a regulated enterprise.
   
   ii. Program Validation Inspections (PVIs) (SMS and non-SMS enterprises): Examine a subset of an enterprise’s processes – one or more of the SMS components or other regulated areas.
   
   iii. Process Inspections (PIs): Review a single process as an entry point that could justify assessing other enterprise processes.
   
   For the most part, all surveillance activities are planned (see next section for details), however, a variety of “reactive” activities can be deployed. These include:
   • Unannounced Assessments, PVIs and PIs (in response to risk indicators, accidents or significant changes to an enterprise);
   • Aircraft ramp inspections;
   • Enhanced monitoring (in response to risk triggers or surveillance activities); and
   • Enterprise Corrective Action Plans (inspections resulting from follow-up on the corrective action plan submitted by an enterprise).
3. Enforcement and Investigations:
   
   Enforcement activities relate to forwarding the oversight findings of non-compliance to the Enforcement unit for follow-up when it is deemed that the non-compliance severity requires more than corrective action plans.
4. Education and Awareness:
   
   Education and awareness supports oversight through a variety of means. Namely, Civil Aviation may attend regional air shows, industry conferences, host education, awareness, and outreach sessions (most recent outreach platform relates to UAVs), engage national and international stakeholders, distribute media publications and establish a social media presence.

Planned Surveillance Activities:

Using information stored within NASIMS, Civil Aviation risk assesses all regulated enterprises on an annual basis. Based on this risk assessment, a risk profile for each enterprise is created (or maintained) and Civil Aviation develops a surveillance plan which strives to inspect each enterprise at least once in a five year period^{Footnote 9}. The type of surveillance activity will be dependent upon the enterprise’s risk profile. For instance, according to Civil Aviation’s “Surveillance Interval Matrix”, if an enterprise is considered to have a “negligible impact & low risk” profile, Civil Aviation is required to conduct a PVI (non-SMS enterprise) or Assessment (SMS enterprise) once every five years. Whereas if an enterprise is considered to have an “extensive impact & very high risk” profile, Civil Aviation is required to conduct a PVI once every year for non-SMS and SMS enterprises as well as an Assessment once every three years for SMS enterprises.

As surveillance activities are scheduled by each region, the inspection intervals are adjusted based on resource availability and not all regions have been able to meet the five year inspection cycle. Some regions are able to inspect all high, medium and low risk sites while other regions cannot.

As previously highlighted, Assessments and PVIs are typically planned while PIs are reactive. However, in 2016-017, in order to increase the inspection capacity to meet the NOP by leveraging leaner surveillance tools, Civil Aviation introduced a temporary change allowing all operational branches to plan and conduct PIs in lieu of up to 30% of planned PVIs.

It should also be noted that the results of all the surveillance activity, including findings of non-compliance, are stored in individual enterprise files. However, Civil Aviation does not have a centralized database(s) or a national process that consolidates surveillance results to track non-compliance findings and identify potential system-wide trends that could support their strategic level risk analysis. A recently deployed “NASIMS Monitoring Module”, once fully operational, would enable the collection of surveillance activity results including non-compliance findings; however, there are no plans yet to utilize the module for this purpose.

Activity Costing:

Inspectors and their supervisors are required to input their time associated with the categories noted below, into a system called the Activity Reporting and Standards System (ARASS) which was developed by TC. It should be highlighted that travel time is also tracked within ARASS, however it is tracked separately.

• Certification;
• Continuous Improvement;
• Enforcement;
• Engagement and Outreach;
• Leadership and Management;
• Program, Financial and HR Management;
• Regulatory;
• Surveillance (planned and reactive); and
• Training.

Civil Aviation has developed processes to forecast and cost its planned surveillance, reactive surveillance and certification activities.

The time standards associated with planned surveillance activities were established several years ago and have been used as the basis for resource allocation. However, the historical ARASS surveillance data could not be used as the basis to update the standards since the system did not differentiate between planned and reactive surveillance activities (e.g. planned or reactive assessments). With respect to reactive surveillance activities, ARASS did not provide set parameters which resulted in inspector / regional data entry inconsistencies. For example, some regions recorded time for follow-up activities, while others did not. Activities were also not well defined in ARASS for certification activities. These issues have impacted the accuracy of Civil Aviation’s ability to forecast.

The National Oversight Office (NOO), created in October 2015, was mandated to improve the accuracy of the time standards as well as improving clarity and consistency of ARASS data entry. To ensure that Civil Aviation can accurately forecast the cost of its oversight activities, the NOO has added subtasks for surveillance activities and defined the parameters for previously undefined activities within each category. The NOO also developed and implemented a new database, National Oversight Plan (NOP) Costing Methodology, which integrates ARASS with other systems, such as the Salary Management System (SMS) and Oracle. The automated costing formula calculates standardized costing estimates for all activities, which can be used for regional comparisons. Civil Aviation plans to use this new database to cost other activities such as enforcement and investigations as well as education, outreach and awareness. The rollout of this new system, as well as the new procedures and activity codes, started as of April 1, 2016 and the office has also carried out an education and consultation campaign to educate inspectors on these new changes. The NOO expects to review and update the time standards by January 2018.

Internal Audit Observations:

While the program has started to identify strategic system-wide risks, it is unclear how mitigations and risk-levels are monitored. As well, to better support the identification and analysis of system-wide risks Civil Aviation needs to fully implement its recently developed central repository of non-compliance findings.

Civil Aviation’s assessment of operators’ risk levels is a useful tool for planning annual oversight. However, it is not clear how many enterprises are risk ranked as high in any given year and what percentage of these are actually inspected. We understand that each year both moderate and low risk ranked enterprises are inspected. However, it is not clear how many resources are targeted to lower risk ranked enterprises and are thus not available to ensure all high risk ranked enterprises are inspected each year.

Finally, Civil Aviation recognizes the importance of improving its costing practices and is taking steps to ensure complete, accurate, and transparent cost data is available to support national resource allocation decisions.

Marine Safety and Security

The Marine Safety and Security Program is responsible for enabling safety, security, clean water and clean air outcomes in the marine aspects of Canada’s Transportation System.

Marine Safety and Security conducts activities to address regulatory requirements under the following strategic objectives: clean air, clean water, marine safety, and marine security. Marine Safety and Security’s Program is a certification-based system^{Footnote 10} with very prescriptive timelines for certification and oversight requirements. The majority of Marine’s oversight activities are prescribed in more than 100 acts, regulations, standards, memorandums of understanding (MOU) and memorandums of agreement (MOA). In addition, Marine inspectors also play a role in ensuring the safe design and construction of vessels. This is due to the fact that there is variability in design as the majority of vessels are custom built.

Risk Identification and Intervention Strategies:

Strategic Level Risks

In 2014-15, Marine Safety and Security developed an approach to assess risks at a strategic level. In 2015-16, Marine Safety and Security completed an environmental scan through an analysis of research papers and a survey of program staff. The resulting “National Risk Survey Report” categorized the risks into common categories and prioritized the risks based on staff input. However, the prioritization of these risks did not assess the likelihood and impact of each risk. This approach is being further developed and Marine Safety and Security intends to incorporate a broader view of transportation risks with more input from both internal (e.g., non-compliance data) and external sources (e.g., accident and incident data).

Interventions

Marine Safety and Security does not have a process to identify the most cost effective intervention measure, such as regulation versus education and outreach. There has been no performance assessment of existing intervention strategies to confirm that they are still required and/or are working effectively.

Although there is no process, there are some instances where Marine Safety and Security has systematically assessed the merits of various interventions to mitigate identified risks:

1. Liquefied natural gas (LNG) was identified as an area of risk and Marine Safety and Security established a working group to review this issue and make recommendations to senior management with respect to the best intervention approach to mitigate this risk. This working group continues to meet and will make recommendations to senior management on a mitigation approach.

2. A Concentrated Inspection Campaign (CIC) program is an intervention strategy which often involves further monitoring or targeted inspections. Results from inspections have been analyzed periodically. For example, in 2012, the CIC identified bulk carriers and structural safety as areas of risk. In 2014, the CIC identified fire safety appliances and lifesaving equipment for small to medium sized passenger vessels as a possible risk.

3. Some foreign vessels entering Canadian waters are considered a relatively higher risk prompting Canada to enter into two Memorandums of Understanding (Paris MOU and Tokyo MOU), as an intervention strategy to mitigate the risk. As a signatory to the MOUs, Canada has access to risk assessments of vessels based on data entered from each country that inspected the vessel prior to entering Canadian waters.

Oversight Strategy:

Marine Safety and Security’s National Oversight Plan (NOP) breaks down its population into the following common elements:

• Control of Foreign and Domestic Vessels;
• Marine Personnel Qualification and Protection;
• Protection of Marine Infrastructure and Navigation;
• Marine Environment (Clean Air and Clean Water); and
• Operations Surveillance, Incident Management and Enforcement.

These common elements are further divided into specific components:

• Domestic vessels
  • Large: => 24 meters
  • Vessels > 15 gross tonnes, < 24 meters or carry > 12 passengers
  • Small < 15 gross tonnes, < 12 passengers
  • Pleasure Craft with a focus on Boating Safety
  • Marine Security requirements apply to Ferries & Safety of Life at Sea (SOLAS) vessels

There were 4,888 inspections of domestic vessels completed in 2015-16.

• Foreign Vessel Oversight
  • Port State Control
  • Tanker Safety
  • Marine Security

There were 1,318 inspections of foreign vessels completed in 2015-16.

• Certification & Examination of Seafarers
  • Marine Medical Program
  • Oversight of Schools, Training Institutions and Placement Agencies
  • Pleasure Craft Operator Competency licensing

There were 15,611 seafarer certifications and examinations completed in 2015-16.

• Pollution Prevention
  • Ballast Water
  • Environmental Response Organization oversight
  • Oil-Handling Facilities oversight

There were 930 pollution prevention inspections completed in 2015-16.

• Safe Carriage of Cargo
  • Port Warden inspections
  • Transportation of Dangerous Goods oversight

These inspections were not tracked separately in the 2015-16 NOP.

• Ports and Marine Facilities security certification

There were 1,045 port and marine security certifications completed in 2015-16.

Marine Safety and Security operates under a regulatory regime that dictates the type of vessels and facilities that require certification and the regulations even prescribe the frequency of certification inspections. Due to the regulatory requirements for certain vessel types and facilities to be certified, Marine Safety and Security is aware of the existing population. However, it does not have a complete estimate of the population of small vessels or pleasure craft used by Canadians. As well, it does not have a formal process to identify new entities that are currently not regulated but may pose a risk. The focus of oversight is on currently regulated entities.

The 2015-16 NOP for Marine Safety and Security includes different types of vessel inspection activities carried out by Marine Safety and Security inspectors:

• Statutory inspections for certification, such as inspections of large domestic vessels for the purpose of issuing or maintaining various operating certificates. These inspections are scheduled at the request of the vessel operator and, as such, are always announced.
• Review of technical documentation supporting the compliance status of a vessel for the purpose of enrolling it in the Delegated Statutory Inspection Program (DSIP) for large vessels or the Small Vessel Compliance Program (SVCP) for small vessels. These inspections are always announced.
• Inspections of vessels based on risk evidence (planned and unplanned) for the purpose of verifying compliance. This type of inspection includes inspections of vessels enrolled in DSIP (including domestic tankers) and of small vessels. For all vessel types, Marine Safety and Security responds to complaints of deficient/defective vessels, and is on-site at most vessel accidents. These inspections may be announced or unannounced, depending on the situation.
• Inspections pursuant to international requirements, such as inspections of foreign vessels under the Port State Control regime, domestically-mandated inspections of foreign tankers and ballast water inspections of vessels in Canadian waters. These inspections are always announced.

Activity Costing:

To collect the time spent by inspectors on oversight activities, Marine Safety developed a database called the “National Time and Activity Reporting Management System” (NTARS). Time is tracked in detail by each type of oversight activity and although management time can also be tracked in the system, it is not consistently entered into the system. Travel time is tracked separately from the actual inspection time. Similar information is recorded in the self-developed Marine Security Information System (MSIS) for Marine Security activities.

Marine Safety defines the various types of activities that must be tracked, however there is no guidance provided to inspectors to help ensure consistent input of their time into the system. Marine Safety monitors and tracks inspectors’ input of time into the system, however there is no approach to help ensure the integrity of the data inputted at the modal level or even at the regional level.

Marine Safety and Security monitors some actual costs of some specific oversight activities for cost recovery purposes. It does not have a process in place to establish a standard cost per oversight activity nor does it perform any analysis of costs across regions for similar activities. The audit team did not see any analysis to ensure that resources are allocated to the regions consistent with regional workload.

Internal Audit Observations:

Marine Safety and Security is in a unique situation compared to other modes as an estimated 96%^{Footnote 11} of its resources are used to oversee regulations which dictate when a vessel or facility requires an inspection. The regulations that set the certification requirements and the cycle for certification inspections may have been risk-based when first established but there has not been an ongoing review of the regulations to assess if the certification inspection cycle is still warranted. Despite being bound by prescriptive regulations, Marine Safety and Security is working at developing and implementing a risk assessment process to better target their resources.

While Marine Safety and Security has the information available to complete standardized activity costing, there has been little analysis of the data to compare across regions. Marine Safety and Security should start using the data available to analyze costs of the various inspections, identify variances and follow up on anomalies to determine whether greater standardization or reallocation of resources may be required.