Evaluation of Safety Management Systems in Civil Aviation

July 2019

Evaluation report outlining results of the assessment of the Safety Management Systems (SMS) in Civil Aviation at Transport Canada as well as management’s action plan in response to recommendations identified in the evaluation.

Table of Contents

• Executive Summary
• Evaluation of Safety Management Systems in Civil Aviation
  • Introduction
  • Program Profile
    • Overview
    • Safety Management Systems at TCCA
    • Surveillance Methodologies at TCCA
    • Current Surveillance Approach
    • Current Status of SMS in Canada
  • Evaluation Approach and Scope
  • Evaluation Methods
    • Stakeholder Survey
    • Interviews
    • Case Studies
    • Document and Literature Review
  • Considerations
• Evaluation Findings
  • Section 1: Safety Outcomes of SMS
    • Summary Statement
    • Introduction
    • Organizations’ Views on SMS Safety Benefits
    • Civil Aviation Organizations and Key SMS Attributes
    • Empirical Data
    • Conclusion
  • Section 2: SMS for Smaller Operators
    • Introduction
    • Summary Statement
    • Conclusion
  • Section 3: SMS in Practice
    • Introduction
    • Summary Statement
    • Guidance and Training
    • Resources
    • Conclusion
  • Section 4: Long-Term Risks with respect to SMS
    • Conclusion
• Appendix A: Transport Canada SMS Requirements
• Appendix B: References
• Appendix C: List of Recommendations
• Appendix D: List of Findings
• Appendix E: Management Action Plan
  • Transport Canada Civil Aviation (TCCA) Description of the Current State of SMS
• Appendix F: Survey Questions
• Appendix G: Interview Questions
  • Interview Guide for External Stakeholders
  • Interview Guide for TC Staff
• Appendix H: List of Acronyms

Executive Summary

Since 2008, airline operators, private operators, approved maintenance organizations that service airline operator aircraft, air navigation services, and aerodromes/airports/heliports are required by Canadian Aviation Regulations (CARs) to have a Safety Management System (SMS) in place.

SMS is a documented process for managing risks that integrates operations and technical systems with the management of financial and human resources to ensure aviation safety or the safety of the public. The overarching intent of SMS regulations is for an SMS to become part of the culture of an organization and everyday work.

Transport Canada’s Evaluation and Advisory Services conducted an evaluation of SMS in Civil Aviation in 2017-2018 in order to answer two key questions: after 10 years of implementation, what has been the impact of SMS on aviation safety, and what lessons were learned from SMS roll out and implementation? We also conducted prospective analysis about the scalability of SMS.

The evaluation findings are based on multiple lines of evidence, including a survey of nearly 1800 industry stakeholders, case studies involving eight enterprises, more than 40 interviews and extensive document and literature review.

What we found

SMS Impact on Aviation Safety in Canada

Our evaluation found that:

• - A number of organizations have in place traits and practices that are indicative of an effective SMS, specifically in the areas of non-punitive reporting, executive commitment, and hazard identification and mitigation.
• - In those organizations a more systematic and documented way of identifying and addressing aviation risks is taking hold.
• - There is notable buy-in to SMS, in particular amongst large organizations.
• - Finally, we noted significant research evidence showing a correlation between SMS and improved safety performance.

Taken together, these suggest a positive impact of SMS on aviation safety in Canada over the last 10 years. However, due in part to data limitations, it was not possible to statistically attribute the extent of SMS’s contribution to aviation safety improvement.

SMS for Smaller Operators

SMS regulatory requirements have yet to be extended to most small operators in Canada. In this evaluation we inquired the extent to which SMS was scalable, that is, it could be adopted by a civil aviation organization regardless of the size of the organization. The majority of interview respondents believed that SMS would be useful to any size organization. Our survey and interviews showed that ‘documentation burden’; ‘cost considerations’; ‘training, awareness and competence’; and ‘lack of perceived benefits of SMS’ were key factors when thinking about SMS in smaller organizations.

In considering SMS scalability, TC should consider lessons learned from the initial roll-out in 2008. These are:

• - Provide more effective, clear and consistent guidance, both through TC materials and the interactions of inspectors with organizations.
• - Communicate well why having an SMS is better and why the cost is justified and that TC will deliver on providing effective support and guidance.
• - Conduct robust analysis of resource implications on TC of extending SMS to smaller organizations.
• - Ensure sufficient resources are available to maintain an adequate (however that is defined) level of ongoing oversight.

SMS Implementation

The evaluation examined how implementation of SMS regulations had an impact on the evolution of SMS, both within industry stakeholders and TC oversight practices. While there has been improvement, most notably during the past five years, the lack of detailed guidance to industry on how to properly implement an SMS within the enterprises, combined with the lack of timely and comprehensive training for TC inspectors, contributed to the confusion surrounding regulatory interpretations for several years post-implementation, something which persisted at the writing of this report. While enterprises are now more comfortable with SMS, there remains some disconnect in parts of the country between TC inspectors and industry that could be rectified to a large extent by improved training for TC inspectors, for industry, or joint training for industry and TC.

Long-term Risks related to SMS in Civil Aviation

Interviewees identified as key risks: TC’s perceived retreat from its oversight responsibilities, complacency or drift in organizations regarding SMS, shortage of qualified people, and lack of collaboration within the industry.

Recommendations

Our evaluation made the following recommendations:

Evaluation of Safety Management Systems in Civil Aviation

Introduction

Since 2008 a number of commercial operators, aircraft maintenance organizations and aerodromes began implementing Safety Management Systems (SMS) as it became a requirement under Canadian Aviation Regulations (CARs). Evaluation and Advisory Services conducted an evaluation of SMS in Civil Aviation in 2017-2018 in order to answer two key questions: after 10 years of implementation what has been the impact of SMS on aviation safety and what lessons were learned from SMS roll out and implementation? The primary intent is to contribute to policy discussions and decision making regarding the future direction of SMS in civil aviation.

The evaluation was included in the Department’s 2016-2017 Evaluation Plan and was conducted between July 2017 and November 2018.

Program Profile

Overview

The Transport Canada Civil Aviation Directorate (TCCA) is responsible for the regulation and oversight of civil aviation transportation in Canada. In the civil sector, TCCA oversees air operators, aircraft maintenance organizations, aerodrome operations, air navigation services, and aeronautical product designers and manufacturers.^{Footnote 1} As part of its regulatory role, TCCA is responsible for the development and distribution of policies, standards, regulations, and guidance and educational material.^{Footnote 2}

TCCA’s oversight activities are divided into two categories: service and surveillance. Services include licensing for pilots, maintenance engineers, and air traffic controllers; medical assessments of personnel; the certification of aeronautical products; confirming aerodrome safety; and granting operating certificates to air operators, aircraft maintenance organizations, and other operations in the industry. Surveillance involves conducting inspections and assessments of organizations to confirm compliance with the Canadian Aviation Regulations (CARs).

Safety Management Systems at TCCA

A Safety Management System (SMS) is a documented process for managing risks that integrates operations and technical systems with the management of financial and human resources to ensure aviation safety or the safety of the public. It is a risk-based and “business-like” approach to safety^{Footnote 3} - with clear accountabilities, policies, and structures where responsibilities and processes are explicitly defined and documented.^{Footnote 4} Individual organizations draft their SMS based on guidance from TCCA, which results in unique SMS tailored to each organization. The intent is for an SMS to become part of the culture of an organization and everyday work.

Internationally, SMS became a requirement of the International Civil Aviation Organization (ICAO) in 2006 as part of the State Safety Program requirements. As of 2013, with the adoption of Annex 19, SMS is an ICAO requirement for all aviation service providers.^{Footnote 5} This applies to operators who conduct international transport, flight training organizations, maintenance organizations, manufacturers, aerodrome operators, and air traffic services.^{Footnote 6} Although the SMS is a requirement, member states are permitted to inform ICAO if it is not possible within their individual context. Canada was the first member state to begin implementing SMS.^{Footnote 7}

In deciding to implement SMS, the overall anticipated result was stronger safety culture and general improvement of safety practices.^{Footnote 8} In addition to the ability to identify and manage any safety issues or risks before an incident/accident occurs, TCCA indicated that there would also be various benefits such as financial savings due to reduced incidents and accidents, improved efficiency and productivity within an organization.^{Footnote 9} In changing the Canadian Aviation Regulations (CARs) to include SMS, it was noted that no alternative existed and that SMS was likely to have a net positive benefit-cost impact.^{Footnote 10}

Surveillance Methodologies at TCCA

TCCA’s surveillance methodologies have changed significantly over the last decade. Prior to SMS implementation, TCCA focused on regulatory compliance through the examination of personnel, aircraft, and activities using audits and inspections. Inspectors looked at specific products and practices to see if they met the relevant regulatory requirements and standards.

At the time of SMS implementation, TCCA was also undertaking an overarching transition from its audit-based oversight to a more risk-based and systems-focused approach. New surveillance tools were developed: Program Validation Inspections, Process Inspections, and, for organizations who would have SMS, Assessments.^{Footnote 11} To support the implementation of SMS, TCCA surveillance resources were focused primarily on these broad assessments, which consisted of a thorough consideration of an organization’s entire system.

However, under this risk-based surveillance approach, all companies could not be inspected as part of planned surveillance due to high-risk priorities as well as TC resource limitations. In response to this and to feedback from inspectors and the Auditor General, TCCA began the Surveillance Program Evaluation and Update (SPEU) project in 2016. This project led to an updated surveillance approach, Surveillance 2.0, in effect as of April 1st, 2018.

Current Surveillance Approach

Surveillance can occur on two fronts: planned or reactive. Planned surveillance consists of activities scheduled at the beginning of each fiscal year based on a variety of risk factors and safety information, while reactive (unplanned) surveillance includes “activities conducted in response to an unforeseen event or issue (accident, incident, increase in an enterprise risk indicator level, etc.).”^{Footnote 12}

Traditional surveillance policy dictated that the examination of organizations occur at specific time intervals (and if not, the decision to delay surveillance was justified based on a risk assessment). With updates to surveillance practices, the goal became “a more robust risk-based decision making framework”^{Footnote 13} in order to ensure oversight activities are flexible and focused on risk rather than simply time. Surveillance 2.0 continues to strive for this.

Currently, there are multiple types of surveillance of regulated entities that TCCA can conduct: process-level surveillance, systems-level surveillance, as well as targeted inspections for companies not currently subject to SMS regulations.

As of 2018-2019, process-level surveillance is the primary means of oversight. Process-level surveillance looks at the functioning of a specific process within the SMS or other system to determine whether it is effective and compliant. TCCA manages this through a process inspection (PI). While originally used primarily as a reactive tool, PIs now allow for quick review and sampling to evaluate how an enterprise is operating. These inspections can lead to additional systems-level surveillance if needed.^{Footnote 14}

Choosing the process to inspect involves preliminary analysis to determine which process is most relevant to the reason or trigger for inspection. PIs may be planned, reactive, part of a systems-level check, or used to support oversight services (e.g., ramp inspection to support the addition of a new aircraft type).^{Footnote 15}

To assess a system, TCCA uses either assessments or Program Validation Inspections (PVI). The primary differences between assessments and PVIs are the points of entry and the timelines. Assessments are used for organizations that have a Safety Management System in place, thus making the scope or focus of the assessment the system itself. For PVIs, on the other hand, the scope is slightly smaller and focused on a unique system, program, or procedure (e.g., the Quality Assurance Program or training program).^{Footnote 16} Differences in timelines may occur throughout the surveillance process; for instance, in notifying an enterprise about upcoming on-site activities, 10 weeks’ notice is required for an assessment, as opposed to 6 weeks for the more concentrated Program Validation Inspection. In every other area, PVIs and assessments follow the same general process.

Current Status of SMS in Canada

In Canada, SMS is now in place for all airline operators, private operators, approved maintenance organizations who service airline operator aircraft, air navigation services, and aerodromes/airports/heliports. Further implementation was put on hold in 2009.^{Footnote 17} At the writing of this report, it has yet to be implemented with smaller operators (i.e., commuter, air taxi, and aerial work organizations), other approved maintenance organizations, flight training units, approved manufacturers, or aeronautical product certification.^{Footnote 18}

Evaluation Approach and Scope

The evaluation covers the period from 2005-06 to 2016-17. The evaluation scope includes all aviation organizations subject to SMS regulations (except for the two air navigation service providers) with an emphasis on commercial operators.

This evaluation focuses on the impacts of SMS implementation in the civil aviation industry, rather than the effectiveness of TC’s oversight. While surveillance practices and related activities are considered, this is to provide a clear picture of SMS, as the role of the regulator cannot be fully disentangled.

Evaluation Methods

Our evaluation employed multiple lines of inquiry, including a document and literature review, stakeholder survey, interviews, and case studies.

Stakeholder Survey

In November/December 2017, an anonymous online survey was deployed to a total of 1799 industry stakeholders, including 804 operators, 296 air maintenance organizations (AMOs), and 699 aerodromes. We received responses from a total of 360 industry representatives (213 operators, 62 AMOs, 85 aerodromes). Of these 360 respondents, 235 were classified as ‘small’ (1-10 employees) organizations, 72 were medium (11-50), 49 were large (51+), and 4 did not specify size. The organization size refers to number of employees within organization who have responsibilities defined under the Canadian Aviation Regulations (e.g. Pilot, Aircraft Maintenance Engineer, Cabin Safety Personnel, Person Responsible for Maintenance).

The overall response rate was 20%.

Figure 1: Survey respondant organizations by size

Figure 1: Text version

This figure is a bar graph showing the number of survey respondents by organization size. The far left bar shows that 235 respondents were classified as small organizations. The second-to-left bar shows that 74 were from medium organizations. The middle bar shows that 49 respondents were from large organizations. The far right bar shows that 4 did not specify size.

Below is the further breakdown of survey respondents. Contact information was derived from the TCCA Directorate database. Several cleaning and formatting steps were taken to arrive at the final contact list for the deployment of the survey, including the removal of organizations with no or incomplete contact emails, scanning through all emails and fixing formatting, aggregation of all emails into master list and removal of duplicates. In cases where there was identical contact email for several organizations, preference was given to operators with other entries removed from master list.

Table 1

	Aerodromes	Operators	AMOs	Total
Overall in the TCCA database	1982	824	776	3582
Master list for survey deployment	699	804	296	1799
Total responded	85	213	62	360
Partial and full SMS implementation	68	133	19	220
Voluntarily	6	72	7	85
Because of regulatory requirements	46	20	7	73
Both of the above	16	41	5	62

Survey questions explored the implementation and the current status of organizations’ SMS, inquiring about topics such as benefits of SMS, types and utility of guidance, and useful or challenging elements of SMS.

Interviews

Interviews were conducted in-person and over the phone with industry stakeholders, TC staff, and Transportation Safety Board (TSB) officials. These key informants provided detailed insight into SMS in aviation based on their experiences and perspectives. Separate interview guides were prepared for industry, TC, and TSB participants.

An introductory round of interviews was conducted with industry participants including representatives from 24 air operators and 2 aerodromes, totaling 26 interviews. The majority (19) of these organizations were considered ‘large’ with 51+ employees, with the remainder (7) having under 50 employees. The interviews were conducted by Evaluation and Advisory Services (EAS) in November and December 2017.

Between May and August 2018, EAS conducted interviews with 15 TC staff including Inspectors, Technical Team Leads, and higher level officials.

Three representatives from TSB participated in a group interview in September 2018.

Case Studies

To provide a more nuanced understanding of SMS in practice in the Canadian aviation industry, case studies were conducted in 3 provinces (Ontario, Alberta, and British Columbia) with a total of 8 companies. Case study companies were selected to ensure a regional spread of diverse operational contexts (e.g., coastal, commuter, oil and gas) and to include different sizes and types of operations (e.g., passenger versus cargo).

Site visits and interviews took place between May and July 2018. The case studies yielded 43 interviews with various company representatives, including: Presidents and Vice Presidents; Directors and Managers of SMS, Quality Assurance (QA), Maintenance, and Operations; Chief Pilots; and ground-level staff. In addition, 21 interviews were conducted with TC inspectors in the regions as part of the case studies.

Document and Literature Review

Key internal and external documents were reviewed to provide a full picture of safety management systems in theory and in practice, with special attention to SMS in Canada and SMS in aviation. The review considered Civil Aviation regulations and standards, internal and publically available guidance materials (e.g., Staff Instructions, Advisory Circulars, training presentations, and promotional documents), TC reports and presentations, findings by the Auditor General of Canada, and ICAO documents. This material was considered alongside literature by academics and experts exploring safety culture, the measurement of safety performance, and the development and impacts of SMS.

Considerations

We note the following considerations and limitations for this evaluation:

• 1) While an in-depth statistical consideration of quantitative safety data would be ideal, such an analysis was not possible at this time due in part to limitations in TCCA’s current data management tools and practices. TCCA provided information such as operator reports for case studies. However, through discussion with TCCA staff (representatives from both Aviation Safety Intelligence, and Policy and Regulatory Services), it became clear that more fine-grained data (e.g., accident rates per subpart based on aircraft movements or flight hours; specific safety issues identified by inspectors) could not be feasibly analyzed by the evaluation team, due to lack of availability and/or centralization of data.
• 2) We selected case study companies through consideration of traits such as organization type, size, and geographical context. As such, the selected organizations provided rich insights into SMS as applied and experienced in different contexts. However, participating companies were not selected through random sample, and may not be statistically representative of all SMS companies in Canada.
• 3) Finally, while our report offers insight into areas that are building blocks of safety culture in participating organizations, it is not a comprehensive assessment of the safety culture in those organizations. Assessing the safety culture of a group of organizations in a rigorous manner requires methodologies that were not feasible for this evaluation. For example, it is recommended at a minimum to deploy a survey that is ideally completed by all of the employees of an organization. ICAO also recommends this approach. As noted earlier, while we did deploy a large survey that was sent out to over 1700 civil aviation organizations, it was completed by organizational representatives only. For this reason, the survey focused on implementation rather than safety culture.

Evaluation Findings

Section 1: Safety Outcomes of SMS

Summary Statement

We found that a number of organizations have in place traits and practices that are indicative of an effective SMS; there are numerous examples of hazards being identified and measures being implemented to address those hazards. There is also notable buy-in to SMS, especially amongst larger operators, and many would have an SMS even if it was not a regulatory requirement. Taken together, these point to a positive impact on aviation safety.

Introduction

In this section we report on:

• - Stakeholder views on perceived SMS safety benefits, with a focus on our stakeholder survey;
• - The extent to which enterprises had in place key attributes and practices that are known to lead to an effective SMS; and,
• - Challenges related to quantitative analysis to attribute SMS impact on safety performance.

A key question we asked for this evaluation was: after 10 years of implementation, what has been the impact of SMS regulatory requirements on civil aviation safety? To answer this question, we focused on the basic common attributes of an SMS to ascertain – through case studies, interviews with civil aviation organization representatives and TC staff, a survey of civil aviation organizations, and a document review – how well some organizations have done with respect to those attributes, particularly the level of commitment and proficiency applied.

The rationale for focusing on key SMS attributes to gauge safety impact is two-fold. First, there are inherent challenges associated with trying to assess direct impact SMS on aviation safety using aggregate statistics of occurrence rates or over a period of time – something we discovered during our evaluation as well. There is wide support in literature that analysis based on aggregate statistics does not always provide optimal insight into SMS effectiveness – “Many modern approaches advocate the use of proactive measures such as safety climate, hazard identification, and the observed percentage of safe behaviour with the focus being on current safety activities to ascertain system success rather than on system failure.”^{Footnote 19} Second, SMS is as good as what everyone involved puts into it – “all the elements of a SMS can be in place, but if people are not using or following them, the efforts are futile.”^{Footnote 20} A comprehensive study which carefully examined research across the world on this matter concluded that the effectiveness of SMS may well not lie in the “specific components of the system, but rather in the level of sophistication and effort applied across the system.”^{Footnote 21}

Organizations’ Views on SMS Safety Benefits

In our survey, we asked organizations to rate, on a scale of 1 to 5, the importance of SMS in enabling their organization to address emerging safety risks, with 5 being ‘rely extensively on SMS’. The larger the organization, the more they relied on SMS to address emerging safety risks. The majority of medium (81%) and large (84%) organizations chose at least 3 out of 5; two thirds (66%) of larger organizations responded 4 or 5. On the other hand, just over a half (53%) of small organizations picked at least 3 out of 5.

Figure 2: The larger the organization, the greater importance it places on using SMS to address emerging safety risks

Figure 2: Text version

This figure is a horizontal stacked bar graph. Each line represents the importance of SMS to organizations based on their size. The bars show six levels that respondents could choose, ranging from 0 (do not use SMS) to 5 (rely extensively on SMS). The top bar displays responses from small organizations: 2% indicated a level of 0, 20% selected a level of 1, 24% selected a level of 2, 20% selected a level of 3, 23% selected a level of 4, and finally 10% selected a level of 5.

The middle bar displays responses from medium organizations: 4% indicated a level of 0, 9% indicated a level of 1, 7% indicated a level of 2, 32% indicated a level of 3, 30% indicated a level of 4, and 19% indicated a level of 5.

The bottom bar displays responses from large organizations: 11% indicated a level of 1, 4% indicated a level of 2, 18% indicated a level of 3, 33% indicated a level of 4, and 33% indicated a level of 5.

Another indicator of belief in safety benefits of SMS is seen when we look at organizations who have implemented SMS voluntarily. These organizations cite safety benefits as the top reason for implementing SMS, followed closely by positive impact on safety culture and continuous improvement, both key notions that underpin SMS.

Figure 3: Top five main reasons organizations voluntarily implemented SMS

Figure 3: Text version

This figure is a horizontal lollipop chart demonstrating the top five reasons organizations voluntarily implemented SMS. “Safety benefits” is listed at the top as being chosen over 100 times. This is followed by “positive impact on safety culture” (chosen between 75 and 100 times), “establish continuous improvement practices” (chosen between 75 and 100 times), “more effective and efficient operations” (chosen between 50 and 75 times), and finally at the bottom, “align with industry trend” (chosen between 50 and 75 times).

Most of the organizations we spoke with reported overall confidence in SMS, many highlighting its value-added as a systematic and documented way of identifying and addressing aviation risks. Overall, of the people we spoke to during the second phase of this evaluation, 59% (70% excluding those we did not ask) stated that SMS makes things safer for individual companies and 43% (60% excluding those we did not ask) stated that SMS makes things safer for the industry as a whole. An important observation for the evaluation is that several case study organizations as well as other industry participants we interviewed said that they would keep SMS even if it was no longer a requirement.

Civil Aviation Organizations and Key SMS Attributes

What follows are our observations on the extent to which enterprises had in place key SMS attributes that are known to be crucial for SMS effectiveness. We focused on the following attributes:

• Trust
  • Non-punitive reporting
  • Executive commitment
• Proficiency in investigations and analysis
• Continuous improvement

Trust

There is clear consensus in the literature and amongst interviewees that without buy-in and confidence in the system SMS will not be effective. The vital ingredient that drives confidence in the system is trust.

We examined two critical areas that both impact and are also shaped by the existence of trust in organizations in the context of SMS.

• Non-punitive reporting
• Executive commitment

Non-punitive reporting

Non-punitive reporting is a key feature of SMS. In order for an SMS to be effective, the culture in an organization must be informed, learning and reporting. If the culture does not encourage reporting, then a key foundation of this systematic approach to managing safety will be weak, as these reports feed into the data analytics, hazard identification and continuous improvement aspects of SMS.

Our survey results show that out of 17 SMS elements that are required by regulations, large organizations identified non-punitive reporting as the most important element of an effective SMS:

1. Non-Punitive Reporting Policy
2. Investigation and Analysis
3. Proactive Processes
4. Risk Management
5. Communications

For smaller organizations non-punitive reporting was less important (for these, the most important element was Training, Awareness and Competence, followed by Communications).

Almost all of the organizations we interviewed agreed that non-punitive reporting is the cornerstone of an effective SMS. This is true both internally to the companies, and in terms of interactions between TC and companies.

Case study companies by and large appear to have embraced non-punitive reporting practices. Amongst all the SMS elements we examined, non-punitive reporting is where we observed the clearest success. For the most part, case study companies felt that the culture in their companies was open and amenable to reporting. Most spoke at length about their reporting scheme and stated that employees felt comfortable reporting and were well aware of the system. Organizations tracked reporting and many improved the process over time, primarily to maintain employee engagement in reporting. For example, some organizations made it easy for employees to report in from anywhere by automating the process or setting up an application on pilots’ phones. One company encouraged pilots “to share how their day went [immediately] instead of waiting to fill something out on the computer three days later”. This led to better data. Some companies used practices such as sending a letter in response to each report to demonstrate that management is taking reports seriously. In general, organizations noted that since SMS implementation, employees use reporting to tell management if they have an issue with something, whether it is safety-related or otherwise. It has improved internal communications and, according to several organizations, has taken the fear out of communicating a problem to senior staff. Employees no longer feel an impulse to cover up an issue or a mistake because they know that reporting it will lead to a fix and not punitive actions.

The non-punitive aspect is significant and supported by literature. Studies surrounding blame indicate that where punishment is feared people are less open to sharing information or admitting mistakes. If a culture is instead built around openness and does not lead to punishment, individuals are more likely to report. The focus is then forward-thinking, looking for opportunities to learn and improve, rather than blame.^{Footnote 22} TC made clear to organizations at the outset that reporting was to be non-punitive.

We also noted a number of challenges. The success of non-punitive reporting policy led to a substantial increase in the number of reports in some organizations. 47% of our case study company interview participants (80% if we exclude those who did not mention it either way, positively or negatively) noted managing reports as a challenge.

Ensuring that the non-punitive policy is not abused and maintaining confidentiality (particularly in small organizations, where keeping reports anonymous may be difficult) were also noted as challenges. Some reported that getting new employees such as pilots hired from a smaller operation to trust that the system is, in fact, non-punitive can take time.

Executive Commitment

Executive commitment is a key element of establishing and maintaining an effective SMS (an accountable executive is required by SMS regulations). There is extensive literature that emphasizes the importance of leadership commitment to developing and maintaining an effective safety culture.

Most interviewees highlighted the importance of senior management commitment. Interviewees accept as true that management behaviour influences the culture of the organization, and most critically, determines the level of resources dedicated to an SMS: “[SMS] will never work if there is no senior management support”; “Top executive’s commitment is critical success factor”; “The culture within a company is much more a result of the management than the fact of an SMS”; “At the executive level, if they haven’t bought into SMS, communicated it, and aren’t supporting it financially (training) then it will be very difficult for it to function.” As an example, if the senior executive sends the wrong signals regarding non-punitive reporting, the system will likely not be effective.

Several case study organizations demonstrated fairly strong executive support for SMS: “Each time we propose changes, we always get the support/buy-in from management”; “When the accountable executive sends the message that he welcomes any report and only positive recourse will happen then safety becomes a responsibility at all levels. There are open conversations on hazards as they arrive. It leads to commercial success.”; “Has been challenging keeping up with the reports so we had to hire more investigators and the AE supported this.”

One executive noted that resources dedicated to SMS relative to other areas in an organization is a sound indicator of executive commitment to SMS. For practical reasons, we did not attempt to obtain this information from the organizations. However, as we report in the Continuous Improvement section, executive support gets tested most acutely when it comes to improvements that go above and beyond regulatory requirements.

Proficiency of Investigations and Analysis

Learning from events and the identification and reduction of hazards are crucial features of an SMS. Identification of safety hazards and corrective action to mitigate those hazards is a common and critical element of an SMS not only in civil aviation or transportation but in all sectors where SMS is used. In Canada, SMS regulations^{Footnote 23} require that enterprises have in place processes for: identifying hazards to aviation safety and for evaluating and managing the associated risks; internal reporting and analyzing of hazards, incidents and accidents and taking corrective actions to prevent their recurrence; and ensuring that personnel are trained and competent to perform their duties.

In our survey of aviation sector, out of 17 SMS elements, large organizations identified investigations and analysis as the second most important element of an effective SMS.

1. Non-Punitive Reporting Policy
2. Investigation and Analysis
3. Proactive Processes
4. Risk Management
5. Communications

A review of literature highlights the importance of analysis and investigations for an effective SMS: “A healthy safety culture … vigilantly remains aware of hazards and utilizes systems and tools for continuous monitoring, analysis and investigation.”^{Footnote 24} In fact, SMS can be understood as a “planned, documented and verifiable method of managing hazards and associated risks.”^{Footnote 25} Realistically, “if you are to prevent the reoccurrence of an event, you need to understand what caused the event, and put in place strategies such that those causes are prevented from occurring again.”^{Footnote 26}

In some case study organizations, sound practices of root cause analyses, investigations into hazards, and trend identification are occurring. Many of the organizations we spoke to felt that the way they conducted analysis, investigations, and trend identification has improved. Of our case study company participants, 60% (96% if we exclude those we did not ask) said trending was a key SMS benefit. The ability of SMS to identify trends that would not have been detected otherwise is viewed by many as the clear value-added of SMS. Put differently, a significant benefit of SMS is not simply catching a single event that is then corrected, but identifying a repetitive event that was thought to have been corrected. Many of the companies pointed to having no repeated events as an indicator of success.

Several examples were provided. One company had received an engine with the wrong data, showing half the hours that the initial data stated. An investigation led to corrective action, consisting of a change in process, where two people now received engines and validated each other’s work. To refine further, the company conducted an audit of all engines and every repair going back 30 years. In this case, SMS helping understand root causes resulted in a significant decline in the number of overruns. One company identified a trend involving extra weight on planes because of unloaded cargo. Another company created a working group to keep better track ground-handling procedures because a trend was noticed after a number of incidents. Yet another company found out they had never installed fire bags they had bought after an investigation into potential battery ignition – “this is how SMS works. It’s not just bird strikes and repairs; it’s also these other little things that you would never find until they cause you grief.”

The organizations that had in place sound practices regarding investigations and analysis emphasize timeliness and competence as key to effectiveness. Tracking trends and taking action to promptly and effectively address issues that have been identified is crucial. These organizations were keen to discuss and showcase their ability to “keep on top of things”.

Despite sound practices observed, in some organizations, resource- and time-related pressures or lack of technical expertise may be hampering the effectiveness of investigations and analysis. Some organizations reported that it can be hard to find the time for large scale investigations for severe and complicated incidents. Paradoxically, that may be due to success in the area of reports being generated by the system: “the worst enemy of the system might be its own internal success”. Others reported that lack of expertise can also be an issue. We’ve heard that organizations may not have people with investigations backgrounds and may therefore struggle with conducting root cause analysis. Several organizations reported that SMS requires skilled workers who understand SMS, the airline business, and how the two can work, and those types of workers can be hard to find.

Some TC inspectors expressed concern about some operators not assigning the appropriate level of risk to risk occurrences in order to avoid undertaking resource intensive investigations and analyses. They noted that occasionally companies tended to write SMS reports that suited them and “soft pedaled risk occurrences because it means they don’t have to do anything about it”. These inspectors point to an inclination on the part of some companies to assign risk based on the outcome rather than the potential hazard. We were provided several examples of incidents not being labeled as severe when they clearly appeared to be, and instead categorized as low-risk because they did not result in something serious. We note failure to assign the appropriate level of risk to occurrences as a key SMS risk and encourage a closer look by TC.

Another challenge that larger organizations in particular may be facing is optimizing the use of the significant amount of data generated by SMS. For example, in one large case study company, there is a feeling of having “hit the wall in terms of doing effective stuff - correlating events, audit observations with all the data churned [by SMS]”. Their aim now is to identify and understand leading indicators in order to detect issues that have the potential to turn into a hazard in the future. Solutions such as a redesigned framework in tandem with software improvements are being considered, which would allow for equating indicators from SMS and linking these to risk profiles.

As noted in the Non-punitive Reporting section, 47% of our case study company interview participants (80% if we exclude those who did not mention it either way, positively or negatively) noted managing reports as a challenge.

In our survey, we found that a majority of organizations rely on manual processes (e.g. spreadsheets, etc.) to process and manage their data (see Figure 4). As would be expected, the use of more advanced tools such as safety intelligence software is more prevalent amongst larger companies (see Figure 5).

Figure 4: The majority of organizations rely on manual process to manage and analyze safety data

Figure 4: Text version

A horizontal bar graph represents the processes companies use to manage and analyze safety data. The top bar indicates that 59% of survey respondents use manual processes (e.g., spreadsheets), followed by 16% using custom built IT solutions, 15% using safety intelligence software, 10% using other processes, and 5% who contract out to a provider.

Figure 5: For small and medium organizations, manual processes are most common, while more complex solutions are used by large organizations

Figure 5: Text version

This figure is a horizontal stacked bar graph indicating which processes were chosen by large, medium, and small organizations. The top bar indicates that large organizations (51+ employees) chose manual processes 10 times, safety intelligence software 19 times, custom built IT solutions 14 times, contracting out to a provider 4 times, and other 2 times.

The middle bar indicates that medium organizations (11-50 employees) chose manual processes 36 times, safety intelligence software 7 times, custom built IT solutions 11 times, contracting out to a provider 2 times, and other 6 times.

The bottom bar indicates that small organizations (1-10 employees) chose manual processes 84 times, safety intelligence software 7 times, custom built IT solutions 10 times, contracting out to a provider 6 times, and other 15 times.

Continuous Improvement

Continuous improvement is central to SMS and to the notion of safety culture: “A healthy safety culture actively seeks improvements.”^{Footnote 27} In our survey of civil aviation organizations, continuous improvement is identified as one of the top reasons for implementing SMS (see Figure 6).

Figure 6: Organizations cited safety conditions and continuous improvement practices as the top reasons for implementing SMS voluntarily

Figure 6: Text version

This figure is a horizontal lollipop chart listing the top reasons for implementing SMS voluntarily. The top bar shows that “safety benefits” was selected as a reason over 100 times, followed by “positive impact on safety culture” (chosen between 75 and 100 times), “establish continuous improvement practices” (chosen between 75 and 100 times), “more effective and efficient operations” (chosen between 50 and 75 times), “align with industry trend” (chosen between 50 and 75 times), “contractual obligations” (chosen between 25 and 50 times), “allocate resources based on better knowledge of risks” (chosen between 25 and 50 times), “competitive advantage” (chosen between 25 and 50 times), “Transport Canada promotion” (chosen just over 25 times), “interactions with partners or third party” (chosen just under 25 times), “increase access to market” (chosen just under 25 times), “insurance reasons” (chosen under 25 times), “ease of transition from existing Quality Management systems” (chosen under 25 times), “international requirements” (chosen under 25 times), “association encouragement” (chosen under 25 times), and finally “other” (chosen between 0 and 25 times).

Some case study organizations also cited continuous improvement as a key benefit of SMS. That said, we’ve heard that a challenge for both the regulator and the regulated organizations is making the shift from the mentality of focusing on regulatory compliance to relying on SMS. This challenge comes into sharpest focus with the notion of continuous improvement of safety, as resource and liability concerns at times may mitigate against safety improvements that are beyond regulatory requirements.

While some interviewees pointed out that they generally received support from the accountable executive for proposed changes and action was taken even if it was beyond regulatory requirements, we’ve also heard from others of instances where executives questioned the need for expending resources on improvements such as installing a piece of equipment that was not a regulatory requirement but would have had clear safety benefit, based on an investigation of a previous incident.

One relevant issue here is whether Quality Assurance (QA) extends to flight operations. While there is no regulatory requirement to do so, it is viewed as a sound practice. Our survey results show that for the majority of organizations surveyed, QA processes do extend to flight operations (see Figure 7). That said, we are also aware of organizations that discontinued this practice.

Figure 7: Overall, for the majority of organizations SMS quality assurance processes extend to flight operations

Figure 7: Text version

This figure is a horizontal stacked bar graph showing the use of SMS quality assurance in flight operations by organization size. The top bar indicates that of the large organizations (51+ employees) 34 chose “yes”, 4 chose “no”, 6 chose “not applicable to my organization”, and 1 chose “Don’t know”. The middle bar indicates that of the medium organizations (11-50 employees), 35 chose “yes”, 7 chose “no”, 14 chose “not applicable to my organization”, and 1 chose “don’t know”. The bottom bar indicates that of the small organizations (1-10 employees), 62 chose “yes”, 21 chose “no”, 33 chose “Not applicable to my organization”, and 2 chose “Don’t know”.

Liability concerns can also stifle continuous improvement. This is true for both the regulated organizations and the regulator. We’ve heard that organizations are wary of doing things like including best practices in their manuals that are beyond regulations, for fear that they may be held accountable if something goes wrong. Similarly, TC inspectors may not want to approve processes or approaches that are beyond regulations because they are also concerned about being held accountable if there is an occurrence (for example, by TSB).

The challenges associated with continuous improvement appear to be a natural consequence of operating for a long time in an environment characterized by a focus on compliance with regulations. If the apprehension we have heard about taking on the potential liability is representative of the aviation sector, this poses an interesting challenge, as continuous improvement is a central feature of SMS and one of its major selling points.

Empirical Data

There has been a downward trend in aviation accidents over the last decade in Canada. The Transportation Safety Board’s (TSB) 2017 statistical summary^{Footnote 28} shows that the accident rate in Canada’s aviation sector has decreased since 2007 (see Figure 8). This rate is based on all accidents (commercial or otherwise) reported to TSB and the total number of flying hours recorded by TC.

Figure 8

Figure 8: Text version

This figure displays a bar graph with an overlain line plot and a trend line. The bar graph shows the number of aviation accidents per year from 2007 to 2017. In 2007, there were under 300 aviation accidents. In 2008, 2009, 2010, 2011, 2012, and 2013, there were between 200 and 250 accidents. In 2014, there were just over 200 accidents. This increased in 2015 to between 200 and 250 accidents. In 2016, there were just under 200 accidents. In 2017, this increased slightly though still remained below 200.

The two overlain lines represent the accidents per 100,000 flying hours between 2007 and 2017 and the trend in this accident rate over time. These lines show a decline in the accident rate, from approximately 6 accidents per 100,000 flying hours in 2007 to approximately 4 accidents per 100,000 flying hours in 2017.

However, demonstrating the extent to which SMS contributed to this downward trend on the basis of objective safety data has proven challenging for our evaluation. A key hurdle is that the aircraft accident rate calculated as the number of accidents per hours flown or per number of movements (takeoff or landing) cannot be calculated by CARs subparts (e.g., 705 certificate holders) because TC no longer collects the hours flown or number of movements by CARs subparts (see Considerations section on page 13 for an explanation of the limitations the evaluation encountered in conducting statistical analysis).

To be fair, even if more fine-grained data was available, there are inherent limitations to this type of analysis. Attributing causes to safety improvements is complex as there are a variety of factors to consider, including the implementation of SMS and improved technology (e.g. better aircraft). At least eight interviewees echoed these views, many singling out improved technology as a key contributor: “Part of the problem when you look at the data saying aviation is safer now … look at the fleet. Airbus is safer. Those old aircrafts are being thrown out. There’s been a fleet renewal within civil aviation in Canada which has lowered the rates of incidents”; “[SMS] played a part in [improving] safety but advancing technology has had the biggest impact on safety”. Several interviewees pointed out that complying with IATA Operational Safety Audit (IOSA) also contributed to safety, for some, more so than SMS.

That said, there is ample research evidence that show a correlation between SMS and safety performance, including research that has studied various models and indicators of safety, factoring in safety climate, proactive hazard identification, and safety behaviours in addition to considering accident rates over time. For example, Yeun^{Footnote 29} analyzed the Flight Safety Foundation’s Basis Aviation Risk Standard audit data from 2011 to 2014 and found that as the number of audits increase, the number of observations generally do also, but in the years following SMS implementation the number of audit observations decreased. Similarly, a survey of companies with and without SMS in different industries found higher safety performance and better risk assessment amongst companies with SMS.^{Footnote 30} All of this suggests that SMS does improve safety performance and there is no reason to believe that Canada’s experience with respect to SMS in civil aviation has been any different.

Conclusion

We found that that in some organizations a more systematic and documented way of identifying and addressing aviation risks is taking hold, a measure of hazard identification and mitigation is occurring, and there is notable buy-in to SMS amongst large organizations in particular. Specifically:

• There is strong agreement amongst stakeholders, as well as independent research that when implemented well SMS will reduce risk in an organization’s operations. We have observed a number of organizations that appear to have in place traits and practices that are known to be conducive to an effective SMS and some basic SMS elements appear to be well implemented. That said, these sound practices and traits are not evenly distributed across organizations or key SMS attributes.
• Through interviews and visits to organizations that were part of our case studies, we compiled a number of examples of hazards being identified and measures being implemented to address those hazards.
• Finally, there is survey evidence showing that, amongst large organizations in particular, there is a belief that SMS enables them to address safety risks. The fact that a number of organizations we spoke to indicated they would have an SMS even if it was not a regulatory requirement is also indicative of belief in SMS effectiveness.

All of this is indicative of positive impact of SMS on aviation safety in Canada. However, due in part to data limitations and in part to inherent difficulties of attributing SMS impact statistically, we were not able to quantify the extent of SMS’s contribution to aviation safety.

Section 2: SMS for Smaller Operators

Introduction

One of our lines of inquiry is to ascertain the extent to which SMS is scalable, that is, could be adopted by a civil aviation organization regardless of the size of the organization. Specifically we wanted to know the aviation sector views whether and how effectively SMS could be implemented by smaller operators. The intent is to inform policy discussion and decision making at TC around potentially expanding SMS regulations to other operators.

The issue of scalability is forward-looking. The analysis in this section is therefore prospective, which is different from most evaluation methods, which are typically used to answer questions about what has happened in the past. However, evaluation tools can be used to provide useful information about priority policy issues.

Summary Statement

With respect to scalability of SMS, the majority of interview respondents believed SMS can be useful to any size organization, while some believed SMS would not add value to some of the smaller operations. There is agreement amongst interviewees that if SMS is extended to additional certificate holders, it will be important for TC to adapt the current SMS requirements to fit the need and capacity of small organizations. Our survey and interviews show that documentation burden; cost/resource considerations; training, awareness and competence; and lack of perceived benefits of SMS are factors that are much more significant for smaller organizations than larger ones.

According to TSB, the basic principles of SMS are scalable:

1. Commitment from senior management to operate safely
2. A process based on the ability to identify and document operational risk and undertake actions to mitigate risk
3. A feedback scheme such as a reporting system

Many interview participants feel that SMS is valuable and possible for an organization of any size: “Aviation companies shouldn’t be running without a valid SMS program”; “702s and 703s should absolutely have SMS.”; “It’s a culture, you don’t need a large staff.” Some interviewees, however, are skeptical of whether SMS makes sense for some of the smaller organizations. What follows is a sample of responses that illustrate this view. “There definitely isn’t a perfect solution. A lot of bush / northern operators have one or two airplanes, the owner is the chief pilot and PRM and he might have two pilots who seasonally work for him and he’s probably 70 and his wife does all the paperwork. When you do an inspection, you go to his house. A good 25% are like this. In this context, SMS doesn’t make sense – waste of time and effort.”; “I really don’t think SMS belongs in an airport with one employee. It’s unrealistic to think that this could work and that they need it.”; “[I] don’t think it would be an effective tool for them. The idea behind SMS is a good one. But … [to] have small companies document what they’re doing would be a huge burden on finances and documentation.”; “SMS tries to manage safety issues in large organizations but for a small organization, it is a complete waste of time and money. Safety in small organizations should be a discussion – issues can be managed by just talking to each other, don’t need all the documentation.”

A few interviewees provided a “size threshold” below which they believe SMS would not be beneficial: “5-7 people? Doesn’t need SMS. In a small operation, need TC to come in and see you’re following safety protocol. In this context, safety can be easily and quickly assessed.”; “For a 2- or 10-person organization, it doesn’t make sense. Upwards of 50, SMS is fitting.” However, most interviewees did not provide a specific threshold when directly asked.

Overall, the majority interview respondents believed SMS can be useful to any size organization, and some believed SMS would not add value to some of the smaller operations. However, there is notable agreement amongst those we spoke to that if SMS requirement is extended to smaller organizations, TC will need to adapt the current SMS scheme to fit the need and capacity of these small organizations.

When prompted on how to adapt the current SMS scheme to fit the need and capacity of these small organizations make SMS, there weren’t many specific ideas put forth. When directly asked whether it’s a question of having fewer elements than the 17 found in the current SMS regulatory requirements, almost no respondent agreed (one interviewee likened the idea to having a car with three wheels). When asked whether it’s a question of making the expectations that define these elements less exacting, there were more who agreed. However, most of the comments centered on capacity of smaller organizations.

What follows are considerations when thinking about expanding SMS to additional certificate holders – in particular the small operators.

In addition to significant cost and resource concerns for medium and small organizations discussed in the previous section, documentation burden, as well training, awareness and competence are also key hurdles. In our survey, amongst all SMS elements, small organizations flagged these two SMS elements as most challenging to implement. As seen in Figure 9, smaller organizations identified documentation, records management, and identification and maintenance of applicable regulations the most challenging to implement. Several interviewees also flagged documentation burden as an important obstacle when thinking about SMS in small organizations. On the basis of our survey and interviews, it is clear that documentation requirements will need to be trimmed for small operators.

Training, awareness, and competence is one SMS element where we find the sharpest contrast between large organizations and small/medium organizations. In our survey, it was identified as challenging to implement both by small and medium size organizations (23% and 28%, respectively), much more so than large organizations (4%). This places onus on TC to provide appropriate support to smaller organizations, in particular guidance. A lesson learned for the initial implementation noted in this evaluation was that TC’s guidance materials were viewed as lacking.

Figure 9: For several SMS elements, a significantly higher proportion of small organizations identified them as challenging to implement, compared to medium and large organizations

Figure 9: Text version

This graph demonstrates the most challenging elements of adopting SMS by organization size. SMS Documentation was indicated as a challenge by 24% of large organizations, 35% of medium organizations, and 41% of small organizations. Records Management was indicated as a challenge by 16% of large organizations, 16% of medium organizations, and 25% of small organizations. Identification and Maintenance of Applicable Regulations was indicated as a challenge by 9% of large organizations, 19% of medium organizations, and 31% of small organizations. Training, Awareness, and Competence was indicated as a challenge by 4% of large organizations, 23% of medium organizations, and 28% of small organizations.

Lack of perceived SMS benefits is another key challenge when thinking about SMS adoption by smaller companies. In our survey, this viewpoint is cited as a key reason for not adopting an SMS, more so than financial reasons.

Some interviewees indicated that elements such as communication may be easier for smaller organizations, as all employees would likely be located in the same building. Other interviewees were concerned that, considering cost is the big driver civil aviation, not requiring all carriers (e.g., 704 certificate holders) to have an SMS creates an uneven playing field.

When considering how SMS may work with small organizations, the experience of small aerodromes provides useful insight. We’ve heard that several small aerodromes dislike the additional regulatory burden and don’t necessarily view it as vital for safety. Some TC inspectors we interviewed say they noted a lack of buy-in from some airports, as they don’t see the value of SMS. Very small airports don’t have the capacity to handle the administrative burden resulting from SMS. These have only a few staff, with little or no experience with SMS in many cases. In addition to other duties, responsibilities associated with SMS can be overwhelming: “one person managing an airport who does basically everything, you already have a guy who is putting in 60 hours a week, and now he’s supposed to have weekly safety meetings and things on top of this. SMS is geared for a bigger organization and the requirements are out of place for these smaller organizations”; “Manager of garbage within a town is also the airport manager, and you’re asking that person to also upkeep an SMS?” In addition to extra workload, maintaining an SMS requires skills that some airport managers might not have, such as computer skills and ability to understand all the different processes, and the towns don’t have the budget to hire a more suitable person.

The experience of smaller aerodromes show that in their current form SMS regulatory requirements are viewed as overly burdensome to implement and maintain, with no clear safety benefits, reflecting the results of our survey. If the SMS requirement is extended to additional certificate holders, it would be reasonable to assume that the experience of smallest operators (of which there are many) would not be significantly different than that of small aerodromes.

Conclusion

In considering the scalability of SMS, TC will need to heed the lessons learned from the initial roll-out. For example, TC will need to ensure that more effective, clear and consistent guidance is provided, both through its guidance materials and the interactions of inspectors with organizations. TC will also need to be effective in articulating clearly to smaller organizations the benefits as well as resource implications of adopting whatever form of SMS it proposes to facilitate buy-in. In other words, TC needs to communicate well why having an SMS is better, why the cost is justified, and that it will deliver on providing effective support and guidance.

Another key lesson learned from the initial SMS roll-out is that there needs to be robust analysis and realism about the resource implications on the Department of extending SMS to smaller organizations. Numerous interviewees noted that TC underestimated resource implications of initial SMS roll-out and subsequent oversight. The Auditor General found in 2008^{Footnote 31} that TC did not appropriately document risks and identify personnel needs. It found in 2012^{Footnote 32} that TC’s human resources planning had improved, though substantial work remained to determine the actual resources needed for surveillance. Regardless of the type of SMS configuration (e.g., SMS-light), extending SMS requirements to additional small organizations – which make up the bulk of the industry with over 2900 entities – has the potential to place significant pressures on TC. Resources will need to be expanded by TC for the roll-out – inspectors need to be fully trained and available to guide small organizations (of which there are a considerable number) in their implementation, data collection and analysis to ascertain the impact of SMS, and useful guidance materials prepared. In addition, TC will in all likelihood need to promote and facilitate information-sharing between small organizations more extensively.

Finally, TC will need to ensure sufficient resources are available to maintain an adequate (however that is defined) level of ongoing oversight. As one key informant stated “smaller operators may not have the resources to develop an SMS and might go in a problematic direction without proper oversight.” This is in line with literature that in terms of regulatory approach small organizations tend to require more deterrence rather than large ones. However, “unless it is carefully targeted, [this approach] can actually prove counterproductive, particularly when it prompts firms and individuals to develop a culture of regulatory resistance.’”^{Footnote 33}

Section 3: SMS in Practice

Introduction

In this section we discuss issues related to the practice of SMS both within TC and amongst industry stakeholders. The manner in which regulations are initially rolled out can have an impact on their successful implementation. Therefore, the evaluation conducted document review, the stakeholder survey, and key informant interviews with both TC staff and case study companies to determine ongoing effects from TC’s initial SMS implementation.

Summary Statement

The evaluation examined the way implementation of SMS regulations had an impact on the evolution of SMS, both within industry stakeholders and TC oversight practices. Three key themes emerged from interviews with both TC staff and case study participants: guidance; training; and resources. The lack of detailed guidance to industry on how to properly implement an SMS within their company, combined with the lack of timely and comprehensive training for TC inspectors, contributed to the confusion surrounding regulatory interpretations for several years post-implementation. While this has improved over time, issues of inconsistent interpretation of regulations and confusion in the sector appeared to persist at the writing of this report.

Literature shows that SMS can result in cost savings, primarily through accidents and incidents averted. Case study interviewees supported this and provided examples of savings, such as reduced insurance premiums. However, it is not clear to what extent savings from SMS offset the cost of system implementation and maintenance.

Guidance and Training

Guidance Materials

Overall, both TC and industry interviewees reported that SMS was introduced without proper preparation on both sides. Many interviewees felt that TC assumed that industry would figure SMS out on their own.

TC’s intention was for SMS implementation guidance materials to be useful for all sizes and types of organizations. In theory, providing more generalized guidance would encourage companies to design an SMS that fits their operational needs. However, in practice, industry key informants found the lack of specificity significantly limited the guidance’s usefulness. Both TC and industry interviewees stated that TC’s guidance materials were unclear and ineffectual.

Interviewees suggested that TC could provide more training for industry to ensure a consistent understanding of SMS and TC’s expectations. Among all interviewees, 43% (or 92% excluding the interviewees who provided no answer to this question) stated that TC could provide more training. From industry’s perspective, TC’s templates are not enough guidance to demonstrate how to develop and operate an SMS. Similarly, TC inspectors agreed that proper training and outreach for industry could reduce the burden on inspectors by filling in common information gaps. Alternatives to TC-led training include ensuring that pilot schools and technical training programs introduce SMS concepts to industry employees early on. Similarly, joint information or training sessions between TC and industry could ensure that there is a common understanding of SMS on both sides.

According to industry interviewees, the implementation process would have been smoother if TC offered clearly defined expectations for the SMS and examples of final products. In addition, industry felt that the guidance documents were written in complex language that was difficult to understand and interpret. Plain language, examples of SMS on paper, and templates would have assisted companies in knowing where to start. According to industry interviewees, implementation was a confusing process but in some cases was made easier thanks to specific TCCA teams and staff members.

Due to the generic nature of TC’s guidance materials, the evaluation’s survey results show that industry turned to training and advice from external consultants and events organized by industry associations to help fill gaps in information. In all, industry found that non-TC sources of information were more useful than TC’s documentation.

Figure 10: Overall top five guidance on the implementation of SMS received by organizations

Figure 10: Text version

This figure is a horizontal lollipop chart listing the top five types of guidance and their sources on the implementation of SMS received by organizations. The top bar shows that “documentation/guide from Transport Canada” was chosen over 150 times, “training from external consultants” was chosen between 100 and 125 times, “expert advice from external consultants” was chosen approximately 100 times, “conferences/symposiums by industry associations” was chosen between 75 and 100 times, and “documentation/guide from industry associations” was chosen over 75 times.

Figure 11: Overall ranking of sources of guidance on the implementation of SMS received by organizations

Figure 11: Text version

This figure is a horizontal lollipop chart listing the top five sources of guidance on the implementation of SMS received by organizations. The top bar shows that “external consultants” was chosen 327 times, “industry associations” was chosen 312 times, “Transport Canada” was chosen 292 times, “peer organizations” was chosen 181 times”, and “international associations” was chosen 107 times.

Industry personnel also relied on external consultants to help navigate SMS implementation. While some industry interviewees benefited from consultant support, others noted that hiring a consultant is extremely expensive and could easily lead to generalized processes and manuals that do not fit the company’s unique needs, which would negatively impact SMS effectiveness.

Training for TC Inspectors

The Auditor General of Canada noted in its May 2008 report that TC provided SMS training to inspectors and engineers in 2004 and 2005, and that “many took the training well before they could apply it. This limited the course’s effectiveness.”^{Footnote 34} The 2008 report also noted that “no regular recurrent SMS training has been planned.”^{Footnote 35} TC interviewees echoed the concern that their training during implementation was offered too far in advance of when they were expected to put their skills into practice.

Following up in 2012, the Auditor General reported that, “by the end of March 2011, a majority of inspectors had received mandatory training on the concepts and principles of SMS, on proactive interview skills, and on quality assurance. However, at that time, only 40 percent of inspectors had been trained on the new surveillance methodology. As a result, we noted that many inspections were carried out in 2010–11 by inspectors who had not received this training.”^{Footnote 36} When we conducted the evaluation in 2018, the issue of training was still a prominent concern among TC interviewees who felt that it was lacking, too superficial, and in some cases non-existent. There remained inspectors who felt ill-prepared to enforce the new regulatory requirements and, as has been previously stated, this led to inconsistent understandings among inspectors. In sum, this was a common problem during the implementation process, but concerns appear to persist in 2018.

TC interviewees noted that training opportunities have not grown, but have reduced from multi-day classroom sessions to e-learning formats. Some inspectors noted that it is difficult to confidently fulfill their top mandate activity of surveillance when there is no recurring training. Neither is there training to update inspectors on changes such as the introduction of targeted inspections, or updates to the surveillance program.

TC’s introduction of expectations in 2013 exacerbated already existing confusion over the implementation of judgement-based findings. TCCA introduced expectations in order to “define the intent of regulatory requirements”.^{Footnote 37} TC’s staff instructions noted that “there may be several ways for an enterprise to meet an expectation in an effective manner”.^{Footnote 38} Findings could be issued against regulations but not against expectations, however the expectations remained until 2018 and confused both inspectors and industry. From industry’s perspective, expectations added another layer of compliance that was not officially regulated by TC. The Auditor General of Canada agreed that SUR-001 “establishe[d] a vague policy with subjective guidelines and does not provide TC or carriers with clear and objective legal standards”.^{Footnote 39}

For the training that is offered, inspectors feel that the content is not always relevant, sufficient, or consistent. Some inspectors also note that due to the custom-fit nature of SMS, each company presents unique challenges in real-world situations that the basic training does not take into account. For example, the training may instruct inspectors to review the organization’s documents before arriving for the inspection, but if the organization does not have electronic copies of their files, inspectors need to plan alternative actions. In another example, training does not account for operators without internet that would have to be notified about the inspection by mail in addition to other work-arounds. Inspectors feel that training focuses too much on theory rather than SMS in practice and that instructors could benefit from experience conducting SMS inspections in the field.

Some TC inspector interviewees indicate that after ten years of implementation, they are finally beginning to understand SMS enough to provide confident oversight. It is also important to note that some organizations also noted improvement: “[we are] seeing that it’s getting there after 10 years of implementation”. That said, the issue of inspector training has not gone unnoticed in the industry, and is viewed as in need of improvement. As one key executive stated, if TC is looking for SMS competence and skill in the industry, then it needs to ensure that those who are responsible for oversight are also competent and skilled: “Give [inspectors] the tools and the training they need … there is inconsistency between what TC requires of others and what it exhibits.”

The evaluation found that while inspectors may differ in their preferred methods and opinions within a given TCCA office, they believe differences are more pronounced between regions. Interviewees state that there was no regional cross-training during implementation and minimal communication about SMS currently occurs between inspectors in different regions. We note that many case study organizations acknowledged that consistency had greatly improved in the decade since SMS implementation, particularly within each regional office. However, a lack of uniformity still exists and is most apparent between regions. While noting their frustration at the implementation process, many industry interviewees praised the commitment of their primary TC inspectors in guiding them through both SMS implementation and ongoing execution, especially given the absence of clear TC guidelines.

Some of these issues are echoed in comments from inspectors given to TC’s Multimodal Integrated Technical Training (MITT) group regarding SMS training in the Department. MITT is responsible for developing and delivering standardized technical training to TC’s Safety and Security staff. According to MITT, while the training is generally considered informative for what it is, feedback has also consistently included: concerns about the use of e-learning over classroom training; the desire to better understand SMS requirements and how to assess them; the need for a more practical (rather than generalized and theoretical) look at SMS in real-world contexts; and the need to update the training for current surveillance practices and regulatory requirements. It should be noted that in 2019/20, MITT is updating its SMS modules to address these and other issues, as well as developing training for the State Safety Program (SSP) to be rolled out prior to ICAO’s Annex 19 coming into effect in November 2019. Specifically, there will be added focus on SMS requirements, practical activities to apply what is covered in e-learning and in class, safety culture, and links between SMS, surveillance, and SSP.

While many case study interviewees confirmed that TC findings are helpful in strengthening their operations and closing gaps, several revealed that findings can be difficult to decipher and action. Echoing comments heard about advisory circulars during implementation, the most common complaint from operators is that current findings are not written in plain language.

Operators felt that the findings they receive weeks after the inspection is complete can be unpredictable and unrelated to the topics discussed at TC’s post-inspection debrief. On the other hand, inspectors affirm that they thoroughly discuss findings with Primary Operations and Maintenance Inspectors before submitting findings to companies.

TC interviewees noted varying levels of understanding between companies and thus varying levels of compliance. Some organizations agree that their lack of understanding of SMS has translated to an inability to comply with the regulations. As one key stakeholder noted “biggest issue with [SMS] regulations – tell us clearly what the regulation means so we could comply.” Organizations unanimously expressed that they would benefit from access to the newest versions of TC’s standards so that both industry and regulator are following the same documentation.

The implementation of SMS was a heavy lift for organizations, requiring significant change in the form of new policies, procedures, manuals, and experts to support the system, and for some, considerable culture change. The feeling in the industry is that TC was not always helpful during this period. When surveyed about the most difficult aspects of implementation, industry listed the relationship with TC, second only to costs associated with maintaining SMS.

Figure 12: The most difficult aspects of SMS implementation were costs associated with maintaining SMS followed by relationship / partnership with TC, and employees' resistance to change

Figure 12: Text version

This figure is a horizontal lollipop chart that lists the most difficult aspects of SMS implementation. The top bar shows that “costs associated with maintaining SMS” was chosen approximately 100 times. This is followed by “relationship/partnership with TC” (chosen nearly 100 times), “employees’ resistance to change” (chosen nearly 100 times), “costs associated with implementation” (chosen between 75 and 100 times), “capacity to implement” (chosen between 75 and 100 times), “ability to implement organization changes” (chosen between 75 and 100 times), “long-term planning and decision making” (chosen over 75 times), “recruitment for roles and responsibilities related to safety” (chosen between 50 and 75 times), “internal communications” (chosen over 50 times), “relationship/partnership with peer organizations” (chosen between 0 and 25 times), and finally “relationship/partnership with industry associations” (chosen between 0 and 25 times).

Ten years later in 2018, case study interviewees attest that TC and industry have built a good working relationship that functions on open communication from both sides. Companies reach out to TC inspectors if they have questions or need clarification about SMS. If companies are going through large organizational changes such as new management, a new aircraft, or relocating the base of operations, they will invite TC to visit their facilities. This not only helps build a positive and open relationship between industry and inspectors, but also offers a layer of transparency that can simplify the process of approving SMS manuals, since inspectors are already familiar with company operations and recent changes.

A strong relationship between TC and industry is viewed also as helpful in managing inspection findings. Industry interviewees confirm that their TC inspectors will advocate on behalf of the company to the larger SMS inspection team, since they are familiar with the company’s organizational context. This transparent relationship helps TC write more targeted findings which lead to more meaningful corrective actions.

Resources

As a recognized best-practice and ICAO requirement, there was really no alternative to implementing SMS in Canada. The Regulatory Impact Analysis Statement^{Footnote 40} stated that implementing SMS regulations would help to address safety issues, strengthen the industry-TC relationship, and improve safety practices and safety culture overall. In both 2005 and 2007, the RIAS^{Footnote 41} stated that no alternative to SMS implementation existed. The RIAS, along with guidance and promotional materials available publically, highlighted the economic argument for SMS in addition to the safety rationale.

Case study company interviewees were asked about the challenges of SMS. Among the industry interviewees who discussed resources as an SMS challenge, 87% agreed that both human and financial resources are a challenge to running SMS (47% of all company interviewees). Interviewees noted that they would have appreciated a more realistic messaging from TC regarding the cost and workload involved in implementing and maintaining an SMS before the work began.

Nevertheless, literature such as research by Lercel et al.^{Footnote 42} demonstrates that SMS can result in saved costs by reducing the number of accidents and incidents along with other savings such as decreased insurance premiums. Lercel^{Footnote 43} argues that at the macro-level, an accident has various direct (e.g., damage to airplane) and indirect (e.g., loss of public confidence) costs, which can impact market value. At the micro-level SMS can lead to savings through reducing time lost and workers compensation costs.

Due to the diversity of organizations within the civil aviation industry, it is not currently possible to estimate a representative average savings generated by putting an SMS in place. However, savings are clearly a consideration when thinking about SMS. Our stakeholder survey found that organizations that voluntarily implemented SMS did so at least in part in consideration of its financial benefits. Specific benefits included: more efficient operations, competitive advantage, increased access to market, and insurance savings.

Figure 13: Organizations cited safety considerations and continuous improvement practices as the top reasons for implementing SMS voluntarily

Figure 13: Text version

This figure is a horizontal lollipop chart listing the top reasons for implementing SMS voluntarily. The top bar shows that “safety benefits” was selected as a reason over 100 times, followed by “positive impact on safety culture” (chosen between 75 and 100 times), “establish continuous improvement practices” (chosen between 75 and 100 times), “more effective and efficient operations” (chosen between 50 and 75 times), “align with industry trend” (chosen between 50 and 75 times), “contractual obligations” (chosen between 25 and 50 times), “allocate resources based on better knowledge of risks” (chosen between 25 and 50 times), “competitive advantage” (chosen between 25 and 50 times), “Transport Canada promotion” (chosen just over 25 times), “interactions with partners or third party” (chosen just under 25 times), “increase access to market” (chosen just under 25 times), “insurance reasons” (chosen under 25 times), “ease of transition from existing Quality Management systems” (chosen under 25 times), “international requirements” (chosen under 25 times), “association encouragement” (chosen under 25 times), and finally “other” (chosen between 0 and 25 times).

64% of interviewees also agreed that cost-savings are a benefit of SMS implementation, although savings may not directly translate to increased profit for the company or savings for the customer. Lower insurance costs, less time lost to injury were mentioned as sources of savings. Some case study interviewees noted that there are businesses, particularly in the oil and gas sector, will not retain the services of an air operator that does not have an SMS.

Survey results of civil aviation organizations show that among small organizations 32% cited financial burden as the reason for not implementing SMS while 23% cited insufficient time and resources. Among medium organizations, however, a slightly less 27% cited financial burden whereas 53% mentioned insufficient time and resources as reasons for not implementing SMS.

Figure 14: Small and medium organizations highlighted various reasons for not implementing SMS, including no perceived benefits and resource challenges

Figure 14: Text version

This figure is a horizontal clustered bar graph representing the various reasons organizations did not implement SMS. The top cluster indicates that 25% of large organizations, 53% of medium organizations, and 55% of small organizations stated that “SMS is not required for our business.” The second cluster indicates that 75% of large organizations, 33% of medium organizations, and 51% of small organizations stated that “SMS regulation does not apply to the organization.” The third cluster indicates that 60% of medium organizations and 23% of small organizations stated there were “no perceived SMS benefits.” The fourth cluster indicates that 53% of medium organizations and 23% of small organizations also responded “insufficient time and resources”. The fifth cluster indicates that 27% of medium organizations and 32% of small organizations responded “financial burden”. The second from the bottom bar indicates 8% of small organizations responded “other”. The bottom cluster indicates 7% of medium organizations and 8% small organizations responded that they “would not know how to start.”

Interviews confirmed that smaller companies may struggle more than their larger counterparts to implement SMS because of the human and financial resources required. To create and run an SMS, companies may need additional staff, such as a dedicated SMS manager or external consultants to advise in the system’s development. TC interviewees estimate that purchasing existing software or building an internal system could easily double in price after considering employee time and other in-house costs. Where larger companies may have the flexibility to allocate these additional resources to SMS development and maintenance, small organizations do not have the same capacity and therefore feel the financial and administrative burden of SMS more severely.

Conclusion

While there has been improvement, most notably during the past five years, issues of inconsistent interpretation of regulations and confusion in the sector persisted at the writing of this report. For both TC staff and industry the lack of consistent, timely, comprehensive training for TC inspectors is an ongoing challenge. A lesson learned from the initial implementation is that more effective guidance and training should have been provided to industry in order to ensure that they understood requirements and to help companies implement an effective SMS. While companies are now more comfortable with SMS, there remains a disconnect in some parts of the country between TC inspectors and industry that could be rectified to a large extent by improved training for TC inspectors, for industry, or joint training for industry and TC.

Literature shows that an SMS has the potential to alleviate pressure on resources. However, there is also a financial investment required for implementing, maintaining, improving and in TC’s case, adequately overseeing these systems. These costs are of particular concern for smaller organizations.

Section 4: Long-Term Risks with respect to SMS

We asked all interviewees to identify long-term risks the civil aviation sector (including TC) should be aware of regarding SMS. What we’ve heard could be grouped into four areas.

1. Several external interviewees expressed concern that TC appeared to be retreating from its oversight responsibilities and identified this perceived withdrawal as the primary long-term concern. Among many comments we heard were: “we shouldn’t go back to the practices of the ‘old days’, but it seems like we’re going too far in the other direction, with TC pulling back from the industry.”; “Dialogue with TC is being reduced at various levels.” A number of interviewees expressed concern that TC was “offloading regulations” onto operators.

   Quite a few interviewees expressed empathy for TCCA’s challenge to deliver adequate oversight of the sector with the resources at its disposal: “[we] see the strain on TC to be able to apply adequate level of resources to high-risk operators.” Others questioned the extent to which TC had policy clarity with respect to SMS. One highly experienced stakeholder advised that “TC needs to have a good grasp of what success is [regarding SMS], and see what it needs to do.” These are key considerations when discussing the future of SMS, particularly if the SMS requirement is extended to more organizations which, as noted earlier, is likely to place significant resource pressures on TC.

2. Complacency (or drift) is another long-term risk identified by key informants. Drift has the potential to set in in organizations as their SMS becomes increasingly mature. A slow “drift” can begin to occur over time due to other business priorities, in the form of doing just a little less of something (e.g., a decrease in training or proactive measures). Suggestions for ways to combat complacency included increased information-sharing within the industry. That said, this risk is accentuated when taken together with the perception of TC’s new approach to oversight as less present.

3. Another risk identified by the interviewees is the increasing difficulty in finding qualified people required to set up and maintain an effective SMS “Running out of people who can do this work – having a hard time filling positions”; “[there is] now a huge risk in Canada because there is a shortage of qualified people to work for these companies”. One case study company highlighted the importance of having experienced and skillful people run its SMS: “an awful lot of experience came in, people who knew what ‘good’ looked like. That helped significantly.”

4. Finally, at least five interviewees cited lack of communication and collaboration in the industry as an area of concern. For example, there is a view that risk assessment should be collaborative process between various stakeholders (operators, airports, TC) rather than individual. This can prevent one stakeholder offloading risk to another (e.g., airport to operator). One interviewee suggested that TC conduct a risk assessment on the future of SMS and keep the industry informed of potential changes. Another form of collaboration may come through improved industry data sharing and aggregation. For instance, an anonymized and centralized repository for safety data and information maybe enhance industry-wide risk management and improve trend analysis.

Conclusion

Stakeholders identified TC’s perceived retreat from its oversight responsibilities, complacency/drift in organizations regarding SMS (for example, slightly less training conducted over time), shortage of qualified people, and lack of collaboration within the industry as key long-term risks with respect to SMS. Whether actually true or not, the perception of TC oversight as increasingly “less present”, a key long term risk we’ve heard from several stakeholders, as well as perceived policy ambiguity regarding SMS, are important considerations when thinking about effective oversight of the civil aviation sector.

Appendix A: Transport Canada SMS Requirements

The CARs require that each SMS include the following components and elements^{Footnote 44}:

1. Safety Management Plan
   • 1.1. Safety Policy
   • 1.2. Non-Punitive Reporting Policy
   • 1.3. Roles, Responsibilities, and Employee Involvement
   • 1.4. Communication
   • 1.5. Safety Planning, Objectives, and Goals
   • 1.6. Performance Measurement
   • 1.7. Management Review
2. Documentation Management
   • 2.1. Identification and Maintenance of Applicable Regulations
   • 2.2. SMS Documentation
   • 2.3. Records Management
3. Safety Oversight
   • 3.1. Reactive Processes
   • 3.2. Proactive Processes
   • 3.3. Investigation and Analysis
   • 3.4. Risk Management
4. Training
   • 4.1. Training, Awareness, and Competence
5. Quality Assurance
   • 5.1. Operational Quality Assurance
6. Emergency Response Preparedness
   • 6.1. Emergency Preparedness and Response

Appendix B: References

“Aeronautics Act: Regulations Amending the Canadian Aviation Regulations (Parts 1, IV, V, and VII)”. Canadian Gazette Part II, vol. 139, no. 12, 2005, p. 1413-1456.

“Aeronautics Act: Regulations Amending the Canadian Aviation Regulations (Parts I, III, V, and VIII)”. Canadian Gazette Part II, vol. 141, no. 26, 2007, p. 2660-2684.

Auditor General of Canada. “Oversight of Air Transportation Safety.” Report of the Auditor General of Canada. 2008.

Auditor General of Canada. “Oversight of Civil Aviation.” Report of the Auditor General of Canada. 2012.

Bottani, E. et al. “Safety management systems: Performance differences between adopters and non-adopters.” Safety Science, vol. 47, 2009, p. 155-162.

David-Cooper, Rene. Implementation Safety Management Systems in Aviation: The Legal Impact on Canadian Commercial Air Carriers (Master's Thesis). McGill University, 2014.

Government of Canada. Canadian Aviation Regulation: Aeronautics Act, SOR 96-433, 1996. https://laws-lois.justice.gc.ca/eng/regulations/SOR-96-433/FullText.html, retrieved January 4, 2019.

Gunningham, Neil. Compliance, Enforcement, and Regulatory Excellence (Paper prepared for the Penn Program on Regulation’s Best-in-Class Regulator Initiative), 2015, p. 4.

Hodges, Christopher and Ruth Steinholtz. “How to Learn and Improve Performance: An Open Culture without Blame.” Ethical Business Practice and Regulation: A Behavioural and Values-Based Approach to Compliance and Enforcement. Hart Publishing, 2017, p. 84-101.

International Air Transport Association (IATA). Safety Report 2017, 2018.

International Civil Aviation Organization (ICAO). Safety Management Manual (Doc 9859), 2013.

Lercel, D. et al. Aviation Safety Management Systems Return on Investment Study. Center for Aviation Safety Research, Saint Louis University, 2011.

Thomas, Matthew J. W. A systematic review of the effectiveness of safety management systems (ATSB Transport Safety Report Cross-modal Research Investigation, XR-2011-002), 2012.

Transport Canada. Introduction to Safety Management Systems (TP 13739), 2001.

Transport Canada. Guidance on Safety Management Systems Development (Advisory Circular 107-001 Issue 01), 2008.

Transport Canada. Improving Canada’s Civil Aviation Safety Program: An Action Plan to April 2013, 2012.

Transport Canada. SMS: Where are we now? Where are we going? What have we learned? (Briefing to Canadian Aviation Safety Officer Partnership). 2014.

Transport Canada. Surveillance Policy (Civil Aviation Direction SUR-008 Issue 02), 2012.

Transport Canada. Surveillance Procedures (Staff Instruction SUR-001 Issue 05), 2013.

Transport Canada. Civil Aviation. https://tc.canada.ca/en/aviation, retrieved June 21, 2017.

Transport Canada. Process Inspections (Online Training), 2018.

Transport Canada. Surveillance Procedures (Staff Instruction SUR-001 Issue 07), 2018.

Transport Canada Civil Aviation. Civil Aviation Surveillance Program Evaluation and Update Project: Project Charter, 2016.

Transportation Safety Board (TSB). Statistical Summary: Air occurrences in 2017, 2018. http://www.tsb.gc.ca/eng/stats/aviation/2017/ssea-ssao-2017.asp, retrieved January 18, 2019.

Yeun, Richard. The Impact of Safety Management Systems on Safety Performance: Commercial Aviation Operations (Doctoral Thesis). Griffith University, 2015.

Yeun, Richard et al. “Aviation safety management systems.” World Review of Intermodal Transportation Research, vol. 5, no. 2, 2014, p. 168-196.

Appendix C: List of Recommendations

Appendix D: List of Findings

Appendix E: Management Action Plan

This management Action Plan addresses the recommendations of the evaluation report with proposed actions detailed with forecasted completion dates and convening authorities.

Transport Canada Civil Aviation (TCCA) Description of the Current State of SMS

TCCA has committed to undertake a policy review of SMS, to identify issues and opportunities, and to propose an approach to consulting with stakeholders on the application of SMS in the Canadian civil aviation sectors. There will also be an examination of ongoing pressure of the implementation of SMS within the sector.

In addition, TCCA is focused on modernizing the approach to surveillance through the delivery of the Surveillance 2.0 initiative, leaning processes to facilitate the implementation of SMS.

In addition to Surveillance 2.0 and as a member of state of the International Civil Aviation Organization (ICAO), Canada is required to implement a State Safety Program (SSP) and the use of data is a stepping stone in meeting those requirements. This strengthens the department’s ability to look comprehensively and formally at overlapping concerns within an organization, mitigate risk, increase safety, and strengthen enforcement efforts.

Furthermore, the Strategic Safety Risk Assessment methodology (SSRA) is working collaboratively with industry to identify, assess and rank significant safety issues in the Canadian Aviation System. In the event that risk management practices in SMS organizations were determined to be ineffective and represented a significant risk to the aviation system, collaborative work with internal and external subject matter experts would be undertaken to identify mitigation options and put forward safety solutions for prioritization and implementation.

When combined with the Management Action Plan, these initiatives will allow for the continuous improvement of Safety Management Systems in Canada; maintain and improve safety and sustainability in the aviation system; and uphold Canadian safety values around the world.

It is important for Canada to continue to examine the application of SMS in civil aviation in order to assert Canadian leadership in the international for and promote Canadian aviation industries and their economic competitiveness.

Recommendation #1

To realize the full benefits of SMS, TC should explore ways with the civil aviation enterprises to improve their root cause analysis capacity.

TCCA Response	TCCA partially agrees with this recommendation but notes the level of difficulty in gauging sufficient capacity in an effective and appropriate manner to drive improvement. This area will be explored with industry as recommended.
Proposed Action	An SMS policy review is underway for organizations currently regulated under SMS. It will evaluate the existing challenges and opportunities and assist TCCA in its determination of next steps. The capacity issue would be considered as part of broader industry consultations in Winter 2020. TCCA will reach out to the aviation industry associations through the Safety Collaboration Forum to cooperate on addressing issues that may be hampering the effectiveness of investigations and the avoidance of resource intensive investigations. Collaboration will include relaying the issues and recommendations to industry associations via their annual meetings and communication platforms; examining issues such as the need for more training; exploring mentorship opportunities within the aviation community; peer assistance; and delivering awareness campaigns about the importance of properly assessing risk. The outcome of the review could lead to a developed plan that takes into full consideration any limitations; constraints; and issues identified around scalability.
Forecasted Completion Dates	SMS Discussion Paper for Fall 2019, to be used for consultation with industry in Winter 2020. Develop white paper with options in Spring 2020 and plan for way forward.
OPI	AART AARB

Recommendation #2

TC should determine the extent to which organizations’ risk assignment practices are appropriate and take the necessary steps to mitigate if it detects a pervasive issue.

TCCA Response	TCCA agrees with this recommendation.
Proposed Action	TCCA is implementing Canada’s State Safety Program (SSP) as per Annex 19 Requirements of the International Civil Aviation Organization (ICAO). A sub-project of Canada’s SSP implementation project is the “SMS for Enterprises” initiative. The purpose of the initiative is to develop the methodology, guidelines and guidance material needed for applicable enterprises to set Safety Performance Indicators (SPI’s) and for TCCA to effectively monitor safety performance aligned with ICAO Annex 19. The scope of the project includes three components and one sub-component: guidance for industry; guidance for collaboration; and guidance for TCCA staff. The sub-component will identify gaps between Amendment 1 of Annex 19 and Canada’s current regulatory regime. Completion of the project is anticipated in June 2021. Furthermore, the Strategic Safety Risk Assessment methodology (SSRA) is working collaboratively with industry to identify, assess and rank significant safety issues. If risk management practices are identified as a gap or safety issue, then the safety issue would be included in the SSRA risk register and assessed to determine the level of risk that this issue represents in the Canadian aviation system. In the event that risk management practices in SMS organizations were determined to be ineffective and represented a significant risk to the aviation system, collaborative work with internal and external subject matter experts would be undertaken to identify mitigation options and put forward safety solutions for prioritization and implementation.
Forecasted Completion Dates	November 2019 for SSP 1st Stage of Implementation SSRA work underway - update to be provided at January 2020 Safety Forum
OPI	AART AARB

Recommendation #3

In order to be able to conduct quantitative analysis of SMS’ impact on aviation safety, Transport Canada should identify its information needs and develop and execute a data strategy to address those needs.

TCCA Response	TCCA partially agrees with this recommendation. As stated in the report, the lack of quantitative data demonstrating the impacts of SMS is a globally experienced challenge. As stated in the report, “This is primarily due to the infrequent occurrence of compromised safety, and the problems associated with the ‘fine art of measuring nothing.’”
Proposed Action	Recognizing the challenges with evaluating information that is not easily quantifiable, TCCA will consult with like-minded Civil Aviation authorities for solutions and best practices to address this recommendation. TCCA will also complete its business intelligence project - the Civil Aviation Oversight Application Rationalization (CAOAR) - currently underway that will lead to further support the recommendation. The CAOAR business intelligence project is a broader TC Transformation Initiative and identifies business applications across the organization to determine which applications should be kept, replaced or consolidated with the goal of achieving improvements in business operations. The desired end state will consolidate IT tools, expedite access to data; maximize resource utilization; and facilitate information sharing. This analysis will facilitate the structuring of big data for further analysis. The initiative will align to Government of Canada Digital Standards, the IM/IT Strategic Plan; and the Official Languages Act. As part of the State Safety Program (SSP) implementation, TCCA will implement safety performance measures to track the effectiveness of the Aviation Safety Program. Once implemented, TCCA will be in a better position to quantitatively track the evolution of aviation safety.
Forecasted Completion Date	Work underway; Consultation (next 12 months); CAOAR project will take approximately five years to implement (March 2024). SSP safety performance measures will be developed over the next two years.
OPI	AART AFM

Recommendation #4

TC should build capacity for continuous improvement in industry by encouraging innovative approaches to safety management that go beyond regulations while still meeting minimum expectations and safety standards.

TCCA Response	TCCA agrees with this recommendation.
Proposed Action	TCCA is working to foster innovative approaches to safety management through existing exemption authorities under the Aeronautics Act. Furthermore, there are new exemption powers to support innovation in the Budget Implementation Act and the Update on 2019 Annual Regulatory Modernization Bill. TCCA is working to lay the foundation for a safety collaboration framework to encourage new approaches to safety management that go beyond regulations. Canadian stakeholders envision a hybrid between some of the already established models such as InfoShare and Collaborative Safety Teams (e.g. Brazil and the United States). The aim is to create a data-driven Canadian solution that promotes government-industry partnerships and reduce safety risks. Two safety forums have been held in 2018 and 2019 with the objective of promoting effective safety education, collaboration, and development. These initial forums have provided an opportunity to engage with industry partners to openly discuss and identify some of the foundational parameters necessary to encourage innovative approaches to safety management and enable a culture of proactive action. These parameters include the need for confidentiality; collective discretion; and a cooperative approach. Any approach adopted by TCCA will need to include these parameters to ensure an open exchange of information, to build trust and to share experiences in a safe and secure environment with the aim of building a safer industry and strengthening our aviation programs. The proposed action involves a two-phased approach in collaboration with industry: Phase 1 - a feasibility study for consultation, framework, engagement and development (2-3 months). Phase 2 – a proof of concept involving governing documents, coordination, technical specifications, and the development of a long term strategic plan (2-3 years). The Safety Collaboration Forums are conceptually associated with activities of the SSRA Framework and will be an integral part of the SSRA Safety Risk Management cycle currently in development. TCCA also will look to ICAO literature for guidance under SMS framework (3.3) ‘Continuous Improvement’ and incorporate the strategies into the safety collaboration initiatives.
Forecasted Completion Date	Ongoing project Next Safety Forum scheduled for January 2020. Proposed action Phase 1 Dec. 2021, Phase 2, Dec. 2022. Project maturity January 2023.
OPI	AART

Recommendation #5

TC should ensure that updated training addresses the issues raised in this report, that is, it is consistent across regions, occurs in a timely fashion, and is relevant to assessing SMS in practice. Inspectors need a shared understanding of SMS principles and applications, and this knowledge should be refreshed regularly.

TCCA Response	TCCA agrees with this recommendation.
Proposed Action	TCCA is currently conducting Phase 1A of the SMS policy review (completion Fall 2019) which focuses on current SMS practices, while also looking backward at the initial SMS implementation in 2005. The review (to be extended over a multi-year period) will identify issues and opportunities, propose an approach to consulting with stakeholders on the application of SMS in the Canadian civil aviation sectors, and conduct consultations on the current state. Phase 1B (stakeholder engagement) is planned following the Federal election. The project charter can be found in RDIMS #14957482. The review will reflect past training issues identified. Avenues that may be explored as part of the SMS application include: Exploring potential for SMS training from outside organizations; Enhancing current SMS training within TC with more practical knowledge; Addressing challenges with consistency and improving inspector skillset across all regions; and Exploring possible SMS training delivery to industry and a mentorship program (similar to an Australian Civil Aviation Safety Authority initiative currently underway).*Note: Asterisk (*) Undefined as to who will deliver training (outside association, TC, etc.). Once this review is complete, recommendations will be made on potential approaches. End note In the short term, TCCA will look at leveraging new technology that is available through the TC digital transformation initiative. For example, Microsoft Teams is a platform where inspectors could quickly gain a shared understanding of issues and information can be refreshed in a timely manner. The platform could assist with best practices and building mentoring relationships between inspectors. * Undefined as to who will deliver training (outside association, TC, etc.). Once this review is complete, recommendations will be made on potential approaches.
Forecasted Completion Date	Review software platforms to address this recommendation – Summer 2019. Develop SMS Discussion Paper for Fall 2019, to be used for consultation with industry in Winter 2019/20. Develop white paper with options in Spring 2020 and plan for way forward.
OPI	AARB AART MITT

Recommendation #6

TC should engage with industry and TC inspectors to explore what level of collaboration in risk assessment and data-sharing is appropriate. TC could determine data needs for monitoring/improving aviation safety and assess the feasibility of more open data-sharing, which would be particularly relevant for smaller organizations, who may not generate sufficient data as individual entities and may therefore greatly benefit from aggregated data for trend analysis.

TCCA Response	TCCA agrees with this recommendation.
Proposed Action	TCCA continues to work on the Canadian Aviation Safety Collaboration Forum as a means of data-sharing and improving/monitoring aviation safety. Work is underway for the SSRA program. In 2018, TCCA completed an internal SSRA proof of concept exercise that included consultations with each TCCA Branch and Region. This exercise led to the identification of an initial list of significant safety risks and the establishment of TCCA’s top high risk occurrence categories. At the Collaboration Safety Forum held in Montreal in January 2019, the SSRA initiative was introduced to industry. Since then, several meetings have taken place with industry and external stakeholders who are progressively involved in the SSRA process, development, and risk identification, as well as discussions on data sharing possibilities. Industry will be contributing to the 2020-21 SSRA risk identification and assessment cycle, and results will be presented at the 2020 Safety Collaboration Forum. SSRA has been recently identified as a use case for the TC Data Lake initiative (a TC wide data management initiative part of TC Data modernization activities), which will further improve integration of data analysis in 2020-21. Work is underway to develop a data sharing strategy for civil aviation to facilitate data sharing between industry and TC in 2019. TCCA will explore enhanced information sharing with partner authorities around the world to assist in the identification of common risks and proactively working to address these risks.
Forecasted Completion Dates	Safety Forum scheduled for January 2020. Work underway on data sharing and SSRA , update to be presented at January 2020 Safety forum.
OPI	AART AARB

Appendix F: Survey Questions

1. Approximately how many employees within your organization have responsibilities defined under the Canadian Aviation Regulation (e.g. Pilot, Aircraft Maintenance Engineer, Cabin Safety Personnel, Person Responsible for Maintenance, etc.?). Employees with multiple responsibilities need to be counted only once.
   1. 1-10
   2. 11-50
   3. 51+
2. Has your organization implemented SMS?
   1. Yes
   2. No
   3. Partially
3. What has prevented the organization from introducing SMS? Please identify the main reasons.
   • SMS is not required for our business
   • SMS regulation does not apply to the organization
   • Insufficient time and resources
   • Financial burden
   • No perceived SMS benefits
   • Would not know how to start
   • Other
4. What year did your organization begin the implementation of SMS?
   1. Enter year (YYYY)
   2. Don’t know
5. Did your organization implement SMS
   • Voluntarily
   • Because of regulatory requirements
   • Both of the above
6. If your organization voluntarily decided to implement SMS (despite not being required to do so under regulations), indicate the main reasons for the decision. Please check all that apply.
   • Increase access to market
   • Insurance reasons
   • Contractual obligations
   • International requirements
   • Safety benefits
   • Association encouragement
   • Transport Canada promotion
   • Ease of transition from existing Quality Management systems
   • More effective and efficient operations
   • Establish continuous improvement practices
   • Positive impact on safety culture
   • Allocate resources based on better knowledge of risks
   • Competitive advantage
   • Align with industry trend
   • Interactions with partners or third party
   • Other
7. Identify which elements of SMS were already in place in your organization before SMS regulatory requirements came into effect. Please check all that apply.
   • Safety policy
   • Non Punitive Reporting Policy
   • Roles, Responsibilities & Employee Involvement
   • Communications
   • Safety Planning, Objectives and Goals
   • Performance Measurement
   • Management Review
   • Identification and Maintenance of Applicable Regulations
   • SMS Documentation
   • Records Management
   • Reactive Processes
   • Proactive Processes
   • Investigation and Analysis
   • Risk Management
   • Training, Awareness and Competence
   • Operational Quality Assurance
   • Emergency Preparedness and Response
8. How many times has the SMS in your organization been evaluated?
   1. Transport Canada Program Validation Inspection (PVI)
      • Never
      • 1
      • 2
      • 3
      • 4
      • 5
      • More than 5
      • Don’t know / Can’t remember
   2. Transport Canada Assessment
      • Never
      • 1
      • 2
      • 3
      • 4
      • 5
      • More than 5
      • Don’t know / Can’t remember
   3. Internal audits of SMS by your organization
      • Never
      • 1
      • 2
      • 3
      • 4
      • 5
      • More than 5
      • Don’t know / Can’t remember
   4. SMS audits by an external party
      • Never
      • 1
      • 2
      • 3
      • 4
      • 5
      • More than 5
      • Don’t know / Can’t remember
9. Do SMS quality assurance processes extend to flight operations?
   1. Yes
   2. No
   3. Not applicable to my organization
   4. Don’t know
10. From whom and what type of guidance did you receive on the implementation of SMS? Please check only those that apply to your organization.
    1. Documentation / Guide
       1. Transport Canada
       2. Peer Organizations
       3. Industry Associations
       4. External Consultants
       5. International Associations
    2. Expert advice
       1. Transport Canada
       2. Peer Organizations
       3. Industry Associations
       4. External Consultants
       5. International Associations
    3. Conferences, symposium
       1. Transport Canada
       2. Peer Organizations
       3. Industry Associations
       4. External Consultants
       5. International Associations
    4. Training
       1. Transport Canada
       2. Peer Organizations
       3. Industry Associations
       4. External Consultants
       5. International Associations
11. Please select the top three most useful sources of guidance you received.

    blank space	Organizations	Type of Guidance
    1	Transport Canada Peer Organizations Industry Associations External Consultants International Associations	Documentation / Guide Expert Conferences, symposium General meeting Training
    2	Transport Canada Peer Organizations Industry Associations External Consultants International Associations	Documentation / Guide Expert Conferences, symposium General meeting Training
    3	Transport Canada Peer Organizations Industry Associations External Consultants International Associations	Documentation / Guide Expert Conferences, symposium General meeting Training

12. Identify up to five (out of 17) elements of SMS which have been the most challenging to implement in your organization.
    • Safety policy
    • Non-punitive Reporting Policy
    • Roles, Responsibilities & Employee Involvement
    • Communications
    • Safety Planning, Objectives and Goals
    • Performance Measurement
    • Management Review
    • Identification and Maintenance of Applicable Regulations
    • SMS Documentation
    • Records Management
    • Reactive Processes
    • Proactive Processes
    • Investigation and Analysis
    • Risk Management
    • Training, Awareness and Competence
    • Operational Quality Assurance
    • Emergency Preparedness and Response
13. In which of the following areas did your organization have difficulty when implementing SMS? Please check all that apply.
    1. Business
       • Ability to implement organizational changes
       • Long-term planning and decision making
    2. Communications
       • Internal communications
       • Relationship / partnership with TC
       • Relationship / partnership with industry associations
       • Relationship / partnership with peer organizations
    3. Financial
       • Costs associated with implementation
       • Costs associated with maintaining SMS
    4. Employee
       • Capacity to implement
       • Recruitment for roles and responsibilities related to safety
       • Employees’ resistance to change
14. Overall, how important is SMS in enabling your organization to address emerging safety risks?
    • 0 – Do not use
    • 1
    • 2
    • 3
    • 4
    • 5 – Rely extensively on SMS
15. Identify up to five elements which you consider the most important to achieve the objective of continuous improvements within your organization.
    • Safety policy
    • Non-punitive Reporting Policy
    • Roles, Responsibilities & Employee Involvement
    • Communications
    • Safety Planning, Objectives and Goals
    • Performance Measurement
    • Management Review
    • Identification and Maintenance of Applicable Regulations
    • SMS Documentation
    • Records Management
    • Reactive Processes
    • Proactive Processes
    • Investigation and Analysis
    • Risk Management
    • Training, Awareness and Competence
    • Operational Quality Assurance
    • Emergency Preparedness and Response
16. How does your organization manage and analyze safety data?
    1. Manual process (spreadsheets, etc.)
    2. Safety Intelligence Software
    3. Custom built IT solution
    4. Contract out to a provider
    5. Other
17. Over the past three years, which type of chances has triggered a revision of your organization’s risk profile? Please check all that apply.
    • A round of hiring new staff
    • Changes to regulations
    • Adoption of new equipment / technology
    • Corrective / preventive actions
    • Internal audit / inspection results
    • External audit / inspection results
    • Hazard / accident / incidents occurrences
    • Performance measurement analysis
    • Revision of the environment scan
    • Safety issues reported
    • Quality Assurance reviews
    • Changes in airline routes and / or destinations
    • Changes to enterprise facilities
    • Changes in internal organization key personnel
    • Changes to civil aviation industry peers, partners or contractors
    • Company mergers
    • Reducing existing enterprise services
    • Other
    • Not applicable to my organization
18. Has your organization seen benefits other than safety after implementing SMS? Please check all that apply.
    1. Business
       • Better reputation among consumers and in media
       • Better stability
       • Ability to implement any organizational changes
       • Better long-term planning and decision making
    2. Communications
       • Improved internal communications
       • Improved relationship / partnership with TC
       • Improved relationship / partnership with industry associations
       • Improved relationship / partnership with peer organizations
    3. Financial
       • Improved revenue stream and less overhead costs
       • Reduced cost of lost / damaged equipment
       • Reduced medical costs
       • Lower insurance premiums
    4. Employee
       • Better health and safety in the workplace
       • Improvement in recruitment
       • Reduced turnover
       • Reduced absenteeism

Appendix G: Interview Questions

Interview Guide for External Stakeholders

1. Could you briefly describe your role within your organization, specifically with respect to managing safety?
2. In your opinion, what features make an SMS effective?
3. Since the implementation of SMS, have you observed any changes in your safety record? For example, number of incidents, number of repeated issues, etc.
   1. If so, how did your SMS contribute to this shift?
   2. Do you collect data to support this?
4. The concept of ‘safety culture’ is a key part of SMS. Has SMS influenced the culture of your organization? Can you provide some examples?
   1. Has the adoption of a safety culture stretched to partner organizations, suppliers, or third-parties?
5. Does your organization share knowledge, experience, or risk management techniques with other organizations? Are there any key benefits to this type of information sharing?
6. Can you describe the role of your department in the internal process for investigating SMS reports?
   1. Do the SMS quality assurance processes extend to your flight operations? If so, have there been any particular benefits to this extension?
7. In your opinion, what are some of the strengths and weaknesses of TC’s SMS oversight and inspection regime?
   1. Is your SMS usually inspected internally or by TC?
8. Have you seen any benefits from the implementation of SMS in areas of your organization that aren’t regulated by TC?
   1. Has the implementation of an SMS resulted in any impacts other than on safety?
   2. Is SMS a consideration in yearly planning? If so, how?
9. Can you describe the level of effort required to implement your SMS?
   1. Was the guidance material developed by TC useful in the implementation of the SMS?

Interview Guide for TC Staff

• 1) Please briefly describe your role within TCCA.
  • - How does your work relate to SMS?
  • - What was your role, if any, during the implementation process?
• 2) In your opinion, after 10 years of SMS regulatory implementation, has the adoption of an SMS by civil aviation organizations increased safety in the organizations?
• 3) In your view, has SMS influenced safety culture within regulated entities? If so, how?
  • - Could you provide specific examples?
  • - Do you think organizations have maintained a reactive approach or shifted toward a proactive approach to safety?
  • - Have you noticed any changes (in safety culture, practices, awareness, etc.) in non-SMS-regulated entities due to the safety measures/practices of regulated entities?
• 4) How does TC determine whether the SMS regulatory regime is effective?
  • - What are the tools/processes at TC’s disposal?
  • - Based on your knowledge, have there been any changes in safety levels industry-wide?
  • - Data?
• 5) In your view, has SMS influenced the relationship and/or level of collaboration between regulated entities and TCCA? If so, how?
  • - What about interactions between various regulated entities?
  • - Are you aware of any SMS related information-sharing among regulated entities?
  • - Have you noticed any changes (in safety culture, practices, awareness, etc.) in non-SMS-regulated entities due to collaboration with regulated entities?
• 6) In your opinion, what are some of the strengths and weaknesses of TC’s SMS oversight and inspection regime?
  • - Is TC working on any updates or additions to SMS or the management of SMS? If so, what brought about the need for this change?
• 7) Beyond safety, what impacts (positive or negative) have occurred as a result of SMS implementation?
  • - In your opinion, what is the value-added of SMS for organizations?
• 8) Were there any lessons learned from the SMS implementation process? What can we take away from this implementation experience?
  • - Could you provide specific examples?
  • - Did financial cost play a role in organizations’ implementation of SMS?
  • - How did organization size (small, medium, large) affect SMS implementation?
  • - Did you notice any differences in SMS project plans for different organizations?
  • - In your opinion, how effective was TC’s guidance (materials, other) in supporting SMS implementation amongst regulated entities?
    • Was it created and made available in time for organizations to use? How was it shared?
    • How frequently was/is guidance material updated? Are organizations made aware of updated materials?
    • Was TC guidance applicable to diverse organizational contexts? Was it sufficient?
• 9) What, if any, long-term risks should the civil aviation sector (including TCCA) be aware of regarding SMS?

Appendix H: List of Acronyms

AMO

Air Maintenance Organization

CARs

Canadian Aviation Regulations

EAS

Evaluation and Advisory Services

IATA

International Air Transport Association

ICAO

International Civil Aviation Organization

IOSA

IATA Operational Safety Audit

PI

Process Inspection

PVI

Program Validation Inspection

RIAS

Regulatory Impact Analysis Statement

QA

Quality Assurance

QAP

Quality Assurance Program

SMS

Safety Management Systems

TC

Transport Canada

TCCA

Transport Canada Civil Aviation

TSB

Transportation Safety Board