EXECUTIVE SUMMARY
Introduction
A key responsibility of Transport Canada (TC) is conducting transportation safety and security oversight through legislative, regulatory, surveillance and enforcement activities. The TC Safety and Security Group’s national oversight programs contribute to achieving the strategic outcome of “A Safe and Secure Transportation System”. Quality management is an integral component of an oversight program to help ensure consistent and effective delivery within and across regions.
Audit Objective and Scope
The Audit of Quality Assurance (QA) Practices for Oversight Programs was included in TC's 2014-15 Risk-Based Audit Plan (RBAP). At the outset, the audit objective was to assess the effectiveness of the QA activities within the various Safety and Security oversight programs. Based on consultations with senior management, Internal Audit expanded the scope of the audit to include an assessment of the design of the quality management system (QMS) for each of Safety and Security’s oversight programs. The scope of the audit included all quality management related activities at the Safety and Security Corporate, HQ and the Regional levels. The oversight programs from the Program Alignment Architecture (PAA) included in the audit were Civil Aviation, Marine Safety and Security, Rail Safety, Motor Vehicle Safety, and Aviation Security. The Transportation of Dangerous Goods oversight program was excluded since an internal audit had been recently completed that included a review of the design of its QMS.
Conclusions
The results of our audit indicate that the Department lacks an overarching QMS framework. Inconsistencies and gaps with respect to QMS design within and between the oversight programs impedes the Department’s ability to monitor and report on the effective delivery of its oversight programs. However, the many initiatives under Safety and Security Transformation 2020 will address the majority of these gaps. As well, Internal Audit has made several observations for Safety and Security’s consideration as it moves forward with its initiatives to address its changing environment. We are confident that these initiatives will support the Department’s efforts in managing its oversight programs.
STATEMENT OF CONFORMANCE
This audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of an external assessment of Internal Audit’s quality assurance and improvement program.
Signed
Dave Leach (CIA) Director, Audit and Advisory Services
Date
Signed
Martin Rubenstein (CIA, CPA, CFE), Chief Audit and Evaluation Executive
Date
1.0 INTRODUCTION
1.1 BACKGROUND
One of the ongoing challenges for Safety and Security is delivering its national oversight programs consistently within and across regions. Safety and Security has been continually looking at ways to improve delivery of its national programs including conducting its own Blueprint 2020 exercise to identify initiatives to modernize its oversight programs. It is within this context that an assessment of quality assurance (QA) of the oversight programs was included in Transport Canada’s (TC) Risk-Based Audit Plan (RBAP). At the outset, the audit objective was to assess the effectiveness of the QA activities within the various Safety and Security oversight programs. Based on consultations with senior management, Internal Audit expanded the scope of the audit to include an assessment of the overall “quality management system” of Safety and Security’s oversight regime.
Quality Management System (QMS)
A QMS is a management system used to define objectives and standards for quality and it directs and controls how an organization achieves these. Overall, QMS is a collection of inter-related policies, processes, and practices that focus on achieving an organization’s quality objectives.
QMS should apply to all components of an oversight program. Internal Audit defined Safety and Security’s oversight programs as including three basic components:
- creation of rules and regulations;
- oversight activities to promote and determine compliance with the rules and regulations; and
- enforcement activities to manage instances of non-compliance with the rules and regulations.
The following diagram depicts the phases of a QMS around the three basic components of an oversight program.
Figure 1: QMS Phases of an Oversight Program
Effective quality controls (QC) and QA are required at all levels involved in the delivery of a national oversight program to help ensure consistency. At TC, there are three levels involved in the delivery of the oversight programs:
- Safety and Security Corporate (hereafter referred to S&S Corporate) – provides overall Safety and Security corporate direction including objectives and policies.
- HQ Functional Authority for each modal oversight program (hereafter referred to as HQ) – provides modal oversight program-specific direction including guidance (e.g., procedures, training) based on S&S Corporate direction. Each oversight program should have the necessary controls in place to ensure that the regions deliver the national program as designed. These controls should also include a function to conduct periodic risk-based quality assurance reviews.
- Region – delivers the oversight programs based on requirements from HQ. Regions should also have the required controls to ensure they are delivering the program as designed.
The following diagram depicts these three levels as well as their responsibilities and the bi-lateral flow of information between them.
Figure 2: Levels of QMS within Transport Canada
1.2 AUDIT OBJECTIVES AND SCOPE
Given the majority of Safety and Security resources are dedicated to oversight activities and these resources are distributed across five regions, there is inherently greater risk of inconsistent program delivery. As such, our assessment focused on this component. The objective of the audit was therefore to assess the design of the QMS for each of Safety and Security’s oversight programs.
The scope of the audit included all quality management related activities within S&S Corporate, HQ, and the Regions. The oversight programs from the TC Program Alignment Architecture (PAA) included in the audit were Civil Aviation, Marine Safety and Security, Rail Safety, Motor Vehicle Safety, and Aviation Security. The Transportation of Dangerous Goods oversight program was excluded since an internal audit had been recently completed that included a review of the design of its QMS.
1.3 AUDIT APPROACH AND ASSESSMENT CRITERIA
During the planning phase, the audit team interviewed the Safety and Security ADM and the DGs and Directors of each oversight program, as well as Strategies and Programs Integration. We also conducted interviews with a broad sample of oversight program staff in the Ontario, Pacific, and Prairie and Northern regions (RDGs, RDs, Managers, Technical Team Leads, and Inspectors).
We concluded early in the planning phase of the audit that Safety and Security does not have an overarching framework defining QMS at the S&S Corporate level, nor does it provide a model that sets out expectations for QMS in the various oversight programs. However, Safety and Security has implemented the Transport Canada Directive on Safety and Security Oversight (DOSSO) (effective April 4, 2014) that includes elements of a QMS. Although, the purpose of the DOSSO is to provide organizational direction for the design, development and delivery of oversight programs across all modes and regions, it does not address all elements of a mature QMS framework. In addition, Safety and Security is implementing a transformation agenda (Safety and Security Transformation 2020) which will focus on various initiatives related to the safety and security of the transportation system. Many of these planned initiatives, to be completed over the next several years, will have a direct impact on various elements of a fully functioning QMS framework.
Due to the lack of a formal QMS framework and supporting guidance, Internal Audit conducted research to create a model of a mature and robust QMS framework relevant to TC’s oversight programs.
Defining the QMS Framework
Through our research, the audit team developed a QMS expected practice framework based on the International Standards Organization (ISO) 9001:2008 standard as well as extensive conversations with Safety & Security. ISO 9001 defines QMS fundamental elements and it is the standard used by many organizations to implement QMS. Since it represents QMS in a manufacturing environment, we tailored and streamlined its criteria (expected practices) and descriptions to reflect Safety and Security’s oversight program environment. We also developed examples of potential activities that would satisfy the requirements of these criteria.
We then grouped these criteria into the following six QMS phases:
- QMS Design and Development
- QMS Implementation (at the S&S Corporate level)
- Program Design and Development
- Program Delivery
- Program / Oversight Activity Continuous Improvement
- QMS Continuous Improvement
We also reviewed QMS-related documentation from the S&S Corporate, HQ, and Regional levels. This information, as well as the information gained from extensive consultations with HQ and regions, helped us to assess the design of each oversight program’s QMS against our benchmark QMS expected practice framework. The assessment included assigning a rating for each QMS criteria (i.e. rating of 1 if met, rating of 0.5 if partially met, or rating of 0 if not met). We shared the assessment results with each oversight program for validation. Additionally, we assessed the DOSSO against the framework to determine its alignment with requirements of QMS and we validated our assessment with the DOSSO team.
2.0 FINDINGS
Findings are grouped under two headings: Directive on Safety and Security Oversight and Safety and Security Transformation 2020 Initiatives. As previously described, Safety and Security has many initiatives underway to transform the way it manages its oversight programs. Since these initiatives are not yet completed, our observations highlight their status and certain gaps or challenges requiring attention or consideration. Furthermore, we have identified future audit work to provide assurance that the Department has a mature functioning QMS framework in place.
2.1 DIRECTIVE ON SAFETY AND SECURITY OVERSIGHT
The results of our assessment of the DOSSO and oversight programs against the QMS expected practice framework are summarized in Appendix A.
At the S&S Corporate level, DOSSO provides the necessary corporate direction for the design, development and delivery of oversight programs across all regions. While the DOSSO was not designed to be a QMS framework, it does provide many of the elements and sets expectations that can be used by the oversight programs when developing a QMS.
The development and implementation of DOSSO is on going and since the audit began, Safety and Security has been considering ways to evolve the DOSSO as a stepping-stone towards establishing a QMS framework.
Within the oversight programs, Civil Aviation and Rail Safety have the most defined and documented QMS with established QMS manuals and accompanying policies, procedures and staff instructions. Marine Safety and Security is currently taking steps to bolster their quality related activities into a more systematic QMS. Aviation Security has pockets of good practices around quality assurance that satisfy some of the requirements of a QMS but these practices differ greatly between the various program elements of Aviation Security. Motor Vehicle Safety is the program with the least structured QMS, which is understandable given its centralized operations and relatively small size compared to the other regionally delivered programs.
2.2 SAFETY AND SECURITY TRANSFORMATION 2020 INITIATIVES
Safety and Security has many initiatives underway that will address many of the changes required to have a fully functioning QMS framework. It is clear that Safety and Security recognizes the changes required and is in the process of taking action, including developing a comprehensive overarching plan to manage and report on the initiatives.
Presented below are key observations that Safety and Security should consider as it makes improvements to its existing QMS programs and practices. As well, we have highlighted specific areas where further audit work will be required to ascertain effectiveness of the stated improvement. Internal Audit will consider this audit work during the planning for the new three-year departmental internal audit plan (for fiscal years 2015-16, 2016-17 and 2017-18).
Project Management
Work is underway to develop an overarching project management plan to manage and report on the various Safety and Security initiatives. The majority of the initiatives are multi-modal in nature with some modal specific projects. To help support successful implementation of these numerous and broad initiatives, Safety and Security recognizes the need for an overarching and comprehensive project and change management “Master Plan” in order to:
- define clear objectives and accountabilities;
- align projects and priorities to manage interdependencies;
- enable tracking, measuring and reporting progress; and
- support clear communication.
As part of the implementation process, Internal Audit will review, on an ongoing basis, the framework and controls surrounding the improvement initiatives. This "real-time, continuous review" will provide senior management with the assurance that the framework is robust and meeting its objectives.
Civil Aviation’s QMS
Through the QMS design assessment, it appears that Civil Aviation has addressed most of the criteria required for a QMS through its Integrated Management System (IMS). As a Follow-up Audit of Civil Aviation Oversight was already included in the current three-year internal audit plan, we will expand the audit scope to include an assessment of the adequacy and effectiveness of Civil Aviation’s IMS to assess its operating effectiveness. This assessment will assist Safety and Security in considering QMS development and implementation in the other programs.
Standard Operating Procedures
As part of its DOSSO, Safety and Security is requiring the oversight programs to implement internal QA processes. Guidance and directions such as Standard Operating Procedures (SOPs) are a prerequisite for setting quality standards and assessing conformance. Currently, the SOPs of the various oversight programs have varying levels of quality, completion, and implementation. The Safety and Security Inspectorate must have clear and concise SOPs that are nationally consistent within oversight programs and meet a minimum standard across oversight programs with respect to format and level of detail.
We plan to carry out further audit work to assess the adequacy and effectiveness of the existing and planned revisions to SOPs. This assessment will assist Safety and Security in ensuring its programs have SOPs that support accurate and consistent delivery of the oversight programs. It will also assist in the development and implementation of effective QA processes.
Risk-Based Inspection/Audit Planning and Reporting
The Safety and Security oversight programs develop and carry out annual national inspectionFootnote1 plans to support the planning and execution of inspections as well as tracking progress and reporting performance against those plans. However, we did not observe evidence of reporting to S&S Corporate on the quality or relevance of inspections. It appears the emphasis is ensuring that the regions complete the quantity of inspections indicated in the plans. An effective risk-based inspection/audit planning and reporting process that integrates business needs with operational needs is required across all oversight programs. It enables prioritization of program activities and allocation of resources based on risk. It also provides the basis for monitoring and reporting progress to senior management and supports timely decision making. The planning process should include activities such as risk assessments and review of previous inspection results and of changes in the industry. The reporting process should include the status of progress in completing inspections based on the plans as well as assurance that quality standards are being met. Given the Department’s record of year-end surpluses and the fact that Safety and Security represents 60% of the Department’s human resources, the integration of risk-based operational planning with business planning is critical to sound budgeting and resource allocation decisions.
We plan to assess the adequacy and effectiveness of the risk-based inspection/audit planning and reporting approach across all oversight programs and we will identify best practices. The objective would also be to assess the type and utility of the information reported to Senior Management on progress against the inspection plans.
Safety Management Systems/Security Management Systems
Regulations require various stakeholders to implement either a Safety Management System (SMS) or Security Management System (SeMS). An SMS/SeMS is a set of management practices for systematically addressing safety/security risk within an organization, including the necessary corporate accountabilities, policies, and procedures. Through our consultations with Safety and Security, it appears that each oversight program currently has its own understanding and practices with respect to SMS/SeMS oversight and enforcement. A common understanding and approach of the requirements of SMS/SeMS regulations is necessary across and within all oversight programs to help ensure consistent program delivery. The S&S Transformation 2020 Project to develop a Transport Canada Strategic Framework for the Oversight of Safety Management Systems and Security Management Systems will provide the Department with a common narrative to internal and external stakeholders on why, how and where TC has chosen to use SMS/SeMS regulations to improve the safety and security of Canada’s transportation system. It will inform the design and delivery of TC’s SMS/SeMS related oversight activities, and it will guide decision-making around SMS/SeMS regulations for industry.
We plan on examining each of the oversight programs’ understanding and approaches related to oversight and enforcement activities for companies with SMS/SeMS in place. We will also determine whether the relationship between SMS/SeMS oversight activities and other risk control activities provides adequate oversight and optimization of resources. This will also include assessments by the Department's Evaluation Group on the effectiveness of SMS/SeMS activities.
Staffing and Recruitment
For the various oversight programs, TC generally recruits inspectors with extensive industry experience. We observed that this type of recruitment results in several challenges. Difficulties are often experienced in attracting industry personnel for various reasons such as wage gaps. Also, as these individuals usually transition to TC in the latter part of their careers, they have a relatively brief employment period and must be replaced. In order to minimize the challenges in recruiting and retaining inspectors, competencies should be reviewed and alternative approaches should be explored.
We plan to assess the efficiency and effectiveness of staffing and recruitment strategies. This would include examining the current Safety and Security model for hiring inspectors as well as identifying best practices for recruitment and retention.
Oversight Program Controls
HQ functional management is responsible for developing and implementing various controls (e.g., SOPs, training, etc) for their respective oversight programs to follow in order to obtain a level of assurance that the regions are delivering national programs as designed and in a consistent manner. It appears that these required controls for the various oversight programs are not always in place or not working as intended, resulting in the risk of inconsistent program delivery.
We plan to assess the adequacy and effectiveness of the oversight program controls to ensure the regions are delivering national programs as designed. This may include the review of the development, communication and use of SOPs, training, functional guidance, supervision, QC, and QA.
System Architecture
There are numerous information systems used by each of the oversight programs. Some of the challenges with the Safety and Security system architecture include inconsistent practices with respect to data entry and lack of alignment among systems resulting in duplication of work. Efficient system architecture allows for efficient entry and reporting of data that is accurate, complete, and useful. It also promotes sharing of data within and between oversight programs which helps address emerging risks.
We plan to review the system architecture in the oversight programs to determine if data is being collected accurately, completely and efficiently. As well, we plan to also identify opportunities to either consolidate systems or ensure systems communicate with one another, within and across oversight programs. In addition, we will determine if plans are in place for examining how the transition to government-wide Finance (SAP) and HR (PeopleSoft) systems may impact existing safety and security operating systems.
3.0 CONCLUSION
The results of our audit indicate that the Department lacks an overarching QMS framework. Inconsistencies and gaps with respect to QMS design within and between the oversight programs impedes the Department’s ability to monitor and report on the effective delivery of its oversight programs. However, the many initiatives under Safety and Security Transformation 2020 will address the majority of these gaps. As well, Internal Audit has made several observations for Safety and Security’s consideration as it moves forward with its initiatives to address its changing environment. We are confident that these initiatives will support the Department’s efforts in managing its oversight programs.
Appendix A –Assessment of DOSSO and the S&S Oversight Programs against the QMS Expected Practice Framework
| Expected Practice Framework | Assessment Results | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| QMS Phase | Expected Practice | Expected Practice Details | Baseline | DOSSO | Civil Aviation | Rail Safety | Marine Safety & Security | Aviation Security | Motor Vehicle Safety |
| 1. QMS Design and Development | 1.01 Management Commitment | Assign responsibility to senior management to develop and implement the QMS and its continual improvement, including the appointment of a senior management representative to oversee and report on the QMS performance and required improvements. | 1 | 0.5 | 1 | 1 | 0.5 | 0.5 | 0 |
| 1.02 Quality Policy | Develop, communicate and maintain a Quality Policy that defines the overall quality intentions and directions within the organization and the commitment to continuous improvement. | 1 | 1 | 0.5 | 1 | 1 | 0 | 0 | |
| 1.03 Quality Objectives | Develop, communicate and maintain specific and measurable quality objectives that are consistent with the Quality Policy. | 1 | 0 | 0.5 | 1 | 0.5 | 0.5 | 0 | |
| 1.04 QMS Manual | Develop, communicate and maintain a QMS Manual that clearly defines the QMS scope/exclusions, responsibilities and authorities, processes and interrelationships. | 1 | 0.5 | 1 | 1 | 0.5 | 0 | 0 | |
| 1.05 QMS Documents & Records | Develop, communicate and maintain procedures for the control and retention of QMS related documents (e.g. QMS Manual, policies, procedures, SOPs, etc.) and supporting activity records (e.g. quality control approvals and reviews, training records). | 1 | 1 | 1 | 1 | 1 | 0.5 | 0 | |
| 1.06 Stakeholder Focus | Consider stakeholder requirements in program design and delivery. | 1 | 1 | 1 | 0.5 | 0 | 0 | 0 | |
| 1.07 Internal Communication | Develop and implement processes to communicate QMS requirements, processes and performance throughout all levels of the organization. | 1 | 1 | 1 | 1 | 1 | 0 | 0 | |
| 2. QMS Implementation | "2.01 QMS Implementation Plan" | Develop and carry-out a project plan for the implementation of the QMS in accordance with the Quality Policy, Objectives and Manual. | 1 | 0 | 1 | 1 | 1 | 0 | 0 |
| 2.02 QMS Resources | Determine and provide the resources needed to implement and maintain the QMS and to continually improve its performance. | 1 | 0 | 1 | 1 | 1 | 0 | 0 | |
| 2.03 Employee Awareness | Ensure that personnel are aware of the relevance and importance of their activities and how they contribute to the achievement of the quality objectives. | 1 | 0 | 1 | 0.5 | 0.5 | 0 | 0 | |
| 3. Program Design and Develop | 3.01 Program Design and Development | Procedures Develop and implement procedures and controls over the design and development of new programs/oversight activities. | 1 | 0 | 1 | 0.5 | 0 | 0 | 0 |
| 3.02 Program Parameters | Determine the parameters within which the programs/oversight activities are to be developed (e.g. Transportation Regulations, MOUs, TBS Policies and Directives). | 1 | 1 | 1 | 0.5 | 0 | 0 | 0 | |
| 3.03 Provision of Operational Resources | Determine and provide the resources needed to deliver the programs/oversight activities. | 1 | 1 | 1 | 1 | 0 | 0 | 0 | |
| 3.04 Personnel Competencies | Determine personnel competencies necessary to deliver the programs/oversight activities. | 1 | 1 | 1 | 1 | 0.5 | 0.5 | 0 | |
| 3.05 Design and Development Review | Perform systematic reviews at suitable stages during design and development to ensure conformance with defined program parameters. | 1 | 0 | 1 | 1 | 0 | 0 | 0 | |
| 3.06 Design and Development Validation | Validate that the programs/oversight activities have been developed in accordance with the design and development process and stated requirements. | 1 | 0.5 | 1 | 1 | 0 | 0 | 0 | |
| 4. Program Delivery | 4.01 Guidance, Information Systems, Technology and Tools | Provide the guidance (SOPs), information systems, equipment, technology and tools necessary for program delivery. | 1 | 1 | 1 | 1 | 1 | 1 | 0.5 |
| 4.02 Resource Competency | Ensure that personnel have the required competencies (e.g. education, training, skills and experience) necessary for program delivery. | 1 | 1 | 1 | 1 | 0 | 0 | 0 | |
| 4.03 Provision of Training | Provide training to meet required competencies. | 1 | 1 | 1 | 1 | 1 | 0 | 0 | |
| 4.04 Supervision of Program/Oversight Activities | Implement appropriate supervisory activities necessary to ensure the delivery of programs/oversight activities in accordance with requirements (including EPMs). | 1 | 1 | 0.5 | 1 | 0 | 0 | 0 | |
| 4.05 Program Exemptions | Carry-out oversight activities in accordance with specific exemptions to rules or regulations. | 1 | 1 | 1 | 0.5 | 0 | 1 | 0 | |
| 4.06 Identification and Status | Identify each program/oversight activity to be conducted and track progress. | 1 | 1 | 1 | 1 | 0.5 | 1 | 0.5 | |
| 5. Program/ Oversight Activity Continuous Improvement | 5.01 Monitoring and Measurement | Continuously monitor and measure the program/oversight activity performance to ensure that delivery is consistent with stated requirements. | 1 | 1 | 1 | 1 | 1 | 0.5 | 0 |
| 5.02 Corrective Action | Take corrective action to eliminate the causes of non-compliance by inspectors in order to prevent recurrence. | 1 | 1 | 1 | 1 | 1 | 0.5 | 1 | |
| 5.03 Preventative Action | Implement appropriate controls to prevent non-compliance by inspectors before occurrence. | 1 | 1 | 0 | 1 | 0 | 1 | 0 | |
| 5.04 Employee Performance Management | Evaluate employee competencies against requirements and the effectiveness of training programs. | 1 | 1 | 1 | 0.5 | 0.5 | 0 | 0 | |
| 5.05 Controls over issuance of permits in error | Controls and processes to deal with licenses, certifications, and permits that have been issued and later found to not comply with stated requirements. | 1 | 0.5 | 1 | 1 | 0 | 1 | 0 | |
| 6. QMS Continuous Improvement | 6.01 Continuous Monitoring and Measurement | Continuously monitor and measure the performance of the QMS and its ability to achieve stated Quality Objectives. | 1 | 1 | 1 | 1 | 0.5 | 1 | 0 |
| 6.02 Periodic Review/ Audit by Senior Management | Initiate and oversee periodic reviews/audits on the effectiveness and suitability of the QMS and related quality activities. | 1 | 1 | 1 | 1 | 0.5 | 0 | 0 | |
| 6.03 Stakeholder Input | Incorporate stakeholder feedback on programs/oversight activities into reviews/audits and continuous improvement activities. | 1 | 0.5 | 1 | 1 | 0 | 0 | 0 | |
| 6.04 Continual Improvement | Continually improve the effectiveness of the QMS through the use of the Quality Policy, Quality Objectives, review/audit results, analysis of data, corrective and preventive actions, and management review. | 1 | 1 | 1 | 1 | 1 | 0 | 0 | |
| 6.05 Update and Integrity | Maintain the integrity of QMS when updates and changes are implemented. | 1 | 0 | 1 | 1 | 1 | 0 | 0 | |
| Total | 32 | 22.5 | 29.5 | 29 | 15 | 9 | 2 | ||
Key: 1 = Met, 0.5 = Partially met, 0 = Not met