Follow-Up Audit of Aviation Security

June 2017

Table of contents

• Executive Summary
  • Introduction
  • Audit Objectives & Scope
  • Conclusions
  • Statement of Conformance
• 1. Introduction
  • 1.1. Context
  • 1.2. Background
  • 1.3. Audit Objective, Scope, Criteria and Approach
    • 1.3.1. Audit Objective
    • 1.3.2. Audit Scope
    • 1.3.3. Audit Criteria
    • 1.3.4. Audit Approach
  • 1.4. Report Structure
• 2. 2011 Audit of Aviation Security Regulatory Oversight Recommendations
  • 2.1. Governance
    • 2.1.1. Leadership and Functional Direction
    • 2.1.2. Managing Program Performance and Quality
  • 2.2. Risk Management
  • 2.3. Controls
• 3. Conclusions
• 4. Management Response

Executive Summary

Introduction

Transport Canada’s Audit and Advisory Services conducted an audit of Transport Canada’s Aviation Security Regulatory Oversight in 2011. The purpose was to provide reasonable assurance that the governance, risk management and control processes in place within Aviation Security are effective and achieving the objectives of the Aviation Security Regulatory Framework, Aviation Security Oversight, and Air Cargo Security programs. The audit findings resulted in 13 recommendations for which Aviation Security was responsible for developing and implementing management action plans (MAPs). Aviation Security has reported to TMX and the Departmental Audit Committee (DAC) that all of the recommendations have been fully implemented.

Transport Canada’s (TC) Risk Based Audit Plan (RBAP) 2016-2019 includes a Follow-up Audit of Aviation Security including Air Cargo Security with the objective of providing assurance that the recommendations from the 2011 Internal Audit of Aviation Security Regulatory Oversight have been fully implemented and are working as intended. Initially, it was planned that the follow-up audit would also examine the design and delivery of the Air Cargo Security Program which was not in place at the time of the original audit. However, in April 2017, after carrying out work related to the follow-up portion of the audit, Internal Audit determined with program management’s agreement that it would be best to separate the work into two audits. This approach will provide the time needed to complete a comprehensive audit of the Air Cargo Security program.

Audit Objectives & Scope

The objective of this follow-up audit was to provide assurance that the recommendations from the 2011 internal audit have been fully implemented and are working as intended. The scope of the audit is national.

Conclusions

At the time of the 2011 audit, Aviation Security was undergoing significant changes such as formalizing strategic program direction, implementing new risk assessment methodologies, updating and improving its oversight activities, and designing a new Air Cargo Security regime. The Program has come a long way since then with substantial improvements in the areas of governance, risk management and controls.

Our follow-up examination confirms that Aviation Security has taken action to implement the recommendations from the 2011 Internal Audit of Aviation Security Regulatory Oversight. We have concluded that 12 of the 13 recommendations are fully complete and one recommendation is partially complete and requires action. We have also identified some improvement opportunities we expect management to address as part of their continuous improvement efforts.

Statement of Conformance

This Audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of an external assessment of Internal Audit’s Quality Assurance and Improvement Program.

Dave Leach (CIA, MPA) Director, Audit and Advisory Services

Martin Rubenstein (CPA, CIA, CFE) Chief Audit and Evaluation Executive

1. Introduction

1.1. Context

Transport Canada’s (TC) Risk Based Audit Plan (RBAP) 2016-2019 includes a Follow-up Audit of Aviation Security including Air Cargo Security with the objective of providing assurance that the recommendations from the 2011 Internal Audit of Aviation Security Regulatory Oversight have been fully implemented and are working as intended. Initially, it was planned that the follow-up audit would also examine the design and delivery of the Air Cargo Security Program which was not in place at the time of the original audit. However, in April 2017, after carrying out work related to the follow-up portion of the audit, Internal Audit determined with program management’s agreement that it would be best to separate the work into two audits. This approach will provide the time needed to complete a comprehensive audit of Air Cargo Security.

The purpose of the original audit completed in 2011 was to provide reasonable assurance that the governance, risk management and control processes in place within Aviation Security are effective and achieving the objectives of the following program activities:

• Aviation Security Regulatory Framework;
• Aviation Security Oversight; and
• Initial planning related to Air Cargo Security (limited to review of the steps being taken to develop an Air Cargo Security program).

The audit findings resulted in 13 recommendations for which Aviation Security was responsible for developing and implementing management action plans (MAPs). Aviation Security has reported to TMX and DAC that all of the recommendations have been fully implemented.

1.2. Background

The Aviation Security Program helps to safeguard the integrity and security of Canada’s aviation transportation system. It helps to assure that Canada’s commitment to meeting the International Civil Aviation Organization (ICAO) standards are upheld. The Aviation Security Program is responsible for:

• security at Canadian airports (Class 1, 2 and 3 and other) including industry stakeholders along the perimeter of the restricted area (Primary security line partners);
• security of air carriers (passenger, express and cargo-only carriers) both foreign and domestic operating in Canada;
• Air Cargo Secure Supply Chain (businesses that handle secure cargo from shipper all the way through to the air carrier); and
• overseeing Canadian Air Transport Security Authority (CATSA).

The Aviation Security Program develops, administers and oversees the policies, regulations and standards to support the secure conduct of aviation activities in a manner harmonized with international standards. The Program is risked-based and fosters security within the aviation transportation system and provides security oversight of the aviation industry while ensuring that Canada complies with international standards (i.e. MOU, Mutual Recognition agreements).

Located at Headquarters (HQ), the Aviation Security Directorate is within the Safety and Security Group managed by the Assistant Deputy Minister, Safety and Security. This Directorate is led by the Director General (DG) Aviation Security and the Program Accountable Executive for the national program. In the regions, Aviation Security is managed by four Regional Directors (RDs), Transportation Security and one Regional Director, Aviation Security (Ontario Region) who each report to their respective Regional Directors General, who in turn report directly to the Deputy Minister.

Regional offices are responsible for day-to-day oversight operations while the HQ Aviation Security Directorate provides the functional direction to guide those operations.

In HQ, the Aviation Security Directorate is comprised of the following four branches:

1. Operations: Delivers the national oversight program and includes the functional areas of:
   • Oversight Programs and Tools;
   • Regional Coordination;  
   • Quality Assurance/ Quality Control;
   • Risk Management;
   • Technology;
   • Passenger Protect Program; and
   • Incident Management.
2. Program Development: Responsible for the strategic development of new aviation security programs, including the air cargo program and aviation security management programs. It is also responsible for negotiating mutual recognition agreements respecting new aviation security programs in the economic interest of Canada.
3. Policy and Planning: Develops Aviation Security policy options and includes the functional areas of:
   • Policy Development;
   • Strategic Analysis (i.e. Risk Assessments);
   • Strategic Partnerships; and
   • Business Planning and Reporting.
4. Regulatory Affairs: Develops and updates Aviation Security regulations and legislation and includes the functional areas of:
   • National and Specialized Security Regulatory Instruments;
   • Program and Policy Development;
   • Issues and Incident Management; and
   • International Standards and Harmonization.

While the Aviation Security Program’s key roles and responsibilities are focused on regulation and oversight, its stakeholders are responsible for the daily work of screening passengers, baggage and cargo, as well as securing aerodromes and aircraft and the warehouses and vehicles that transport cargo. The primary regulated stakeholders include air carriers (foreign and domestic), aerodromes, Primary Security Line Partners, participants in the Air Cargo Security Program, and CATSA.

1.3. Audit Objective, Scope, Criteria and Approach

1.3.1. Audit Objective

The objective of this follow-up audit was to provide assurance that the recommendations from the 2011 internal audit have been fully implemented and are working as intended.

1.3.2. Audit Scope

The scope of the audit was national.

1.3.3. Audit Criteria

The 13 recommendations from the 2011 audit represent the criteria for the follow-up audit. For each recommendation, the audit team reviewed, assessed and confirmed if the recommendation and its associated MAP:

• has been implemented,
• is meeting the expected outcomes, and
• is addressing the original audit finding.

1.3.4. Audit Approach

The audit approach included interviews, document review, walk-throughs of Aviation Security-related systems, and regional site visits/teleconferences.

The audit team visited the Quebec region in November 2016 as a pilot to test the interview questions for adequacy and appropriateness. The audit team visited the Ontario region (Toronto and Ottawa International airports) in December 2016 and held teleconferences with other regions during the conduct phase.

1.4. Report Structure

The 2011 Internal Audit recommendations were grouped by the following categories:

• Governance
• Risk Management
• Controls

We stated each recommendation and management’s action plan as it originally appeared in the 2011 audit report. As well, we describe our expectations and our assessment of progress at the time of this follow-up audit. For the one recommendation assessed as partially completed, we have identified specific areas requiring further management action. For some of the other recommendations assessed as fully complete, we have identified opportunities for management to address as part of their continuous improvement efforts.

We used the following scale to assess the implementation of each recommendation:

Management has provided their response to the audit following the Conclusions section.

2. 2011 Audit of Aviation Security Regulatory Oversight Recommendations

2.1. Governance

2.1.1. Leadership and Functional Direction

Recommendations 1 and 2

1. The Assistant Deputy Minister, Safety and Security should ensure the following:
   Finalize the drafting of the National Civil Aviation Security Program (NCASP) with particular emphasis on the Aviation Security Oversight Framework component so as to provide staff and stakeholders with a clear and documented oversight philosophy, and lay the foundation for critical operational improvements and changes, including consistent implementation of national standard operating procedures.

   Aviation Security Management Action Plan:
   Finalize the draft National Civil Aviation Security Program (NCASP) with the following milestones and timelines:

   • - Complete the oversight philosophy component.
   • - Develop implementation and communication plan
   • - Seek cabinet support as required.
   • - Release document

2. The Assistant Deputy Minister, Safety and Security should ensure the following:
   Develop an overarching logic model to articulate the Program’s expected results chain (activities, outputs, immediate and intermediate outcomes) and define how TC contributes to the Program’s expected outcome, i.e., security of civil aviation. This will help ensure that all program elements, particularly those that are in development or undergoing change, are clearly and explicitly aligned to a common outcome and will facilitate performance tracking and reporting.

   Aviation Security Management Action Plan:
   Complete overarching logic model in accordance with the following milestones and timelines:

   • - Update current logic model to clearly articulate the results chain.
   • - Map how all program elements and change initiatives contribute to the program’s ultimate outcomes.
   • - Finalize and adopt logic model for all of the AvSec Directorate. Utilize the Program Management Office to ensure integration into project level planning and priorities.

Internal Audit expected

Aviation Security would have developed and implemented an NCASP, Oversight Philosophy and Logic Model to define the program’s functional direction and support national consistent, effective and efficient operations.

Internal Audit’s Assessment

These two recommendations are assessed as fully complete. 

Observations to Support Internal Audit’s Assessment

Aviation Security developed and implemented an Oversight Philosophy, Logic Model, NCASP and Performance Measurement Strategy. Each document provides for clear program governance and functional direction.

In 2011, Aviation Security developed and implemented the Aviation Security Oversight Philosophy that lays the foundation and direction for TC’s regulatory oversight function. It provides information on the evolution of the oversight program, its guiding principles for ongoing design and implementation, oversight categories (i.e. inspections, education, awareness and promotion actions and enforcement actions), resource recruitment and training, frequency of cyclical risk based inspections, and detailed roles and responsibilities including the requirement of national standards and guidelines as well as a Quality Assurance review program. The Aviation Security Policy and Planning branch is currently updating the Oversight Philosophy. 

Also in 2011, Aviation Security developed and implemented the Aviation Security Logic Model and in 2012, it was revised and formally adopted as part of the National Civil Aviation Security Program (NCASP). The Logic Model defined four key activity streams (direction setting, regulation; technological infrastructure, oversight and monitoring) and sub-activities, as well as the outputs, immediate outcomes, ultimate outcomes and the strategic outcome (A Secure Aviation Transportation System).

In 2014, the TC Evaluation unit assessed the Logic Model as part of its Evaluation of Aerodrome and Air Carrier Security Regulation and Oversight and recommended that Aviation Security develop its Logic Model into a Performance Measurement Strategy. In 2015, Aviation Security implemented a Performance Measurement Strategy that includes an updated Logic Model and Performance Measurement Strategy Framework. The Logic Model defines four revised key activities:

• Develop policy on aviation security;
• Develop regulatory and non-regulatory instruments on aviation security;
• Develop and implement oversight of regulated parties; and
• Provide strategic guidance and support program management.

It also provides detailed descriptions of sub-activities as well as outputs and outcomes of the activities. The Performance Measurement Strategy Framework includes performance indicators for each of the outputs of the four activities, as well as for the immediate and intermediate outcomes (that tie into the Program Measurement Framework (PMF)) and an ultimate outcome. For each performance indicator, the data sources, collection frequency and responsibility, target date, and value of indicator are identified.

As part of our follow-up audit, we asked the Evaluation team to assess the Logic Model and Performance Measurement Strategy and they determined that overall, the design is comprehensive and sufficiently addresses the past Evaluation recommendation.

In 2013, Aviation Security published the National Civil Aviation Security Program (NCASP) which provides a consolidated set of expectations for TC and stakeholders to guide their respective aviation security responsibilities. It includes a clear overview of the program such as strategic objectives, guiding principles, approaches to managing risk and sharing information, as well as roles and responsibilities. The NCASP should be reviewed and updated as necessary to reflect any program changes; specifically, the Logic Model must be replaced with the most recent version.  

Opportunities for further improvement

Aviation Security should update the NCASP to reflect program changes such as the most recent logic model.

2.1.2. Managing Program Performance and Quality

Recommendation 3

3. The Assistant Deputy Minister, Safety and Security should ensure the following:
   Develop a succession plan for the Director responsible for the implementation of the Operational Performance Framework (OPF) so as to ensure the successful completion and implementation of the OPF, a key component of the NCASP and critical to its successful implementation.

   Aviation Security Management Action Plan:
   One year extension of the incumbent Director will be sought. And implement a migration plan to ensure effective transition to a new Director which will include appropriate documentation and knowledge transfer.

Internal Audit expected

An effective succession plan for the Director Position would have resulted in the development and implementation of an OPF framework that includes tools and guidance to identify performance expectations of stakeholders, measure their level of performance, record the data, and report the results.

Internal Audit’s Assessment

This recommendation is assessed as fully complete.

Observations to Support Internal Audit’s Assessment

At the time of the original audit, Aviation Security was developing an OPF to identify performance expectations of stakeholders and measure the effectiveness of their practices. A succession plan for leadership of the OPF project was put in place, and much work was completed, but the project was canceled and the OPF framework as originally designed was not developed. However, an equally acceptable performance measurement approach in the form of a Compliance Assessment Tool (CAT) was developed for inspectors to use in conducting their inspection activities.

Each CAT consists of questions related to specific regulatory requirements to determine entity compliance. For the Airport Security Program (ASP) inspections, it also includes a measurement scale with criteria to measure the level of compliance or non-compliance. The measurement scale is the component of the CAT that meets the objective of the original OPF.

The ASPs are Security Management Systems (SeMS). Inspections of these systems represent a relatively small portion of the overall oversight inspection activities. The International Civil Aviation Organization (ICAO) requires security programs (or SeMS) for airport operators, primary security line partners
(PSLP)^{Footnote 1} and air carriers. Aviation Security is responsible for developing and implementing security program (SeMS) regulations as well as CATs and Standard Operating Procedures (SOPs) to oversee and enforce industry’s compliance to them.

The ASP was the first security program (or SeMS) to be introduced in the 2012 Canadian Aviation Security Regulations (CASR). Currently, the requirements are applicable to the 89 aerodromes that are designated for Canadian Air Transport Security Authority (CATSA) security screening and their PSLPs (approx. 361). Once security program regulations have been implemented for Air Carriers, additional CATs will be developed.

The compliance of the ASP regulatory requirements is determined by management-based and performance-based regulatory requirements as they focus on assessing the operators’ systems/ processes (i.e. establishing procedures for monitoring and correcting problems) and outcomes - what the regulatory objective is and how to measure it (e.g. an operator may be required to control access to a restricted area without being told what systems or controls to put in place).

Aviation Security is implementing the ASP regulations in two phases: Phase 1 consists of the ASP foundational regulatory requirements (i.e. security policy statement) while Phase 2 consists of more complex requirements (i.e. airport security risk assessments). For each ASP foundational requirement, Aviation Security developed and implemented CAT questions to assist inspectors in assessing airport/ PSLP operators’ compliance:

• Does the measure exist (yes or no)? (e.g. a security awareness program is documented)
• Has the measure been implemented? (e.g. security awareness sessions are delivered)
• Is the measure effective? (e.g. interviews with airport employees identify that they are familiar with their responsibilities with respect to aviation security and what to do in the event of an incident)

Inspectors use the measurement scales to measure performance in terms of implementation and effectiveness. The scales include five levels along with assessment criteria for each level:

• Compliance: Meets Regulatory Requirements or Exceeds Regulatory Requirements (to identify best practices)
• Non-compliance: Non-existent, Attention Required, or Opportunity for Improvement

There is guidance for inspectors on how to use the measurement scales and record the results of the assessments in the Transportation Security Inspection System (TSIS). Inspectors must record as much information as possible in order to assess, measure and justify whether or not the airport/ PSLP operators comply with each ASP requirement and to what degree.

As part of the 2016 Audit of the Oversight Practices of Safety and Security Management System (SMS/SeMS), Internal Audit reviewed the performance measurement approach (CAT) for ASPs (SeMS) and identified it was a good practice that the other TC programs could consider adopting. Also as part of that audit, the Evaluation unit assessed that the ASP CAT measurement scale was generally well designed and that it would allow Aviation Security to measure some key expected outcomes of its ASP requirements.

Opportunities for further improvement

There are no specific opportunities for further improvement.

Recommendation 4

4. The Assistant Deputy Minister, Safety and Security should ensure the following:
   Implement a Quality Assurance function that regularly monitors regional delivery of the Aviation Security Program so as to ensure appropriate oversight is carried out, is consistent across regions and supports continuous improvement.

   Aviation Security Management Action Plan:

   • Develop a new quality management program with implementation options based on availability of resources.
   • Roll out the quality control program incrementally as new SOPs come online.
   • Design a program wide quality management function.
   • Review and revise the quality control program as required.

Internal Audit expected

Aviation Security would have developed a Quality Management program including QC and QA programs. The QC program would consist of activities to review the quality and consistency of oversight inspections based on national standard operating procedures. The QA program would consist of an overarching process to assess processes in place to ensure that the program meets its objectives. 

Internal Audit’s Assessment

This recommendation is assessed as partially complete.

Observations to Support Internal Audit’s Assessment

Quality Control

Aviation Security focussed its initial efforts on developing a Quality Control (QC) program.

A Quality Review office was established in the Operations branch. The Quality Control (QC) program was developed but implementation was delayed due to the absence of Standard Operating Procedures (SOP) for several aspects of inspection and enforcement activities, thus limiting the review of national inspection consistency. When the QC program was implemented nationally in January 2015, face-to-face orientation sessions were held with regional managers responsible for conducting QC reviews. QC review procedures were provided to standardize the process that includes a thorough assessment of inspection findings and reports to ensure they are completed and filed in accordance with SOPs, Policy Directives and the TSIS User Guide. 

Overall, the design of the QC process is sound as it requires a managerial review of inspectors documented inspection results using a standard template and a review of QC results in the region and HQ. QC results are maintained in HQ, corrective action is taken for deficiencies and inspectors QC review results are reflected in employee performance appraisals.

There have been some implementation challenges such as managers’ varying assessment approaches and issues raised concerning inconsistencies in the level of detail required in inspection reports. A QC Working Group, including HQ and the regions, was implemented to discuss and find solutions to common issues. For example, they are considering plans to develop a standard report template to improve the quality and consistency of inspector reports and QC reviews. As well, HQ has plans for trend analysis once sufficient QC data is collected.

Despite these implementation challenges, our interviews with regional managers and inspectors found that they see value in the program. The QC reviews provide the opportunity to enforce the requirement of documenting detailed inspection findings rather than yes or no answers. Of note, the regions commented positively on the QC guidance provided by HQ.

The planned and actual number of QC reviews per region are recorded in the Aviation Security National Oversight Plan (NOP)^{Footnote 2}. Based on a priority list, regional managers conduct a QC review per quarter, on one inspection file (which could include over a hundred compliance questions) for each inspector reporting to them. This equates to a relatively small percent (approximately 3% for Q4 2016-17) of inspection files reviewed, so Aviation Security is assessing whether this is a sufficient sample.

In addition, it is worth noting that as part of the 2016 Audit of the Oversight Practices of Safety and Security Management System (SMS/SeMS), Internal Audit identified that QC/QA processes across the modes are still in an early stage of development and do not yet provide sufficient assurance on the quality of inspection files.

Quality Assurance

The development of a Quality Assurance (QA) program is underway but is taking longer to develop than originally anticipated. Draft foundational QA documents, such as a QA Framework and QA Management Review Process, remain works in progress as the QA program develops.  

A QA implementation plan proposed a 2015/16 to 2016/17 timeline for the completion of stage 1 (Domestic Oversight Program). However, the first step in this stage, an environmental scan, anticipated for completion within three months, is not complete and will not be for some time as it involves a comprehensive review of the entire Aviation Security Operations oversight program. Stage 2 completion was proposed for 2017/18; however, this timeline in turn will not be met.

Areas requiring action

The Quality Review branch should determine a sufficient sample size of inspection files to review and continue working towards performing analysis of the QC data to identify common issues and report on trends.

With regional input, the QC Working group should define what is considered sufficient evidence to support inspection findings. A report template with standards could be developed, to provide inspectors with guidance, ensure greater quality and consistency, and assist managers’ QC review.

The draft QA foundational documents, including implementation timelines should be reviewed, updated, finalized and communicated to present a clearer picture of the status and expected completion of the QA program.  

Recommendation 5

5. The Assistant Deputy Minister, Safety and Security should ensure the following:
   Finalize the Service Level Agreement between the Aviation Security Program and the Security Program Support Directorate to ensure clear understanding of roles and responsibilities in relation to functions such as training and development of inspectors. Given the pace of change occurring in the Aviation Security Program, this is critical to ensuring training and development needs of Aviation Security personnel are met.

   Aviation Security Management Action Plan:
   Signed March 2011.

Internal Audit expected

Aviation Security would have established a Service Level Agreement with the training organization and ensure that its development and training needs are met.

Internal Audit’s Assessment

This recommendation is assessed as fully complete.

Observations to Support Internal Audit’s Assessment

To close this recommendation, a Service Level Agreement for fiscal year 2011/12, providing training services and deliverables, was signed between the Aviation Security Education and Training Division and Aviation Security Operations.

Subsequently, the Multimodal Integrated Technical Training (MITT) organization was developed and designated the functional lead for the identification, design, development, delivery and evaluation of modal specific and multimodal technical training across all Safety and Security modes. Two documents, an Accountability Framework, which replaces Service Level Agreements, and a MITT Governance Framework, provide the overarching training regime including the governance structure, roles and responsibilities, processes and approaches.

The Aviation Security training requirements are defined in two Policy Directives: one provides the process for issuance of credentials including the application process, while the other sets out requirements to ensure a national approach of the AVSEC Learning Continuum (includes the general AVSEC stream and Air Cargo Security stream) that comprises of three phases, including the levels of authority and associated powers, training requirements and completion timelines:

• Phase 1: Assessment (open Learning File for new employees)
• Phase 2: Initial Mandatory Core learning (Levels 1 – 3, includes structured on-the-job training (SOJT) activities)
• Phase 3: Specialized Learning

Inspectors acquire the skills and competencies, under each level of authority, required for the job as they progress through the training. The policy also provides processes for mandatory recurrent training, refresher training and course exemptions.

The MITT Learning Management System (LMS) database is used to track and report training. It lists the individual training requirements for each Aviation Security employee organized by region. Aviation Security legacy training data was integrated in the LMS and MITT is continuing to gather and add historical data. Inspectors have access to their training records in LMS and register for courses through LMS which provides course content information and availability. LMS includes a user guide, course catalogue and course registration procedures. An assigned MITT Technical Training Specialist validates all learning requests through LMS to ensure the training is required and all pre-requisites are met.

Regional managers are responsible for ensuring that their inspectors receive the required training for their credentials (e.g. badge) and have access to their inspectors’ training records in LMS.

When an inspector applies for his/her credentials, the HQ Operations branch reviews and confirms with MITT that the inspector has completed all required training for the credentials.

In response to our request, MITT attempted to reconcile that all Aviation Security inspectors had the mandatory training required for their issued credentials. However, as found in the 2016 Follow-up Audit of Civil Aviation, training records and inspector credentials are maintained in two different databases. These two systems are not aligned and currently the only way to confirm that an inspector has the necessary training to support their credentials is to manually trace and reconcile the records in the two systems. Ideally Aviation Security should be able to confirm that inspectors meet the training requirements for their credentials on a real time basis.

Opportunities for further improvement

There are no specific opportunities for further improvement for Aviation Security. MITT processes will be further examined in a future planned audit examining the training of inspectors.

2.2. Risk Management

Recommendations 6, 8 and 9

6. The Assistant Deputy Minister, Safety and Security should ensure the following:
   Develop an overarching Integrated Risk Management Framework (IRM) for the Aviation Security Program that:

   • a) Specifies IRM objectives and expected results;
   • b) Identifies and communicates roles and responsibilities of all key players in the risk management regime, which in turn should form the basis for coordination and alignment on accountabilities, information sharing, and training;
   • c) Specifies risk tolerance in the form of broad, yet meaningful indicators of risk impact such that all key players working in the various parts of the risk management regime have a common understanding of what constitutes a high or severe risk, boundaries of risk acceptability and what constitutes reasonable responses to unacceptable levels of risk;
   • d) Provides, under the broad umbrella of objectives, approach and tolerance, links to specific risk assessment methods, which themselves should be reviewed to ensure their alignment to the broader framework.
   • e) Contains a common taxonomy of Aviation Security (AvSec) Risks and Threats that would be applicable to and useful for the sub-elements of the risk management regime, including:
     • strategic risk assessment and regulatory decision-making
     • ASA approval
     • inspection planning
     • corporate and business planning
   • f) Aligns to the TC Corporate Risk Profile and its risk management and monitoring strategy by outlining the means by which AvSec risk information will be escalated, as appropriate, to the corporate level, for consideration in the broader departmental risk oversight process.

   Aviation Security Management Action Plan:
   Create an overarching IRM Framework in accordance with the following milestones and timelines:

   • a-b) Develop IRM policy and governance framework document that outlines the objectives and expected results, and program level direction with respect to risk management. Roles and responsibilities of all key players will be included as well as linkages to various risk practices currently in place. Roll out will be aligned to available resources and internal reallocation.
   • c-d) Provide program-level risk management guidance documentation which outlines the program’s overarching approach to risk and tools for risk management.
   • e) Ensure definitions and language (risk taxonomy) are aligned with Department wide work on horizontal and vertical risk alignment when available.
   • f) Review and revise framework, risk tolerance and risk assessment methods as required to align to Corporate and Strategies and Integration documentation when available. Provide training and/or awareness, where required, for integrated risk management for all staff and management within the AvSec Directorate.

8. The Assistant Deputy Minister, Safety and Security should ensure the following:
   Strategic Risk Assessments are one of the key processes currently conducted by the Aviation Security Program. To optimize their value, a plan should be developed and implemented to ensure their regular periodic conduct. The plan should be flexible so as to be responsive to emerging or unanticipated situations.

   Aviation Security Management Action Plan:
   Create and implement a cyclical strategic risk assessment plan.

9. The Assistant Deputy Minister, Safety and Security should ensure the following:
   The Security Programs Support Directorate should implement its plan to improve its intelligence-related threat and risk assessment process by formalizing the ongoing management of threat and risk information. This will help ensure a more systemic and consistent approach to analyzing, sharing and escalating threat and risk information.

   Aviation Security Management Action Plan:
   Security Program Support (SPS) will formalize the management of threat and risk information through the use of the “Risk Audit Matrix” (RAM), in accordance with the following milestones and timelines:

   • - Develop flexible implementation plan, present to Security DGs and seek approval for process.
   • - Organize TRA workshop(s) to address information gaps.
   • - Train employees who will be responsible to contribute.
   • - Communicate matrix in awareness sessions within TC and within the intelligence community.

Internal Audit expected

Aviation Security would have implemented an overarching Integrated Risk Management Framework (IRMF), a processes to carry out regular strategic risk assessments and formalize the ongoing management of threat and risk information.

Internal Audit’s Assessment

These recommendations are assessed as fully complete.

Observations to Support Internal Audit’s Assessment

Aviation Security implemented a Strategic Risk Assessment Plan that identified the requirements of developing an Integrated Risk Management Framework (IRMF) and a cyclical strategic risk assessment (RA) process including tools such as a Risk Assessment Matrix (RAM) and an Aviation Security Risk Statement to communicate results.

Integrated Risk Management Framework (IRMF):

The IRMF was published in January 2014 and its design addresses the required elements from the recommendation. It provides the objectives and expected results, identifying that the development and adoption of consistent risk processes within a comprehensive framework helps ensure that risk is managed effectively, efficiently and coherently across Aviation Security. It includes eleven key principles that form the foundation for the framework and are based on the ISO Standard for Risk management: Principles and Guidelines (i.e. RM creates and protects value, RM is part of decision making). The framework identifies roles and responsibilities, risk information requirements and sources for the Aviation Security Directorate, regions, executives, and committees. The framework also provides a risk assessment process that describes the various risk types (strategic, operational, corporate) and a heat map to assess the severity of the risk. A decision-making process is also provided to assist managers in determining an acceptable level of risk tolerance. As well there is a risk escalation process describing the steps to escalate risks to the appropriate level of management for action. The framework also provides links to an Aviation Security Corporate Risk Profile and Risk Registry. A table provides a listing of all risks to be considered in the risk assessment process. An appendix provides the Aviation Security Taxonomy and Risk Lexicon links.

For continuous improvement purposes, the IRMF is supposed to be reviewed annually with a summary of changes to be documented in a control log. A more comprehensive review is to take place every three years to ensure the IRMF still meets Aviation Security’s needs. However, annual updates have not been performed and the three year review that should take place this year has yet to be scheduled.

Strategic Risk Assessment (RA) Process

The Aviation Security Policy branch was responsible for developing and implementing the strategic RA process with support from other groups such as the TC Security Intelligence Assessment Branch (SIAB)

A national strategic RA process was developed in the form of a two day annual RA workshop. The first RA workshop was held in September 2012 with subsequent workshops held in the fall of years 2013 to 2015. The Policy branch cancelled the 2016 workshop to review the process and identify opportunities for improvement (e.g. increase participation of front line workers with hands-on experience, categorize the large number of scenarios (e.g. by program, types of threat)). The next workshop is planned for fall 2017.

RA workshop participants, cleared at the Secret level, include representatives from TC Aviation Security and SIAB, as well as other government departments (OGDs) and industry stakeholders who together assess that year’s chosen scenarios (e.g. Royal Canadian Mounted Police (RCMP), Canadian Air Transport Security Authority (CATSA), Public Safety, Canada Border Services Agency (CBSA) Canadian Airports Council, National Airlines Council of Canada).

To involve OGDs and industry in the process is considered a good practice because it allows a comprehensive assessment of scenarios by allowing them an opportunity to provide input and feedback on security issues and measures. In advance of the workshop, SIAB consults with the security and intelligence community (e.g. Canadian Security Intelligence Service (CSIS), Communication Security Establishment (CSE)) in order to provide an accurate threat assessment for scenarios. In addition to the threat assessments they provide during the RA workshop process, SIAB communicates regularly with the Safety & Security modes through weekly briefings to directors and managers on threats to transportation systems and situational awareness of the geopolitical context. They also provide monthly briefings to the ADM, AADM, DGs, and directors on threats identified during the month that may influence the modal risk assessments.

Guidance documents and tools were developed in 2012 to assist in the strategic RA process (e.g., scenario development, workshop workbook). A database was also created to maintain all of the assessed scenarios.

Prior to the RA workshop, SIAB and the Policy branch are involved in the identification and development of the scenarios to be assessed at that year’s workshop. There is an increased effort to support the scenarios with current Intel, making the scenarios more realistic. These scenarios relate to:

• {ATIP REMOVED}

The other Aviation Security branches (i.e. Operations, Regulatory Affairs, Air Cargo) are also consulted to ensure that timely and appropriate scenarios are chosen for assessment and/or re-assessment.

Some years, SIAB organizes a threat outreach session with participants a day prior to the RA workshop where guest speakers (e.g. RCMP) discuss relevant current threats {ATIP REMOVED} During the outreach session and the workshop, SIAB briefs the participants on the overall threats to the aviation system.

{ATIP REMOVED}

In addition to the annual strategic RA workshops, Aviation Security performs ad hoc strategic RAs throughout the year based on events, threats and/or requests from stakeholders. These scenarios are also included in the database, and are continuously being reviewed by SIAB and updated according to new threat information.

The results from these annual workshops and ad hoc RAs help guide Aviation Security’s ongoing policy and program development. Further, the results of the RAs help generate a critical source of risk information that helps guide the development of the National Aviation Security Risk Context Statement (RCS). The RCS is a document that provides a high-level risk assessment of current aviation risks related to acts of unlawful interference in Canada, as well as a ranked list of the security risk scenarios assessed. 

Opportunities for further improvement

As defined in Aviation Security’s own process, the review of the IRMF should be conducted to determine if it needs updating.

Recommendation 7

7. The Assistant Deputy Minister, Safety and Security should ensure the following:
   Should stakeholders have the flexibility to use their own risk assessment methods as the basis for their Aviation Security Assessments (ASAs); in order to support comparability of processes and outcomes, the ADM of Safety and Security should ensure that the Aviation Security Program:

   • a) Requires that all inspectors reviewing the risk assessments of airports have a solid understanding of what is an acceptable risk assessment method to be used, in the event that the TC method on which they were trained is not applied. This may include developing common criteria to be in place for assessing the adequacy and appropriateness of stakeholders’ risk assessment methods.
   • b) Develops a common understanding of risk tolerance, against which judgments can be made about acceptable levels of risk.
   • c) Ensures that final approval of all risk assessments and ASAs be performed by a national committee to ensure consistency and comparability.

   Aviation Security Management Action Plan:

   • a) Provide training and guidance on acceptable risk assessment methodology for inspectors after Gazette publication of regulations.
   • b) Communicate to the directorate the risk tolerance, updating as required to ensure alignment to Corporate and Strategies and Integration documentation when available.
   • c) Create a National Committee, chaired by the Director of AvSec Operations and supported by the Regional Directors, with a mandate and terms of reference which will aid in the consistency and the comparability of process and outcomes of security assessments.

Internal Audit expected

At the time of the original audit, regulations requiring stakeholders to develop Airport Security Assessments (ASA) were being developed. Subsequently, the regulations were amended and required Class 1 airport operators to develop an Aviation Security Risk Assessment (ASRA) and a Strategic Airport Security Plan for approval by TC. The operators experienced difficulties in developing their own ASRAs and thus TC became involved in developing a standard risk assessment methodology and assigned regional inspectors to assist the operators.

Thus, it would be expected that Aviation Security developed and implemented a risk assessment methodology including tools and guidance for operators to develop ASRAs. Also, ASRA training would be delivered to the designated regional inspectors to provide them with guidance on the ASRA requirements as well as a common understanding of risk tolerance. It would also be expected that a TC HQ national committee would be implemented to approve the ASRAs to ensure quality and consistency.

Internal Audit’s Assessment

This recommendation is assessed as fully complete.

Observations to Support Internal Audit’s Assessment

The revised regulation requirements were introduced in Phase 2 of the Canadian Aviation Security Regulations (CASR) and required Class 1 airport operators to develop an Aviation Security Risk Assessment (ASRA) and a Strategic Airport Security Plan for approval by TC.

Aviation Security developed a detailed risk management framework providing a risk assessment methodology and a risk management tool, which guides users through the framework and allows for the documentation of assessment scores and rationale. Designated regional Aviation Security inspectors were assigned to assist operators in developing the ASRAs and were provided related training by the Multimodal Integrated Technical Training (MITT) organization. The Aviation Security Integrated Risk Management Framework (IRMF), provided a common basis and guidance on risk tolerance (see Recommendation 6).

All Class I airports developed their ASRAs and Strategic Airport Security Plans and airport authorities approved and submitted them to HQ on March 1, 2017. HQ is currently reviewing the submissions.

A National Approval Committee was implemented to review and provide final approval of all ASRAs and Strategic Airport Security Plans. The committee plans to meet May 9, 2017 to carry out this process.

Our interviews with regional managers and inspectors involved in the ASRA process found that the training provided in preparation for the ASRA development was adequate and that the methodology used to develop the ASRAs with the Class I airport operators was effective.

Opportunities for further improvement

There are no specific opportunities for further improvement.

2.3. Controls

Recommendation 10

10. The Assistant Deputy Minister, Safety and Security should ensure the following:
    Given the magnitude and complexity of change taking place within the Aviation Security Program, develop and implement an overarching change-management plan and integrated project management framework to:

    • a) support clear communication
    • b) ensure clear accountabilities
    • c) align projects and priorities to effectively manage interdependences
    • d) track, measure and report progress in this area

    Aviation Security Management Action Plan:
    Establish an AvSec Program Management Office to provide program management, in accordance with the following milestones and timelines:

    • a-d) Complete program management plan through the design to scale based on various costing options. Communicate design and options and seek feedback throughout process.
    • - Implement plan based on available funding.
    • - Transition to full integration and self-sustainable.
      The AvSec Program Management Office will provide control and risk management along with governance as indicated in the Audit.

Internal Audit expected

Aviation Security would have developed an integrated plan that includes processes for change management and project management. It would include accountabilities, priorities, and a means for tracking and reporting progress on projects and initiatives.

Internal Audit’s Assessment

This recommendation is assessed as fully complete.

Observations to Support Internal Audit’s Assessment

Aviation Security developed the 2012-2013 Aviation Security Integrated Management Framework that provides change management and project management planning information such as:

• types of project documentation required for consistency and transparency, including the associated templates (i.e. Work Breakdown Structure, Project Charter, Statement of Work, Work Plan, Implementation Plan, Communication Strategy);
• planning for and monitoring change;
• roles/ responsibilities, timelines and templates/ resources for each Aviation Security Integrated Management Framework activity organized by:
  • Planning (i.e. Environmental Scan, branch work plans, Integrated Business Plan)
  • Monitoring (i.e. yearly update of the Framework, monthly reports)
  • Reporting (Year-End Close Out Report)
  • Documenting (General Documenting (RDIMS))
  • Change Management (i.e. Determine planned projects involving significant change)

At the time of the audit in 2011, Aviation Security was undergoing massive change. Although this is no longer the case, it continues to use the framework including the change management and project management tools that had been developed. Aviation Security has initiated a review of the framework.

Aviation Security also developed a project management tracking spreadsheet but it was replaced in 2013-14 when the Departmental Integrated Planning and Reporting (IPR) process (now called National Program Integrated Plan) was adopted. Aviation Security uses it to monitor projects and priorities as demonstrated by their 2015-16 IPR Work Plan that lists the ongoing and departmental priority-related activities.

Opportunities for further improvement

There are no specific opportunities for further improvement.

Recommendation 11

11. The Assistant Deputy Minister, Safety and Security should ensure the following:
    Complete and implement as soon as practical, National Standard Operating Procedures for all aspects of TC’s inspection and enforcement activities, including entry and management of inspection data. Standardized operating procedures are essential to ensuring a nationally-consistent application of the inspection and enforcement activities. In the interest of timely dissemination of the standard operating procedures, the Aviation Security Program should roll out new procedures as they are developed, with a deployment plan and communication strategy.

    Aviation Security Management Action Plan:
    Develop the Standard Operating Procedures in accordance with the following milestones and timelines:

    • - Develop
    • - Test
    • - Implement

Internal Audit expected

Aviation Security would have defined, developed and implemented required national SOPs for all aspects of the inspection and enforcement activities, including inspection data entry and management. The SOPs would have the same format for ease of use. A process would be developed for the review, approval, deployment, and communication of SOPs.

A full listing of the SOPs would be created to track the SOPs and their status (current, outdated or obsolete). A process would be developed for reviewing, updating and communicating updated SOPs.

Internal Audit’s Assessment

This recommendation is assessed as fully complete.

Observations to Support Internal Audit’s Assessment

Aviation Security developed many SOPs in 2012 to align with the Canadian Aviation Security Regulations (CASR). SOPs were drafted by HQ and sent to the regions for comments before being finalized and deployed. The SOP format is consistent and clear (i.e. same font, headings, structure, instructions). For example, each ASP SOP identifies the ASP requirement, and what and how inspectors need to assess compliance (i.e. questions to ask, methodologies to test, documents to collect). As new SOPs are developed and older SOPS are updated they will follow this same format. Aviation Security is confident that SOPs exist for all current aspects of inspection and enforcement activities and our interviews with the regions did not indicate otherwise. However, there is no overall listing of all the required SOPs.

The SOPs are organized into chapters by inspection type (i.e. aerodrome security inspections) in the National Standard Operating Procedures (NSOP) Manual which resides in the Transportation Security Inspection System (TSIS) and accessible to inspectors. There are individual SOPs for each regulatory requirement. For example, there are 13 individual detailed SOPs for each of the Airport Security Program (ASP) requirements. The NSOP Manual also provides other information such as forms for each chapter as well as an outdated table of contents. A TSIS User Guide provides guidance on entry and management of inspection data helping to ensure national consistency on inspection reporting. Throughout the Guide there are references to the use of SOPs during the conduct of inspections.

As amendments or additional security measures have come into force since 2012, there may be some SOPs that require updating. There may be other older SOPs that also require updating. However, there is no formal process to review and update the SOPs and an SOP inventory listing does not exit, so the total number of SOPs and the status of each is not known. A Working Group, led by HQ with regional representatives, was recently implemented to update the SOPs and Compliance Assessment Tools (CATs) (see Recommendation #3). Although there is no work plan yet, the priority is to review the CAT questions first and then to review the SOPs. The Working Group is also involved in the mechanism in place for inspectors to provide feedback and improvement suggestions on the SOPs.

Air Cargo SOPs relating to the recent program changes were updated in 2016. This work will be reviewed as part of the upcoming Audit of Air Cargo Security.

HQ recently drafted SOPs for a new program with inspector assistance that will be sent to the Working Group for review before being finalized.

Opportunities for further improvement

A complete listing of required SOPs should be developed to identify the current status of each SOP (e.g., current, obsolete, needs to be revised/developed). A process for reviewing, approving, implementing, and communicating new and outdated SOPs should be developed.

Recommendation 12

12. The Assistant Deputy Minister, Safety and Security should ensure the following:
    The query and reporting functionality for the SEPIRS database should be finalized and rolled out to provide management with the ability to analyze inspection data that supports timely decision-making and functional oversight.

    Aviation Security Management Action Plan:
    Create the query and reporting function of SEPIRS in accordance with the following milestones and timelines:

    • - Analysis/Design and Development
    • - Deploy and Train

Internal Audit expected

Aviation Security would have deployed the query and reporting functionality of the system used to record and maintain national inspection results. Training and guidance would be provided for inspection data entry as well as searching and reporting capabilities.

Internal Audit’s Assessment

This recommendation is assessed as fully complete.

Observations to Support Internal Audit’s Assessment

At the time of the original audit, Aviation Security used the Security & Emergency Preparedness Incident Reporting System (SEPIRS) to enter the results of inspection activities. In mid-2011, the Transportation Security Information System (TSIS) – SEPIRS Search and Reporting Tool was implemented to improve system functionality. It allowed detailed ad-hoc queries and enabled trend analysis for inspections. Training sessions were delivered to regional inspectors.

Through the years, TSIS was further developed and replaced SEPIRS. Archived SEPIRS information remains available in TSIS. TSIS is a centralized repository, including all the tools necessary to search, carry out, record and report inspections. The TSIS User Guide (Jan. 2016), provides standardized guidance for each of the TSIS modules. As new TSIS features/modules are rolled out, New Features’ guides are developed and added into TSIS. Aviation Security plans to roll up all the feature guides into a new comprehensive guide in June 2017. The “Library” module provides additional guidance (i.e. SOPs, Compliance Assessment Tools (CATs), Policy Directives) to inspectors for conducting inspection activities. There are also two links providing email addresses for users to request technical assistance or submit improvement suggestions.

The “Inspection” module is a key module as it is used to create new inspections (i.e. Aerodrome Operator, Air Carrier), enter inspection findings, compile a report, and search inspection files (open, pending, closed, cancelled).

There are additional TSIS modules that include search options to retrieve specific information that may assist in inspection activities. For example, the “Company” module provides the profiles of the regulated entities. The “Exemption” module provides the regulation exemptions that entities are granted under special circumstances.

There are many ad hoc searches that may be performed on the modules in TSIS by HQ and regional Directors, Managers and Inspectors. The entire TSIS database is searchable through the new “Master Search Field”. Searches may be initiated with only one known field or several fields (i.e. location, year, entity). For example, a search for certain deficiencies may be performed for all airlines, by airline, by region, by year etc.

The “Planning, Assigning and Reporting (PAR)”module was implemented in April 2017 to centralize inspection-related reporting activities in TSIS and standardize and automate the inspection planning, assigning and reporting processes. It includes the inspection plans for each year by inspection team. Managers will plan, assign and track, in real time, their inspection activities each fiscal year. Virtual classroom training was provided to regional managers and inspectors in April 2017.

There is also a “Time and Reporting Module” (TARM) for inspectors to enter their time by activity, but it does not link to the type of inspection. As such, it is difficult to determine the time and resources required for each type of inspection. Aviation Security plans to enhance TARM so the data can be used for activity costing and inspection planning. As the recent Audit of Risk-Based Business Planning in Safety and Security identified, without activity costing data, it is not possible to properly define resource requirements or make informed decisions about how to allocate resources.

Reports can be generated from TSIS by HQ and regional employees. For example, TSIS information is extracted for quarterly reports presented to Safety and Security’s Management Board, such as the National Oversight Plan (NOP) and the Aviation Security Program Oversight Delivery Dashboard.

Improvements are continuously being made to TSIS. A Capital Project Proposal, contingent on funding, was recently created for the development of an additional 10 modules aimed at enhancing the search and reporting capabilities of TSIS. Also included in this proposal is moving TSIS from an application to a web-based platform to reduce duplication of effort by allowing inspectors to enter inspections in tablets on-site.

Our interviews with the regions found that managers and inspectors are generally pleased with the TSIS functionality finding it user friendly and a great improvement over the previous system.

Opportunities for further improvement

Aviation Security should continue its work to improve its ability to cost activities. This will also help support work already underway in Safety and Security to address a recommendation from the Audit of Risk-Based Business Planning to improve activity costing in Safety and Security.

Recommendation 13

13. The Assistant Deputy Minister, Safety and Security should ensure the following:
    An employee performance management process should be fully implemented on a priority basis. In addition: a) Core competencies should be developed for supervisors b) A national performance management mechanism should be established to ensure consistent and effective performance management.

    Aviation Security Management Action Plan:
    Develop core competencies for supervisors and enhance the national performance management mechanism in accordance to the following milestones and timelines:

    • a) Identify core competencies for the Security Management Team (TI-07, TI-08 and PM-06)
    • b) Deliver the new HR course(s) on Managing Employee Performance, for security supervisors. A post-course focus group(s) will be conducted to identify future specific needs.
      Further identify enhancements to the existing HR performance management system.

Internal Audit expected

Aviation Security would have implemented a national employee performance management process and more specifically, core competencies for managers/supervisors to ensure they have the skills necessary to carry out a national consistent program.

Internal Audit’s Assessment

This recommendation is assessed as fully complete.

Observations to Support Internal Audit’s Assessment

To address this recommendation, in 2012-13, Aviation Security developed core competencies for managers/ supervisors and implemented a national Employee Performance Management (EPM) process to standardize performance management.

In 2014, the Aviation Security EPM process was replaced by a new mandatory Public Service Performance Management (PSPM) process to be used by all public service employees. Similar to the process Aviation Security had been using, the new process is based on the calendar year (January 1^st to December 31^st), and includes a mid-year review and a year-end performance assessment. The PSPM Application is used by all public service employees to complete their PSPM agreements. There is PSPM guidance available on the TC website (myTC).

The PSPM agreement includes four core competencies that are used for all employees including supervisors/ managers and inspectors:

1. Demonstrating Integrity and Respect
2. Thinking things through
3. Working effectively with others
4. Showing initiative and being action oriented

The competencies are based on the Values and Ethics Code for the Public Sector and the key leadership competencies (expected behaviours) for public service employees. There are behavioural indicators for each of the competencies to assist employees and managers to report and assess the extent to which the behaviours associated with the core competencies are demonstrated.

The TC Human Resources (HR) Directorate does not receive the appraisals since the process is controlled centrally in the PSPM application. The HR Systems group monitors the number of completed appraisals and reports to TC’s executive committee. It reported a high completion rate of almost 97% for fiscal year 2015-16 for regional Transportation Security/Aviation Security.

Opportunities for further improvement

There are no specific opportunities for further improvement.

3. Conclusions

At the time of the 2011 audit, Aviation Security was undergoing significant changes such as formalizing strategic program direction, implementing new risk assessment methodologies, updating and improving its oversight activities, and designing a new Air Cargo Security regime. The Program has come a long way since then with substantial improvements in the areas of governance, risk management and controls.

Our follow-up examination confirms that Aviation Security has taken action to implement the recommendations from the 2011 Internal Audit of Aviation Security Regulatory Oversight. We have concluded that 12 of the 13 recommendations are fully complete and one recommendation is partially complete and requires action. We have also identified some improvement opportunities we expect management to address as part of their continuous improvement efforts.

4. Management Response

With respect to the assessment of ‘partially complete’ for recommendation 4, the Aviation Security Directorate commits to reviewing its current Quality Control sampling rate. The review will include an assessment of appropriate sample rates while considering managers' workloads and other relevant factors. (December 31, 2017)

Guidance with respect to inspections is currently provided to managers and inspectors through documentation available in TSIS. Guidance will continue to be reviewed, updated and communicated to improve the quality and consistency of evidence in support of inspection findings. In addition, the Quality Control working group will continue to examine solutions to improve the quality and consistency of inspector reports. (Ongoing)

Aviation Security will also review Quality Assurance foundational documents. Documentation will also be updated and communicated based on the findings of the review. (March 31, 2018)

With respect to the opportunities identified for further improvement, the Aviation Security Directorate commits to examining ways to address these areas as part of its continuous improvement efforts.