Follow-up Audit of Civil Aviation

January 2016

• EXECUTIVE SUMMARY
  • Introduction
  • Audit Objective and Scope
  • Conclusions
  • Statement of Conformance
• 1. INTRODUCTION
  • 1.1 Context
  • 1.2 Background
  • 1.3 Audit Objective and Scope
  • 1.4 Criteria and Approach
  • 1.5 Report Structure
• 2. FINDINGS
  • 2.1. Training
  • 2.2. Safety Management Systems (SMS)
  • 2.3. Safety Issues
  • 2.4. Risk-Based Planning
  • 2.5. Quality control, Quality assurance
  • 2.6. Performance measurement
  • 2.7. Human Resources (NOTIP)
  • 2.8. Human Resources (Staffing)
  • 2.9. Database Systems
• 3. CONCLUSIONS
• 4. MANAGEMENT RESPONSE

EXECUTIVE SUMMARY

INTRODUCTION

Transport Canada's (TC) Risk Based Audit Plan (RBAP) 2014/2015 to 2016/2017 includes a Follow-up Audit of Civil Aviation Oversight with the objective of providing assurance that the recommendations from the Office of the Auditor General (OAG) audits of Civil Aviation in 2008 and 2012 and the Internal Audit (IA) review in 2011 have been fully implemented.

The purpose of each audit/review was as follows:

• 2008 OAG audit: examined how the Department had managed the implementation of Safety Management Systems (SMS) for large air carriers and associated aircraft maintenance organizations, which were the first aviation industry groups required to implement SMS.
• 2011 IA review: assessed risks relating to the management control framework (MCF) of the Civil Aviation Safety Program, including controls to support ongoing operations and ensure effective change management.
• 2012 OAG audit: determined whether TC has adequately managed the risks associated with overseeing its Civil Aviation Program.

There were 19 unique recommendations that Internal Audit has categorized under nine general themes:

1. Training,
2. Safety Management Systems (SMS),
3. Safety Issues,
4. Risk-Based Planning,
5. Quality Assurance/Quality Control (including Standard Operating Procedures),
6. Performance Measurement,
7. Human Resources NOTIP^{Footnote 1},
8. Human Resources (Staffing), and
9. Database Systems.

Of the 19 recommendations, Civil Aviation reports that all except one (the Database Systems recommendation was not originally due for completion until March 2016) have been fully implemented.

AUDIT OBJECTIVES & SCOPE

The objective of the audit was to confirm that, for each of the 19 recommendations, Civil Aviation's management action plan (MAP)

• has been implemented,
• is working as designed, and
• is addressing the original audit finding.

CONCLUSIONS

We found that Civil Aviation has implemented their management action plans for 17 of the 19 audit recommendations and those action plans appear to be working as designed. The two main areas requiring further work are as follows:

• Database Systems:
  • The action plan for an integrated solution to capture information relevant to the oversight of civil aviation safety was originally scheduled for completion in March 2016. This project is not on track and now has a new target completion date of March 2017.
• Training:
  • The OAG in 2012 recommended that Civil Aviation "should ensure all staff involved in inspections receives the training necessary to execute their responsibilities". In response, Civil Aviation has indicated that it had provided the necessary surveillance training to the majority of inspectors. We however are of the opinion that the recommendation is not referring to just surveillance training but rather all training requirements. {ATIP removed} Finally, we identified the need for training related to the non-surveillance work performed by inspectors

We have also highlighted in the report other specific areas requiring further attention and would expect Program management to address these as part of their continuous improvement efforts.

STATEMENT OF CONFORMANCE

This audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of an external assessment of Internal Audit's Quality Assurance and Improvement Program.

Signatures

1. INTRODUCTION

1.1. CONTEXT

Transport Canada's (TC) Risk Based Audit Plan (RBAP) 2014/2015 to 2016/2017 includes a Follow-up Audit of Civil Aviation with the objective of providing assurance that the recommendations from the Office of the Auditor General (OAG) audits of Civil Aviation in 2008 and 2012 and the Internal Audit (IA) review in 2011 have been fully implemented and are working as intended.

The purpose of each audit/review was as follows:

• 2008 OAG audit: examined how the Department had managed the implementation of Safety Management Systems (SMS) for large air carriers and associated aircraft maintenance organizations, which were the first aviation industry groups required to implement SMS.
• 2011 IA review: assessed risks relating to the management control framework (MCF) of the Civil Aviation Safety Program, including controls to support ongoing operations and ensure effective change management.
• 2012 OAG audit: determined whether TC has adequately managed the risks associated with overseeing its Civil Aviation Program.

Table A below presents the number of recommendations (24) for which Civil Aviation was responsible for developing and implementing management action plans (MAPs). As some of the recommendations from the 2008 OAG audit and the 2011 IA review were not fully implemented by the time of the 2012 OAG audit, similar recommendations were made by the OAG in 2012. To simplify tracking and reporting to the Departmental Audit Committee (DAC) previous recommendations (5) were grouped into the 2012 recommendations. As of June 2015, Civil Aviation has reported that 18 of the 19 total recommendations have been fully implemented.

Table A

Report Date	Auditor	Audit	Total # of recsFootnote 2	# of recs reported in similar rec	# of unique recs
2008	OAG	Chapter 3 - Oversight of Air Transportation Safety – Transport Canada	9	2	7
2011	IA	Aviation Safety Planning Review	5	3	2
2012	OAG	Chapter 5 - Audit of Oversight of Civil Aviation – Transport Canada	10	0	10
Total			24	5	19

The 2015/16 to 2017/18 Departmental RBAP includes other Safety and Security-related audits. In order to minimize the impact of these audits on Civil Aviation operations, the audit teams have been coordinating their efforts and closely collaborating to leverage information and analyses.

1.2. BACKGROUND

The Civil Aviation Program promotes the safety of the national air transportation system through its regulatory framework and oversight activities. As part of the regulatory framework, Civil Aviation develops policies, guidelines, regulations, standards, and educational materials to advance civil aviation in Canada. As part of the oversight activities, Civil Aviation verifies that the aviation industry complies with the regulatory framework through surveillance activities (i.e. assessments, validations, inspections, and enforcement) and service activities to the aviation industry (i.e. licenses, certifications).

The Civil Aviation Program is comprised of the Civil Aviation Directorate at Headquarters (HQ) and the Civil Aviation branches in the five regions.

The Civil Aviation Directorate is within the Safety and Security Group that is managed by the Assistant Deputy Minister, Safety and Security. This Directorate at HQ was previously led by one Director General (DG). A governance structure change, effective April 2, 2015, has resulted in a division of the responsibilities of the DG position into two DG positions. The two Directors General have been appointed with one to lead the Aviation Safety Regulatory Framework and the other to lead Aviation Safety Oversight. This decision was based on the need for additional executive focus on strategic direction and response to the rapidly changing global environment.

In the regions, Civil Aviation is managed by five Regional Directors (RDs), Civil Aviation, who report to their respective Regional Directors General, who in turn report directly to the Deputy Minister. The RDs also have a functional reporting relationship with the HQ DGs.

Regional offices and some HQ branches (e.g. National Aircraft Certification) are responsible for day-to-day oversight operations while the HQ Civil Aviation Directorate provides the functional direction to guide those operations.

The HQ Civil Aviation Directorate is comprised of seven branches:

• Civil Aviation Secretariat: Provides Aviation Safety Program information to stakeholders and the public.
• Management Services: Provides the directorate's administrative resources and coordinates the directorate's program and financial planning and the implementation of the Civil Aviation IMS.
• Medicine: Performs the medical assessments required for the certification of licensed aviation personnel.
• National Aircraft Certification: Sets technical standards for aeronautical products designed and operated in Canada and monitors their continuing airworthiness. This involves providing ongoing guidance to the aerospace industry.
• National Operations: Provides regulatory oversight of Canadian air navigation service providers and air carriers, including assessments, inspections, audits, and enforcement actions.
• Policy and Regulatory Services: Provides strategic advice on the development of aviation legislation and regulations. This branch is responsible for administering the Canadian Aviation Regulatory Advisory Council (CARAC).
• Standards: Develops and revises standards related to airworthiness, safety management systems, aircraft certification, aerodromes, and air navigation. This branch also delivers safety promotion and education.

It must also be recognized that the Civil Aviation groups in the Regions and HQ that deliver the program are not a homogeneous group. Rather they represent many different unique sections such as Flight Operations, Engineering, Cabin Safety/AOSH, Aerodrome, Licensing, and Airworthiness. We could not therefore just identify a sample of inspectors and managers to interview in the regions. Rather the challenge was to interview managers and inspectors in each of these areas in all regions to ensure we were getting responses that were not skewed to any one sector.

1.3. AUDIT OBJECTIVE AND SCOPE

The objective of this follow-up audit was to provide assurance that the recommendations from the OAG audits in 2008 and 2012 and the IA review in 2011 have been fully implemented.

As the subject matter of many of the recommendations overlaps we have categorized the 19 recommendations into 9 general themes:

1. Training,
2. Safety Management Systems (SMS),
3. Safety Issues,
4. Risk Based Planning,
5. Quality Assurance/Quality Control (including Standard Operating Procedures),
6. Performance Measurement,
7. Human Resources NOTIP^{Footnote 3},
8. Human Resources (Staffing), and
9. Database Systems.

Of the 19 recommendations, Civil Aviation reports that all except one (the Database Systems recommendation was not originally due for completion until March 2016) have been fully implemented. We reviewed each recommendation and its associated management action plan (MAP) to determine if the MAP

• has been implemented,
• is working as designed, and
• is addressing the original audit finding.

1.4. CRITERIA AND APPROACH

The 19 recommendations represent the criteria for this audit.

The audit conduct phase included document reviews, interviews, regional site visits, and file reviews.

The audit team first visited the Atlantic region as a pilot to test the interview questions and file review template for adequacy and appropriateness. Interviews and file reviews were then conducted in the remaining regions as well as in headquarters.

1.5. REPORT STRUCTURE

We have provided our findings based on the nine different themes. For each theme, we identified what we expected to find, what we did find, observations in support of our findings and, where appropriate, any areas requiring further attention.

This report is limited to commenting on the recommendations included in the two OAG reports and the one IA report.

Management's response to the audit is included at the end of the report.

2. FINDINGS

2.1. TRAINING

Internal Audit expected:
All Civil Aviation inspectors would have access to and have taken the mandatory training required to support their Consolidated Records of Authority.

Internal Audit's assessment:

{ATIP removed}

The control that ensures training requirements have been met prior to the issuance of an inspector's credentials relies on validation with the inspector's approving manager. A more robust control would be to require independent validation against training records.

{ATIP removed}

Observations to support Internal Audit's assessment: 
The 2008 OAG audit identified the need for Civil Aviation to develop a training strategy for Civil Aviation inspectors that aligns with the human resources plan being developed. It was expected that the training strategy would need to address the inspectorate's required competencies, current/recurrent training needs, and courses. The audit also recommended that Transport Canada ensure its inspectors meet their training requirements or that their authorities are restricted commensurate with their actual training levels.

A subsequent OAG audit in 2012 found that Civil Aviation had developed a training strategy that did identify the training requirements of the inspectorate. However the audit repeated the earlier recommendation that the inspectorate "receive the training necessary to execute their responsibilities in a timely manner."

As a result of the OAG's findings, Internal Audit focused its attention on ensuring that Civil Aviation could demonstrate that its inspectorate has the opportunity for timely and relevant training.

Simply put, Civil Aviation identifies all the activities that a given inspector may conduct; which is based on the various regulations that Civil Aviation oversees. In doing so, Civil Aviation has created a "Consolidated Record of Authority" for each inspector position. Each Consolidated Record of Authority identifies the activities that TC effectively authorizes the inspector to perform. In addition, for each inspector position, Civil Aviation has identified all of the mandatory training requirements for that position.

The Safety and Security Group has recently established a Multimodal Integrated Technical Training (MITT) branch. This branch is responsible for maintaining a record of all training completed by each inspector. The branch also provides training courses, including mandatory training.

The Regulatory Affairs division, of Civil Aviation's Policy and Regulatory Services (PRS) Branch, reviews and confirms with the inspector's manager that the inspector has completed all required training for the credentials they are applying for (e.g. badge, pass, letter, etc). However, based on our review, it would appear that a more effective control would be to have MITT validate that the inspector has completed the required training courses versus solely relying on the manager.

{ATIP removed} Ideally Civil Aviation should be able to confirm that inspectors meet the training requirements for their credentials on a real time basis.

Civil Aviation has defined two key categories of mandatory training for the inspectorate, foundational and specialized. Foundational training is training to acquire common knowledge and competencies for job performance. (e.g. Delegated Officer Initial Training). Certain positions require specialized training. Some of this specialized training includes an initial course as well as recurrent training after a specified period. While foundational training is being provided, {ATIP removed} mandatory specialized training courses and {ATIP removed} mandatory recurrent training courses, {ATIP removed} are in development. Of the {ATIP removed} mandatory training courses in development, one is being offered by a third- party provider in the interim.

{ATIP removed}

Finally, we noted that the recommendations and management action plans concerning training focused on the surveillance activities of inspectors. However, we found that surveillance activities represent only part of an inspector's responsibilities. An inspector has many other work requirements {ATIP removed}.

Areas requiring further attention: 
Civil Aviation should ensure that all mandatory training courses are developed and/or available to the inspectorate to meet the mandatory training requirements.

Civil Aviation should ensure it has the ability to verify in a timely manner that inspectors have met the mandatory training requirements for their Consolidated Records of Authority.

{ATIP removed}

Civil Aviation should define, develop, and provide training requirements for non surveillance work.

2.2. SAFETY MANAGEMENT SYSTEMS (SMS)

Internal Audit Expected:
Civil Aviation's SMS transition plan would include an assessment of the risks and mitigating actions, the expected costs, and the resource requirements for implementation.

Civil Aviation would establish a standard for the minimum planned surveillance level for SMS and non-SMS enterprises to monitor for compliance with the regulatory framework. Civil Aviation would also report on performance against the surveillance plans to senior management on a regular basis.

Internal Audit's Assessment:
Civil Aviation's SMS transition plan has been placed on hold.

Civil Aviation has established a standard for the minimum planned surveillance level for SMS and Non-SMS enterprises to monitor for compliance with the regulatory framework. Performance against surveillance plans is reported to senior management on a regular basis.

Observations to Support Internal Audit's Assessment: 
SMS has not been extended to the other sectors of the civil aviation industry based on an approved Risk Assessment.

Therefore, there are no detailed transition plans other than what was published on Transport Canada's website in 2004. Civil Aviation published an Advisory Circular 503-003 in 2008 which identifies the components and elements that an organization should have in place at the end of each phase.

Aviation enterprises are subject to planned surveillance over a five-year period. The planned surveillance intervals are consistently determined for each enterprise using the "Surveillance Planning" module in NASIMS (National Safety Information Management System). Specific risk related enterprise information is entered in each enterprise's Risk Profile page to determine the Risk Indicator Level and the Impact Value that ultimately determine the Surveillance Interval. The Surveillance Policy (CAD SUR-008) and National Surveillance Planning Standard (SI SUR-009) provide guidance.

2.3. SAFETY ISSUES

Internal Audit Expected:
Civil Aviation had analyzed and revised processes for addressing safety issues in order to address issues more quickly.

Internal Audit's Assessment: 
Civil Aviation has reviewed its rulemaking consultation process and has identified and implemented new ways of addressing safety issues. A new model and new tools were developed that provide improved responsiveness and increased efficiency and effectiveness. Data however was unavailable to confirm that issues are actually being addressed more quickly.

Observations to Support Internal Audit's Assessment:
The CARAC (Canadian Aviation Regulation Advisory Council) Modernization Project was launched in 2011 with the objective of creating a consultation model that better addresses safety issues, streamlines processes, and prioritizes regulatory initiatives in the early stages. The project engaged stakeholders and reviewed best practices from other regulatory departments and counterparts in other jurisdictions (i.e. Civil Aviation Safety Authority (Australia), Federal Aviation Administration (USA), European Aviation Safety Agency (Europe)).

The new model and tools received support from the CARAC Plenary in October 2013 and were implemented by December 2013.

The new tools include:

• Preliminary Issue & Consultation Assessment (PICA) (input to CARAC): assists in the early evaluation of an issue, determines the need for a Focus Group and recommends an appropriate consultation stream;
• Revised Notice of Proposed Amendment (NPA) (output from CARAC): communicates regulatory change proposals to regulations and standards to the aviation community and provides summaries of the assessments, evaluations and recommendations leading to the proposal; and
• Enhanced CARAC Activity Reporting System: now provides information on all CARAC Activities and is a communication tool with stakeholders for relevant information throughout the rulemaking process (more transparent and meaningful consultation).

While a new process has been implemented, it is too soon to determine if it is actually shortening the time to respond. Civil Aviation has seen time efficiencies from the new process in a few cases. {ATIP removed}

This year Civil Aviation is undertaking an initiative to establish an annual Strategic Safety Risk Analysis (SSRA) process to evaluate and monitor system-level safety issues in Canada's civil aviation system. The analysis will be conducted by subject matter experts from Civil Aviation headquarters, regions and industry and will inform planning and priority-setting processes, policy development, and rule making. Civil Aviation has identified an initial list of priority system-level safety issues which helped inform the 2016/17 Business Planning process. The top four safety issues were prioritized by senior executives (NCAMX) and will be considered in developing work plans for the next fiscal year.

2.4. RISK-BASED PLANNING

Internal Audit Expected:
Civil Aviation has put in place its national risk indicator program to assess the risk level and minimum surveillance requirements for each enterprise.

Civil Aviation has identified resource requirements to carry out the surveillance activities.

Civil Aviation is monitoring deviations from the surveillance plans.

Internal Audit's Assessment: 
Civil Aviation has implemented the National Aviation Safety Information Management System (NASIMS) which contains the risk level and minimum surveillance requirements for each enterprise. A process has also been implemented to identify resource requirements to carry out surveillance activities.

Civil Aviation has implemented a process that ensures deviations from the approved inspection plans are reviewed and approved by senior management.

Observations to Support Internal Audit's Assessment: 
Civil Aviation has implemented the NASIMS which is a database for gathering information required for developing a risk profile for each enterprise.

The Risk Profile is the combination of the:

• Risk Indicator Level: likelihood that the enterprise is managing risks to an acceptable level of safety and compliance: and
• Impact Value: determination of the impact of the enterprise on the aviation transportation system and public confidence.

NASIMS uses the Risk Indicator Level and the Impact Value to calculate the surveillance interval for planning purposes.

Based on this information, regional surveillance plans are developed and progress is reported on a monthly basis to the respective Regional Directors, operational management committee and national oversight office. Monthly regional surveillance data is rolled up into quarterly National Oversight Plans that are reviewed and approved by the operational management committee, National Civil Aviation Management Executive (NCAMX) and SO3MB. Deviations from the regional surveillance plans are reviewed by NCAMX and require approval from SO3MB.

Internal Audit is starting an Audit of Risk-Based Business Planning for Activities in Safety and Security that will look in depth at how oversight activities are planned, tracked and reported, including examining the information used to support those plans and reports. The audit will also be looking at the appropriateness of different levels of oversight activities based on whether the enterprise is an SMS or non SMS enterprise.

Areas requiring further attention:
NASIMS is the foundation of the risk assessment process of enterprises. The information required for NASIMS is collected and inputted by inspectors. There is a process in place to ensure the quality of information being inputted into NASIMS. {ATIP removed} We would suggest that Civil Aviation ensure the information inputted into NASIMS is both accurate and complete.

{ATIP removed}Civil Aviation should consider reviewing how it explains the functionality and purpose of NASIMS to the inspectorate.

Finally, there were concerns expressed that the risk indicator questions in NASIMS do not apply to all the sectors of aviation especially the unique issues of small operators. Civil Aviation should ensure the applicability of NASIMS and specifically the factors inspectors assess.

2.5. QUALITY CONTROL, QUALITY ASSURANCE

Internal Audit Expected:
There would be appropriate guidance on inspector surveillance activities, including the level of documentation required to support this activity. There would also be documented managerial oversight of inspector surveillance activities.

There would be a quality assurance program of the inspectors' surveillance activities.

Internal Audit's Assessment:
Civil Aviation has developed and implemented staff instructions on inspector surveillance activities. There is however a need for clearer guidance on sampling methods and the related training for inspectors. Staff instructions include the requirement for management review and approval of the planning, conducting and reporting of surveillance activities. In addition, there is documented managerial oversight of inspector surveillance activities.

There is a quality assurance program that includes inspectors' surveillance activities.

Observations to Support Internal Audit's Assessment:

Quality Control 
Management is reviewing and approving surveillance activities as evidenced in quality assessments, file reviews and group/committee minutes.

Roles and responsibilities for appropriate management review and approval for planning surveillance activities are documented in the Surveillance Policy (CAD SUR-008 2012-05-29) and in the National Surveillance Planning Standard (SI SUR-009 (2012-05-31).

Based on the OAG's audit and a self assessment by Civil Aviation, the Surveillance Procedures (SI SUR-001) were updated to include increased management review, approval and sign off requirements for the Surveillance Manager and Convening Authority throughout the surveillance process (pre-site, on-site, post-site), and a new layer of management review through the Surveillance Review Committee. The Surveillance Procedures also now provide many templates to document the surveillance activities. On-site regional workshops were held in 2013 in order to inform inspectors and managers of the updated Surveillance Procedures.

The internal audit team reviewed 61 regional inspection/assessment files using similar criteria as the 2012 OAG audit {ATIP removed}and found that improvements have been made in the consistency and the compliance to established procedures as well as management review. The internal audit file review found 98% of files had a sampling plan compared to {ATIP removed} the OAG's finding that sampling plans were rarely prepared. The internal audit file review also found evidence of management review in 66% of files compared to{ATIP removed} the OAG's finding of no formal management review and approval process.

The internal audit team also reviewed the Surveillance Procedures that guide inspectors when developing and implementing sampling plans. Specifically we reviewed the terminology, documentation requirements, methods used to determine sample size and samples, and the related templates. As well, a few inspectors were interviewed to better understand how they apply the procedures. We found improvements are needed to clarify terminology and inspectors application of the methodology especially related to random and non-random sampling.

The Civil Aviation Quality Control of Surveillance Activities Directive (CAD) QUA-017 outlines the guiding principles and establishes the minimum standards for quality control (QC) of surveillance activities. The aim is to ensure a documented and nationally consistent approach when conducting QC of the inspection of regulated enterprises in accordance with the process detailed in SI-SUR 001.

As of September 1, 2015, Civil Aviation is conducting 100% QC of surveillance activity files in all regions following the model used in Quebec region. All files will be assessed until an 85% conformance rating is achieved. Once the 85% threshold is achieved and sustained, Civil Aviation, through its Operations Management Committee (OMC), will establish a national standard for QC to be conducted on a random sample of activity.

Note that the Quebec region implemented 100% QC assessments of all surveillance activity files in 2013.

Quality Assurance
Civil Aviation developed and implemented the Civil Aviation Internal Quality Assurance Program (CAIQAP) within the Management and Resource Services branch. Its Risk-Based 5 Year Plan (2012-2013 to 2016-2017) was based on the results of workshops with representation from subject matter experts from across TC Civil Aviation. The proposed QA Assessments were prioritized based on audit recommendations and current departmental priorities. The CAIQAP Plan was approved by NCAMX in 2012. .

Of the 18 QA assessments planned in the five year CAIQAP Plan, four were completed, two are in progress, one is planned for this fiscal year, six will be recommended to management for next fiscal year, and five were deferred. Two additional assessments were conducted outside of the CAIQAP scope.

Staff Instructions have been developed to provide guidance and templates for conducting QA Assessments and developing and implementing Corrective Action Plans.

The CAIQAP Plan is subject to ongoing monitoring/tracking on a regular basis. Management and Resource Services reports the QA Assessment reports as well as the overall status of the Plan to NCAMX (three to four times per year). The Plan's status is also reported to senior management (ADM, Safety & Security, SO3MB) when requested (twice per year). Management and Resource Services reviews and updates the Plan on an annual basis.

Areas requiring further attention:
The guidance related to sampling needs to be clarified and inspectors provided with adequate training.

2.6. PERFORMANCE MEASUREMENT

Internal Audit Expected:
Civil Aviation to have developed key performance indicators that will assess the extent to which its programs and initiatives, including implementation of SMS, are contributing to achieving the Department's long-term objectives.

Internal Audit's Assessment:
Civil Aviation has developed annual Key Performance Indicators, {ATIP removed}.

Observations to Support Internal Audit's Assessment: 
Civil Aviation has developed a set of performance indicators that provide yearly targets for a three-year period. These performance indicators are reported annually in the Departmental Performance Report.

TC's Evaluation function reviewed Civil Aviation's performance indicators and provided a preliminary assessment of these indicators. They noted the Performance Measurement Framework relies more heavily on quantitative than qualitative methodologies, which is not unusual. Civil Aviation proposes client satisfaction indicators which would allow for the use of qualitative methods. Some elaboration would be beneficial to demonstrate the use of qualitative methods with regards to these indicators. Adding a measure of the extent to which the regulatory framework is harmonized with international legislation would provide further opportunity for the use of qualitative methods. {ATIP removed}

In addition, TC's Evaluation function has embarked on a new performance measurement initiative with a view to strengthening performance measurement in the Department. As part of this initiative the evaluation function will work with the Civil Aviation Directorate (as well as other modes in the Safety and Security group) to improve the relevance, quality, and comparability of performance data.

Areas Requiring Further Attention:
While Civil Aviation has developed performance indicators as identified in their MAP, these performance indicators could be strengthened. TC's Evaluation function will be working with Civil Aviation to this end.

2.7. HUMAN RESOURCES (NOTIP)

Internal Audit Expected:
The National Organization Transition Implementation Project (NOTIP) would be completed.

Internal Audit's Assessment:
NOTIP is now complete. The development and implementation of a change management plan is no longer relevant as this project has now been completed.

Observations to Support Internal Audit's Assessment:
Civil Aviation has confirmed, through their NOTIP Closure Report dated March 2014, that this project is now complete. This report assessed the project's performance and lessons learned and confirmed that the essential contractual and other project closure activities have been completed. This document also transferred assets, deliverables, residual issues, and all ongoing administrative functions to the Management Services Branch of Civil Aviation.

2.8. HUMAN RESOURCES (STAFFING)

Internal Audit Expected: 
Civil Aviation to have a national human resources plan that identifies the number of inspectors required to carry out its work, their competencies, and a recruitment strategy to fill any vacancies.

Civil Aviation to complete the staffing of middle management and supervisory positions currently filled on an acting basis.

Internal Audit's Assessment:
Civil Aviation has a human resources plan that identifies the number of inspectors required to carry out its work, their competencies, and a recruitment strategy to fill any vacancies. In addition, Civil Aviation has filled the majority of middle management and supervisory positions on an indeterminate basis.

Observations to Support Internal Audit's Assessment:
Civil Aviation has developed a national human resource plan that is currently embodied in three documents namely the Aviation Safety 2015-2016: Environmental Scanning, SWOT, Risk Analysis document, the HR Planning and the Aviation Safety People Management Action Plan, and the Integrated Planning Report.

In addition, Civil Aviation has identified four "Public Service Key Leadership Competencies" and four specific Civil Aviation competencies as follows:

• Leadership Competencies
  • Values & Ethics,
  • Strategic Thinking,
  • Engagement, and
  • Management Excellence
• Civil Aviation Specific Competencies
  • Communications
  • Partnering
  • Risk Management
  • Safety Management Systems

Civil Aviation's recruitment strategies were contained in the document National Civil Aviation Program Recruitment Strategies for Oversight Positions – Core Occupational Groups (AO, TI, EN and AI). This document is closed since Civil Aviation met its staffing target levels; ongoing activities continue such as national collective staffing inventories.

All middle management and supervisory positions which were occupied on a long-term acting basis have been staffed with the exception of five positions waiting to be classified as a result of NOTIP. Once those positions have been classified, they can be permanently staffed.

2.9. DATABASE SYSTEMS

Internal Audit Expected: 
The Business Intelligence Infrastructure (BII) project would be on track for completion March 2016.

Internal Audit's Assessment:
The BII project is not on track.

Observations to Support Internal Audit's Assessment: 
An extra year will be required to finalize BII. Civil Aviation and the Technology and Information Services Directorate have prepared an investment justification form in order to access capital funds next year. The BII project is listed as the top business priority for Civil Aviation. However, final approval is required to secure resources to complete the project.

Phase III, the final phase of the BII project, is scheduled to be completed by March 2017.

3. CONCLUSIONS

We found that Civil Aviation has implemented their management action plans for 17 of the 19 audit recommendations and those action plans appear to be working as designed. The two main areas requiring further work are as follows:

• Database Systems:
  • The action plan for an integrated solution to capture information relevant to the oversight of civil aviation safety was originally scheduled for completion in March 2016. This project is not on track and now has a new target completion date of March 2017.
• Training:
  • The OAG in 2012 recommended that Civil Aviation "should ensure all staff involved in inspections receives the training necessary to execute their responsibilities". In response, Civil Aviation has indicated that it had provided the necessary surveillance training to the majority of inspectors. We however are of the opinion that the recommendation is not referring to just surveillance training but rather all training requirements. {ATIP removed} Finally, we identified the need for training related to the non surveillance work performed by inspectors.

We have also highlighted in the report other specific areas requiring further attention and expect program management to address these as part of their continuous improvement efforts.

4. MANAGEMENT RESPONSE

Civil Aviation would like to thank the Office of Audit, Evaluation and Advisory Services for undertaking the Follow-up Audit of Civil Aviation.

Civil Aviation takes note of the observations provided in this report and will work towards continuous improvement activities in the areas noted in the report.

• Civil Aviation is in the process of developing a new course on surveillance and structured on-the-job training (SOJT) for new hires and will continue developing mandatory specialized training courses. (target dates vary on the stage of development and level of complexity)

• Civil Aviation, in collaboration with Multimodal Integrated Technical Training (MITT), will develop a systematic process to independently validate inspector training records {ATIP removed} (August 2016)

• Civil Aviation will revise and publish Civil Aviation Directive (CAD) ADM-005 – 'Required Training for Civil Aviation Employees who Develop, Deliver and/or Support Oversight Activities' to enhance and clarify the guidance material that establishes the required Civil Aviation learning program. (December 2016)

• Civil Aviation will develop a documented risk-based process to support the incremental issuance and restriction of inspector credentials. {ATIP removed}

• The Business Intelligence Infrastructure (BII) project is listed as the top business priority for Civil Aviation. Phase III, the final phase of the BII project, is scheduled to be completed by March 2017. Final approval, by TMX, is required to secure resources to complete the project. Civil Aviation and the Technology and Information Services Directorate have prepared an investment justification form in order to access capital funds next year. (March 2017)