Evaluation of Transport Canada’s Grant to the International Civil Aviation Organization (ICAO) for the Cooperative Development of Operational Safety and Continuing Airworthiness Program (COSCAP)

Executive Summary

This evaluation was required as per Section 42(1) of the Financial Administration Act. It addresses the relevance and performance of the grant program which provides funds to the International Civil Aviation Organization (ICAO) in support of the Cooperative Development of Operational Safety and Continuing Airworthiness Program (COSCAP). The evaluation examined the period from 2009-10 to 2013-14. Given that this grant program is of low materiality ($130,000 annually) and low risk, a streamlined approach to the evaluation was adopted, updating key information from the previous evaluation using financial data and information in ICAO documents.

The COSCAP initiatives consist of cooperative agreements among countries and were established to improve international aviation safety. There are several sub-regional COSCAP initiatives currently underway and Transport Canada remains free to allocate its funds towards any of the COSCAP initiative it chooses. Since 2003, Transport Canada has funded the COSCAP North Asia (COSCAP-NA) initiative. This initiative’s Member States include the People’s Republic of China, the Democratic People’s Republic of Korea, the Republic of Korea, and Mongolia.

The evaluation found that there is an ongoing need for the grant program due to the increasing economic and air traffic growth in North Asia. Without the grant to ICAO, Canada would be removed from the international discussions under the COSCAP that could affect Canadians traveling to this sub-region, and this could also reflect negatively on Transport Canada’s collaboration efforts in international aviation safety.

The evaluation also found that the grant program has been achieving its expected outcomes. Member States in North Asia have achieved improved capacity to provide safety oversight as indicated by the growing number of participants who have taken training courses offered through the COSCAP-NA and the improved compliance with ICAO Standards and Recommended Practices (SARPs) by Member States of the COSCAP-NA. The grant program has also been achieving its ultimate outcome of maintaining aviation safety, specifically in North Asia, as indicated by the small portion of accidents that occurred in this region during the time period covered by the evaluation.

Transport Canada has demonstrated efficiency in the use of the grant. Although Transport Canada has financed the COSCAP-NA since 2003, as a result of its participation in this initiative, it has been able to participate in meetings with COSCAP safety teams that cover the entire Asia and Pacific region without the need to finance any other initiatives in this larger regional area.

Recommendations were not warranted in the evaluation of the grant program for the COSCAP.

Introduction

Transport Canada’s Evaluation and Advisory Services conducted an evaluation of the grant program that provides funds to ICAO in support of the COSCAP in the summer of 2015. As per Section 42(1) of the Financial Administration Act, transfer payment programs must be evaluated every five years. The grant program was last evaluated in 2009 and its Terms and Conditions are set to expire in March 2019. This evaluation examined fiscal years 2009-10 to 2013-14.

Background

ICAO works with 191 states and global aviation organizations to develop international Standards and Recommended Practices (SARPs). In 1992, the ICAO Assembly recognized that many Member States might not have the regulatory framework or resources to enforce safety oversight requirements. In response, ICAO created the COSCAP in 1997, which establishes cooperative agreements among countries to improve international aviation safety. Canada has supported the COSCAP since its inception and is free to select which COSCAP initiative, among several, it will fund.

There are currently 7 sub-regional COSCAP initiatives underway worldwide. These sub-regional initiatives fund the development of aviation laws, regulations, procedures, guidance materials, and the provision of training to National Civil Aviation Authorities, service providers and air operators. Three of these initiatives are located in Asia, covering the South, Southeast, and North sub-regions.

Since 2003, Transport Canada has participated in the COSCAP North Asia (COSCAP-NA) initiative. The Member States of this initiative include the People’s Republic of China, the Democratic People’s Republic of Korea, the Republic of Korea, and Mongolia. The United States Federal Aviation Administration, the European Aviation Safety Association, Airbus, and the Boeing Company also support COSCAP-NA through financial or in-kind contributions.

Program Profile

As stated in the Terms and Conditions, the grant to ICAO covers costs associated with salary, travel and technical training in relation to the COSCAP. They also state that the amount of the grant will be at the minimum level required to further the attainment of the objectives of the COSCAP and the results expected from the recipient. The maximum amount payable for the grant to ICAO is $130,000 per year. Between 2009-10 and 2013-14, Transport Canada provided an annual amount of $130,000, totaling $650,000 over the five years.

The grant to ICAO was renewed in 2014-15. No changes were made to the Terms and Conditions of the grant during its renewal.

Evaluation Approach

Given the grant program’s low materiality and low complexity of delivery, the evaluators adopted a streamlined evaluation approach^{Footnote 1}, relying primarily on a review of relevant documents.

The evaluation assessed the relevance/rationale and performance of the grant program, as well as whether efficiency could be demonstrated in the utilization of resources. Performance was assessed based on the activities and expected results outlined in the grant program’s performance measurement strategy. The logic model for the grant program is presented in Table 1 and includes the immediate, intermediate, and ultimate outcomes, as measured according to the grant program’s performance measurement strategy.

Table 1: Logic Model for the grant program in support of the COSCAP

Activities	Outputs	Immediate Outcomes	Intermediate Outcomes	Ultimate Outcomes
Prepare Funding Agreement Attend COSCAP Steering Committee Meetings Review Status Reports and ICAO Audits	Funding Agreement	Increased capacity to provide safety oversight as measured by the provision of training coursesFootnote 2	Compliance with ICAO Standards and Recommended Practices (SARP) by Member States	A safe aviation system

The following research questions were used to support the evaluation:

Lines of Inquiry

The evaluation used the following lines of inquiry to answer the research questions and inform the overall evaluation:

Document Review

Documentary evidence served as the primary source of information for this evaluation and was used to assess the grant program’s relevance and performance. Key documents include Speeches from the Throne, federal budgets, and departmental reports on performance. Transport Canada’s financial records, invoices from ICAO, and the Terms and Conditions of the grant were reviewed to assess outputs and efficiency.

In addition, ICAO Working Papers, Annual Reports, and Records of Discussions at Steering Committee Meetings were reviewed to assess performance by examining whether the COSCAP has lead to an increase in capacity to provide safety oversight and whether compliance with ICAO SARPs has been achieved by Member States.

Counterfactual Analysis

Counterfactual analysis was used to assess the rationale and ongoing need for the grant program in support of the COSCAP, and to assess the impact if Canada were to withdraw its support for COSCAP through the grant to ICAO.

Evaluation Findings

Relevance/Rationale

Economic growth in Asia is projected to increase at an annual rate of 4.3% over the next 20 years. Likewise, Asia continues to experience rapid growth in low-cost air carriers, primarily AirAsia, which currently operates over 300 aircraft and has another 900 on order. An average of 1 billion passengers travel to, from or within the region each year and it is expected that Asia will be the largest aviation market in the world, growing at an annual rate of 6.1%.

Boeing estimates that roughly 50% of air traffic growth in the next 20 years will be in Asia and that within this time period, Asia will require 216,000 new pilots to meet passenger demand. Low-cost air carriers, however, currently do not have enough training programs. Capacity issues with meeting training needs, coupled with increasing growth in air traffic, suggests that there is a continued need for the COSCAPs. As markets and air traffic continue to grow in Asia, this sub-region will remain a key strategic area for Canada, both economically and on the safety front.

Canada has been the host country for ICAO since the organization was founded in 1947 and ICAO plays a key role in the governance of civil aviation in Canada and world-wide. Transport Canada has been engaged in international aviation, and in terms of the COSCAP, participates in the multilateral discussions that take place at the COSCAP Steering Committee meetings.

If Canada were to no longer participate in the COSCAP, it would remove itself from the decisions and discussions that could affect the safety of Canadian passengers traveling on the airlines of Member States that participate in the COSCAPs, and more importantly, Canada’s reputation as an important player in international aviation could be negatively impacted.

Aviation safety is a federal responsibility in Canada, as is the responsibility for entering into multilateral international agreements. According to Transport Canada’s Program Alignment Architecture, the grant program is aligned with Strategic Outcome 3: ‘A Safe and Secure Transportation System’. It is part of Program Sub-Activity 3.1.1, ‘Aviation Safety Regulatory Framework’, which promotes a safe and harmonized aviation safety framework for Canadians and Canada’s aviation industry.

Performance

In terms of performance, as measured according to the grant program’s performance measurement strategy, the evaluation found the following:

In terms of outputs, Transport Canada provided an annual grant of $130,000 to support salary, travel and technical training costs for the COSCAP, as stated in invoices from ICAO. Invoices also indicate that Transport Canada’s annual payments were made on time and were received by ICAO before March 31st of each year for the 2009-10 to 2013-14 time period.

Transport Canada is to attend Steering Committee meetings as part of the grant program’s activities, and the evaluation found that a Transport Canada representative attended most of the annual COSCAP Steering Committee meetings for the 2009-10 to 2013-14 time period. The Records of Discussions for these meetings indicate that presentations were given by Transport Canada on aviation safety.

As part of the grant program’s activities, Transport Canada also received and reviewed annual COSCAP status reports and working papers from ICAO. Although Transport Canada also receives and reviews Audit Reports, ICAO did not conduct any audits on Member States under its Universal Safety Oversight Audit Programme (USOAP)^{Footnote 3} during the period covered by the evaluation. This is because in 2010, the Continuous Monitoring Approach (CMA) was applied to the USOAP, enabling a more proactive approach to audits. A major component of the CMA is the State Safety Program (SSP), which allows the Member State and aviation service providers to interact more effectively in the resolution of safety concerns and therefore provide a system for the management of safety within the State.

The immediate result expected for the COSCAP is that through the completion of training courses, Member States will increase their capacity in aviation safety oversight. Table 1 presents the number of aviation safety training courses provided through the COSCAP-NA between 2009 and 2014 along with the total number of participants. The Records of Discussions from the COSCAP-NA Steering Committee meetings indicate that the training courses provided through the COSCAP-NA have largely focused on State Safety Program (SSP) training for Member States to help improve aviation safety oversight in this sub-region and support the implementation of the USOAP Continuous Monitoring Approach (USOAP CMA).

The evaluation found that the number of aviation safety training courses between 2011-12 and 2013-14 has more than doubled, and with that, the total number of participants trained has also increased significantly.

Table 1: Number of training courses provided and total participants trained through COSCAP-NA during the 2009-10 to 2013-14 time period, as indicated in ICAO Annual Reports

Year	2009	2010	2011	2012	2013	2014	Total
Number of training courses provided	27	18	8	8	17	17	95
Total participants	450	263	124	336	547	799	2,519

A total of 95 aviation safety training courses were delivered through the COSCAP-NA between 2009 and 2014, and as a result of these courses, 2,519 participants received training. The total number of training courses and trained individuals indicates that the COSCAP-NA has helped Member States increase their capacity to provide aviation safety oversight and since Transport Canada has supported this training, it has also contributed to the increased capacity in North Asia.

In regards to compliance with ICAO SARPs by Member States, which is the intermediate result expected for the COSCAP, the Record of Discussions from the COSCAP-NA Steering Committee meeting in 2014 state that all Member States in the COSCAP-NA have achieved a USOAP effective implementation level of greater than 60%. ICAO has indicated that States that have achieved USOAP effective implementation levels over 60% should be in a position to fully implement their SSP by 2017 as they have demonstrated a capacity to provide aviation safety oversight.

As per the grant program’s performance measurement strategy, the ultimate outcome is a safe aviation system. Table 2 presents the number of commercial accidents and fatalities between 2012 and 2014 in North Asia as well as the entire Asia and Pacific region according to annual ICAO Safety Reports.

Table 2: ICAO commercial accident statistics in North Asia and the entire Asia and Pacific region

YearFootnote 4	North Asia		Asia and Pacific region
	# of commercial accidents	# of fatalities	Total commercial accidents	Total fatalities
2012	1	0	20	161
2013	3	0	19	49
2014	3	48	19	449
TOTAL	7	48	38	498

ICAO Safety Reports indicate that 7 of the 38 accidents across the Asia and Pacific region occurred in North Asia (China: 5, Mongolia: 1, Republic of Korea: 1). This suggests that the majority of the commercial accidents in the Asia and Pacific region between 2012 and 2014 occurred outside of North Asia.

The accident that caused 48 fatalities in 2014 occurred in China, and according to the ICAO Safety Report, was the result of a controlled flight into/towards terrain. ICAO Safety Reports also stated that the remaining accidents in North Asia were related to turbulence encounter (weather-related) or runway safety (see Table 3).

Table 3: Accident causes in North Asia as reported in ICAO Safety Reports

Member State	Total Accidents (2012-2014)	Accident Category
China	5	Turbulence encounter (3) Runway safety-related (1) Controlled flight into terrain (1)
Mongolia	1	Runway safety-related (1)
Republic of Korea	1	Turbulence encounter (1)

The accident statistics suggest that given the small proportion of accidents in North Asia relative to the entire Asia and Pacific region, overall aviation safety in North Asia is being maintained.

Efficiency

In terms of efficiency^{Footnote 5}, evaluators found the following:

Transport Canada’s allocation of the grant to ICAO towards the COSCAP-NA has resulted in the efficient use of financial resources. This is because Transport Canada is represented on the Asia and Pacific Regional Safety Team (APRAST) by the Canadian Chief Technical Advisor for the COSCAP-NA. Participation in the APRAST allows for a wider sharing of information and this has enabled Transport Canada to remain informed of aviation safety issues across the Asia and Pacific region, without having to fund the other COSCAP initiatives in this region.

Conclusion

The grant program enables Canada to collaborate on international capacity building projects, participate at the annual COSCAP Steering Committee meetings and thereby be involved in decisions that affect the safety of Canadian passengers traveling on the airlines of Member States that participate in the COSCAP. If Canada were to remove its funding for the COSCAP, Canada would be removed from the multilateral decision-making that takes place under these initiatives, and this could affect its reputation as a key player in international aviation.

The evaluation also found that there is a continuing need for the grant program to fund the COSCAP-NA due to the increasing economic and air traffic growth in Asia. As a result of this growth, there has been an increased need for aviation safety training in the region.

The grant program has been achieving its expected outcomes of increased capacity by Member States to provide aviation safety oversight. Transport Canada’s grant program supported the delivery of safety management oversight training courses through the COSCAP-NA during the 2009-10 to 2013-14 time period, which resulted in 2,519 trained participants. The evaluation found that Member States of the COSCAP-NA have improved their compliance with ICAO SARPs and therefore increased their capacity to provide aviation safety oversight as indicated by their USOAP implementation levels greater than 60%. In addition, it was found that aviation safety in North Asia is being maintained given the small proportion of commercial accidents observed in the region during the time period covered by the evaluation.

The evaluation also found that Transport Canada has demonstrated efficiency in the use of resources through Transport Canada’s participation in COSCAP-NA. Its participation in COSCAP-NA allows Transport Canada to participate in COSCAP-NA Steering Committee meetings, as well as Steering Committee meetings that comprise of representatives from the entire Asia and Pacific region. Transport Canada is able to attend these meetings without having to provide funds to other COSCAP initiatives in this area. Participation in these meetings allows Transport Canada to remain informed of aviation safety issues across the Asia and Pacific region.