Advisory Circular (AC) No. 500-016 Issue 1 - Appendix B

Procedure for Evaluating the Practicality of Applying the Latest Standards to a Changed Product

B.1 Introduction

The basic principle of enhancing the level of safety of changed aeronautical products is to apply the latest standards on significant design changes, to the greatest extent practicable. In certain cases, the cost of complying fully with a later standard may not be commensurate with the small safety benefit achieved. It is recognized that the existing fleet and newly produced aeroplanes, engines and propellers are safe, and that any unsafe condition is immediately addressed through the airworthiness directive process. These factors form the basis by which compliance with the latest standard may be considered not practical, thereby allowing and accepting compliance with an earlier standard. This Appendix gives one method of determining if compliance with a later standard is practical or not. However, this does not preclude the use of other methods that have as a goal improving the safety of aeronautical products.

This AC recognises that other procedures have been used for some products and have, historically, been accepted on a case-by-case basis. These other procedures have not been fully harmonized and may not be acceptable for all products. It is envisaged that other methods will be developed and may become part of this AC in the future. Regardless of which method is used, the method must show that a proposed certification basis is resource-effective when it is able to achieve a positive safety benefit for the overall product. In this regard, any method used must also encourage incorporating the safety enhancements that will have the most dramatic impact on the accident rate and recognize the effective utilization of limited resources. This important point is illustrated graphically in the accompanying figure. The figure notionally shows the interrelation between the incremental resources required for incorporating each potential safety enhancement with the corresponding net increase in safety benefit. Typically, one will find that there are proposals that can achieve a positive safety benefit that are resource effective. Conversely, there are proposals that may achieve a relatively smaller safety benefit at the expense of a large amount of resources to implement. Clearly, there will be a point where a large percentage of the potential safety benefit can be achieved with a reasonable expenditure of resources. The focus of the methods used should be to determine the most appropriate set of safety-significant regulatory standards relative to the respective cost to reach this point

 

B.2 Procedure

This Appendix provides procedural guidance that can be used as a starting point to determine the practicality of applying a standard at a particular amendment level to a changed product. This guidance can be used for evaluating the safety benefit and resource impact of implementing the latest standards in the certification basis of a changed product. The procedure is generic in nature and describes the steps and necessary input that an applicant could use on a project to develop a position:

1. The procedure is intended to be used, along with good engineering judgement, to evaluate the relative merits of a changed product complying with the latest standards.
    
2. This procedure provides a means, but not the only means, for an applicant to present its position on practicality or impracticality. 
   
   The certification basis for a change to a product will not be at an amendment level earlier than the existing certification basis. Therefore, when determining the practicality of applying the latest standard, only the increase in safety benefits and costs beyond compliance with the existing certification basis should be considered.
    
3. The following are seven steps to determine the practicality or impracticality of applying a requirement at a particular amendment level.

The first step will be to identify the regulatory change being evaluated.

Step 1 - Identify the Latest Standard Being Evaluated. In this step it will be necessary to document:

1. The specific standard under evaluation (e.g., AWM 525.365/FAR 25.365);
    
2. The amendment level of the above standard in the existing certification basis; and
    
3. The latest amendment level of the standard under evaluation.

Step 2 - Identify the Specific Hazard that the Latest Standard Addresses.

Each standard and its progressive amendment is intended to address a hazard or hazards. In this step, the specific hazard(s) are identified. This identification will allow for a comparison of the effectiveness of amendment levels of the standard at addressing the hazard.

In many cases the hazard and the cause of the hazard will be obvious. When the hazard and its related cause are not immediately obvious, it may be necessary to review the preamble of the amendment to the standard. It may also be helpful to discuss the hazard with the responsible Transport Canada Aircraft Certification office.

Notes:

The various standards of airworthiness (or Chapter) in the AWM are based mostly on the FARs. Transport Canada's process of introducing FAR Amendments to the AWM usually involves publishing a single Change (e.g. Change 525-8) that actually consists of several (bundled) FAR regulatory amendments for the affected Chapter. As such, the amendment levels indicated at the bottom of each section/paragraph of the AWM are those of Transport Canada rather than the actual FAR amendment. For purposes of obtaining the history and basis of each FAR requirement, the applicant should refer to the preambles of the draft and final rules of the specific FAR Amendment(s) being evaluated.

For AWM standards that are based on the JARs, the amendment information referred to in Note 1 above can be found in the Explanatory Note and Comment/Response documents of the applicable JAA Notice of Proposed Amendment (NPA).

Step 3 - Review the Consequences of the Hazard(s).

Once the hazard has been identified, it is possible to identify the types of consequences that may occur because of the presence of the hazard. More than one consequence can be attributed for the same hazard. Typical examples of consequences would include but not be limited to:

1. Incidents where only injuries occurred;
    
2. Accidents where less than 10% of the passengers succumbed to their injuries;
    
3. Accidents where 10% or more passengers succumbed to their injuries; and
    
4. Accidents where a total hull loss occurred.

The preamble to the regulation may provide useful information regarding the consequences of the hazard that the standard is intended to address.

Step 4 - Identify the Historical and Predicted Frequency of each Consequence.

Another input in determining practicality is the historical record of the consequences of the hazard that led to a standard or an amendment to that standard. From this data a frequency of occurrence for the hazard can be determined. It is important to recognise that the frequency of occurrence may be higher or lower in the future. Therefore, it also is necessary to predict the frequency of future occurrences.

More than one consequence can be attributed for the same hazard. Therefore, when applicable, the combination of consequences and frequencies of those consequences should be considered together.

The preamble (to the regulation) may provide useful information regarding the frequency of occurrence.

Step 5 - Determine How Effective Full Compliance with the Latest Standards would be at Addressing the Hazard.

When a standard is amended it is expected that compliance with the latest standard would be completely effective at addressing the associated hazard. It is expected that the hazard would be eliminated, avoided, or dealt with. However, in a limited number of situations this may not be the case. It is also possible that earlier standards may have addressed the hazard but was not completely effective. Therefore, in comparing the benefits of complying with the existing certification basis with that of complying with the latest standards it is useful to estimate the effectiveness of both amendment levels in dealing with the hazard.

It is recognised that the determination of levels of effectiveness is normally of a subjective nature. Therefore, prudence should be exercised when making these determinations. In all cases, it is necessary to document the assumptions and data that support the determination.

The following five levels of effectiveness are provided as a guideline:

1. Fully effective in all cases - Compliance with the latest standard eliminates the hazard or provides a means to completely avoid the hazard;
    
2. Considerable potential for eliminating or avoiding the hazard - Compliance with the latest standard eliminates the hazard or provides a means to completely avoid the hazard for all probable or likely cases. However it does not cover all situations or scenarios;
    
3. Adequately deals with the hazard - Compliance with the latest standard eliminates the hazard or provides a means to completely avoid the hazard in many cases. However, the hazard is not eliminated or avoided in all probable or likely cases. Usually this action only addresses a significant part of a larger or broader hazard;
    
4. Hazard only partly addressed - In some cases, compliance with the latest standard partly eliminates the hazard or does not completely avoid the hazard. The hazard is not eliminated or avoided in all probable or likely cases. Usually this action only addresses part of a hazard; and
    
5. Hazard only partly addressed but action has negative side effect - Compliance with the latest standard does not eliminate or avoid the hazard or may have negative safety side effects. The action is of questionable benefit.

Step 6 - Determine Resource Costs and Cost Avoidance.

There is always a cost associated with complying with a standard. This cost may range from minimal administrative efforts to the resource expenditures necessary to support full scale testing or the redesign of a large portion of an aircraft. However, there are also potential cost savings from compliance with later standards. For example, compliance with later standards may avoid aircraft damage or accidents and the associated costs to the manufacturer for investigating accidents. Compliance with the latest standard may also facilitate certification of a product by a foreign aviation authority.

When determining the practicality of applying the latest standard, only the incremental costs, and added safety benefits from complying with the existing certification basis should be considered.

When evaluating the incremental cost, it may be beneficial for the applicant to compare the increase in cost to comply with the latest standards to the cost to incorporate the same design feature in a new aeroplane. An estimate of the cost of incorporation in a new aeroplane may be available in the preambles of the referenced standard (or regulations) when the corresponding amendment was first drafted and then finalized. Incremental costs for retrofit/incorporation on existing designs may be higher than that for production. Examples of costs may include, but are not limited to:

1. Costs -The accuracies of fleet size projections, utilisation, etc. may be different than that experienced in actuality for derivative product designs and must be validated.
    
   1. Labour - Work carried out in the design, fabrication, inspection, operation or maintenance of a product for the purpose of incorporating or demonstrating compliance with a proposed action. Non-recurring labour requirements, including training should be considered.
       
   2. Capital - Construction of new, modified or temporary facilities for design, production, tooling, training or maintenance.
       
   3. Material - Cost associated with product materials, product components, inventory, kits and spares.
       
   4. Operating Costs - Costs associated with fuel, oil, fees and expendables.
       
   5. Revenue/Utility Loss - Costs resulting from earning/usage capability reductions from departure delays, product downtime, capability reductions of performance loss due to seats, cargo, range or airport restrictions.
       
2. Cost Avoidance:
    
   1. Avoiding cost of accidents including investigation of accidents, lawsuits, public relations activities, insurance, and lost revenue.
       
   2. Foreign Certification: Achieve a singular effort that would demonstrate compliance to the requirements of most certifying agencies, thus minimising certification costs.

Step 7 - Document the Conclusion Regarding Practicality.

Once the information from previous steps have been documented and reviewed, the applicant's position and rationale regarding practicality, or impracticality, can be documented. Examples of possible positions would include, but are not limited to:

1. Compliance with the latest standard is necessary. The applicant would pursue the change at the latest amendment level;
    
2. Compliance with an amendment level between the existing certification basis and the latest standards t would adequately address the hazard at an acceptable cost, while meeting the latest standards would be impractical. The applicant would then propose an intermediate amendment level of the standard;
    
3. The increased level of safety is not commensurate with the incremental costs associated with meeting the latest standard. Therefore, the applicant would propose the existing certification basis; and
    
4. The results of this analysis were inconclusive. Further discussions with Transport Canada Aircraft Certification office are warranted.

Note:

This process may also result in a required certification basis that renders the proposed modification economically not viable.

B.3 Examples

The following examples are for large aeroplanes and are illustrative of the typical process followed by an applicant. The process will be the same for all product types. For the purpose of these examples, the FARs is used instead of the AWM to facilitate direct reference to specific amendment levels of the standards and use the information presented in the associated preamble to the regulations.

Example 1 - FAR 25.963 Fuel Tank Access Covers

This change is part of a significant transport aeroplane model change that increases passenger payload and gross weight by extending the fuselage 20 feet. The model change will feature increased thrust engines, strengthened wing and fuselage, and a completely redesigned landing gear. To accommodate the higher design weights, increased braking requirements and to reduce runway loading, the applicant will change the landing gear from a two-wheel to four-wheel configuration. The new model aeroplane will be required to comply with the latest applicable standards based on the date of application.

The wing will be strengthened locally at the side of the body and at the attachment of engines and landing gear, but the applicant does not wish to alter the wing access panels and the fuel tank access covers. Although the applicant recognises that the scatter pattern and impact loading on the wing from debris being thrown from the landing gear will change, he proposes that it would not be practical to redesign the fuel tank access covers.

The applicant proposes to change the landing gear from a two-wheel configuration to a four-wheel configuration. This changes the debris scatter on the wing from the landing gear.

The procedures presented in Section B.2 are applied, as follows:

Step 1 - Identify the Latest Standard Being Evaluated.

The existing certification basis of the aeroplane that is being changed is FAR 25 prior to Amendment 69.

Amendment 25-69 added the requirement that fuel tank access covers on transport category aeroplanes be designed to minimise penetration by likely foreign objects, and be fire resistant.

Step 2 - Identify the Specific Hazard that the Latest Standard Addresses.

Fuel tank access covers have failed in service due to impact with high-energy objects such as failed tire tread material and engine debris following engine failures. In one accident, debris from the runway impacted a fuel tank access cover, causing its failure and subsequent fire, which resulted in fatalities and loss of the aeroplane. Amendment 25-69 will ensure that all access covers on all fuel tanks are designed or located to minimise penetration by likely foreign objects, and are fire resistant.

Step 3 - Review the Consequences of the Hazard(s).

Occurrences with injuries, and with more than 10% deaths.

Step 4 - Identify the Historical and Predicted Frequency of each Consequence.

In 200 million departures of large jets,

1. 1 occurrence with more than 10% deaths, and
    
2. 1 occurrence with injuries.

There is no reason to believe that the future rate of accidents will be significantly different than the historical record.

Step 5 - Determine How Effective Full Compliance with the Latest Standards t would be at Addressing the Hazard.

Considerable potential for eliminating or avoiding the hazard.

Compliance with Amendment 25-69 eliminates the hazard or provides a means to completely avoid the hazard for all probable or likely cases. However, it does not cover all situations or scenarios.

Step 6 - Determine Resource Costs and Cost Avoidance.

1. Costs:
    
   1. For a newly developed aeroplane there would be minor increases in labour resulting from design and fabrication.
       
   2. There would be a negligible increase in costs related to materials, operating costs, and revenue utility loss.
       
2. Cost avoidance:
    
   1. There were 2 accidents in 200 million departures. The applicant believes that it will manufacture more than 2000 of these aeroplanes or derivatives of these aeroplanes. These aeroplanes would average 5 flights a day. Therefore, statistically there will be accidents in the future if the hazard is not alleviated. Compliance will provide cost benefits related to avoiding lawsuits, accident investigations and public relation costs.
       
   2. There are cost savings associated with meeting a single certification basis for the domestic and foreign requirements.

Step 7 - Document the Conclusion Regarding Practicality.

It can be concluded that compliance with the latest standard will increase the level of safety at a minimal cost to the applicant. Based on the arguments and information presented by the applicant, through the issue paper process, the Minister determined that compliance with the latest standard would be practical.

Example 2 - FAR 25.365 Pressurized Compartment Loads

For the product change described in Example 1 above, the lengthened fuselage affects the size of the main deck passenger compartment and the lower centre cargo compartment. The applicant plans to comply fully with the latest pressurized compartment loads, except for one interior partition for which the applicant believes that compliance would not be practical.

The applicant proposes to increase the length of the fuselage by installing fuselage plugs. This change affected the size of the main deck passenger compartment and the lower centre cargo compartment.

The procedures presented in Section B.2 are applied, as follows:

Step 1 - Identify the Latest Standard Being Evaluated

The existing certification basis of the aeroplane that is 25.365 required that interior structure of passenger compartments be designed to withstand the effects of a sudden release of pressure through an opening resulting from the failure or penetration of an external door, window, or windshield panel, or from structural fatigue or penetration of the fuselage, unless shown to be extremely remote.

Amendment 25-54 revised FAR 25.365 to require that the interior structure be designed for an opening resulting from penetration by a portion of an engine, an opening in any compartment of a size defined by FAR 25.365(e)(2), or the maximum opening caused by a failure not shown to be extremely improbable.

Amendment 25-71 extended the requirement to all pressurized compartments, not just passenger compartments, and to the pressurization of unpressurized areas. The later requirement had previously been identified as an unsafe feature under FAR 21.21(b)(2)

Step 2 - Identify the Specific Hazard that the Latest Standard Addresses

The hazard is a catastrophic structure and/or system failure produced by a sudden release of pressure through an opening in any compartment in flight. This opening could be caused by an uncontained engine failure, an opening of a prescribed size due to the inadvertent opening of an external door in flight, or by an opening caused by a failure not shown to be extremely improbable. The opening could be produced by an event that has yet to be identified.

Step 3 - Review the Consequences of the Hazard(s)

Occurrences with injuries, less than 10% deaths, and more than 10% deaths

Step 4 - Identify the Historical and Predicted Frequency of Each Consequence

In 200 million departures of large jets:

1. 2 occurrences with more than 10% deaths;
    
2. 1 occurrence with less than 10% deaths; and
    
3. 1 occurrence with injuries.

There is no reason to believe that the future rate of accidents will be significantly different than the historical record.

Step 5 - Determine How Effective Full Compliance with the Latest Standards would be at Addressing the Hazard

Fully effective in all cases. Compliance with Amendment 25-71 eliminates the hazard or provides a means to completely avoid the hazard.

Considerable potential for eliminating or avoiding the hazard. Compliance with Amendment 25-54 eliminates the hazard or provides a means to completely avoid the hazard for all probable or likely cases. However, it does not cover all situations or scenarios.

Adequately deals with the hazard. Compliance with the existing certification basis eliminates the hazard or provides a means to completely avoid the hazard in many cases. However, the hazard is not eliminated or avoided in all probable or likely cases. Usually this action only addresses a significant part of a larger or broader hazard.

Design changes made to the proposed derivative aeroplane bring it nearly into compliance with FAR 25.365 Amendment 25-71. Analyses show that one interior partition would fail when subjected to the pressure differential defined by the latest requirement. However, its failure would not have an impact on continued safe flight and landing. This is because none of the critical or essential systems are affected by failure of this partition and its failure would not present a hazard to a crewmember. Design solutions were considered for this partition, including structural reinforcement and additional venting area, but all were found to require substantial changes. With this design the applicant believes that most of the safety benefits have been achieved and that no appreciable increase in safety would be achieved by complying fully with Amendment 25-71.

Step 6 - Determine Resource Costs and Cost Avoidance

1. Costs:
    
   1. For a newly developed aeroplane there would be a significant increase in costs related to labour and capital to comply with Amendment 25-71 instead of the original certification basis.
       
   2. There would be a negligible increase in costs related to materials, operating costs, and revenue utility loss.
       
   3. There would be savings in both labour and capital costs if compliance were shown to Amendment 25-54, instead of Amendment 25-71.
       
2. Cost Avoidance:
    
   1. There were 4 accidents in 200 million departures. The applicant believes that it will manufacture more than 2000 of these aeroplanes or derivatives of these aeroplanes. These aeroplanes would average 5 flights a day. Therefore, statistically there will be accidents in the future if the hazard is not alleviated. Compliance will provide cost benefits related to avoiding lawsuits, accident investigations and public relation costs.
       
   2. There are cost savings associated with meeting a single certification basis for domestic and foreign requirements.

Step 7 - Document the Conclusion Regarding Practicality

The design is in compliance with FAR 25.365 Amendment 25-54, and nearly in full compliance to Amendment 25-71. The design would adequately address the hazard at an acceptable cost. Therefore, based on the arguments that it is not practical, as discussed in an issue paper, the Minister accepts the applicant's proposal to comply with FAR 25.365 Amendment 25-54.