Advisory Circular (AC) No. 500-016 Issue 1 - Appendix C

The Use of Service Experience in the Certification Process

C.1 Introduction

Service experience may be utilised to support the application of earlier standards if, in conjunction with the applicable service experience and other compliance measures, it provides a level of safety comparable to that provided by the latest standards. It is incumbent on the applicant to provide sufficient substantiation to allow the Minister to make this determination. A statistical approach may be used, subject to the availability and relevance of data. However, sound engineering judgement must be used. For a service history to be acceptable, the data must be both sufficient and pertinent.

The essentials of the process involve:

1. A clear understanding of the amended standards and the purpose of the amendment(s);
    
2. A determination based on detailed knowledge of the proposed design feature;
    
3. The availability of pertinent and sufficient service experience data; and
    
4. A comprehensive review of that service experience data.

C.2 Guidelines

The Issue Paper process would be used and the applicant should provide documentation to support the following:

1. The identification of differences between the standards in the existing certification basis and the amended standards, including the effect of the amendment to the standards.
    
2. A description as to what aspect of the latest standards the proposed change would not meet.
    
3. Evidence showing that the proposed certification basis for the changed product, together with applicable service experience, provides a level of safety consistent with complying with the latest standards.
    
4. A description of the design feature and its intended function.
    
5. Data for the product pertinent to the requirement:
    
   1. Service experience from such sources as the following:
       
      1. Accident Reports;
      2. Incident Reports;
      3. Service Bulletins;
      4. Airworthiness Directives;
      5. Repairs;
      6. Modifications;
      7. Flight hours/cycles for fleet leader and total fleet;
      8. World Airline Accident Summary (WAAS) Data;
      9. Service Difficulty Reports;
      10. (National) Transportation Safety Board Reports (NTSB/TSB); and
      11. Warranty, repair and parts usage data.
           
   2. Show that the data presented represents all relevant service experience for the product, including the results of any operator surveys, and is comprehensive enough to be representative;
       
   3. Show that the service experience is relevant to the issue;
       
   4. Identification and evaluation of each of the main areas of concern, with regard to:
       
      1. Recurring and/or common failure modes;
      2. Cause;
      3. Probability, by qualitative reasoning; and
      4. Measures already taken and their effects.
          
   5. Relevant data pertaining to aircraft of similar design and construction may be included;
       
   6. Evaluation of failure modes and consequences through analytical processes. The analytical processes should be supported by:
       
      1. A review of previous test results; and
      2. Additional detailed testing.
          
6. A conclusion that draws together the data and the rationale.

Note:

These guidelines are not intended to be limiting, either in setting required minimum elements or in precluding alternative forms of submission. Each case may be different, based on the particulars of the system being examined and the requirement to be addressed.

C.3 Example

The following example is for large aeroplanes and is illustrative of the typical process followed by an applicant. The process will be the same for all product types. For the purpose of this example, the FARs is used instead of the AWM to facilitate direct reference to specific amendment levels of the standards and use the information presented in the associated preamble to the regulations.

Transport Aeroplanes: FAR 25.1141(f), Auxiliary Power Unit (APU) Fuel Valve Position Indication

This example comes from a new generation model transport aeroplane where extensive changes where made to the main airframe components, engines and systems. The baseline aeroplane has an extensive service history. The purpose of the example is to show how the use of service experience can support an applicant's position that compliance with the latest standards would not contribute materially to the level of safety, and that continued compliance with the existing certification basis (or later amendment) would be appropriate. The example is for significant derivatives of transport aeroplanes with extensive service history. It is provided to illustrate the process, following the guidelines given in this Appendix, but does not include the level of detail that would normally be required.

The process presented in Section C.2 (Guidelines) is applied, as follows:

1. Identify the differences between the standards in the existing certification basis and the standards as amended, including the effect of the amendment(s) to the standards.
    
   "The existing certification basis of the aeroplane that is being changed is the initial release of FAR 25. Amendment 25-40 added the requirement FAR 25.1141(f) that power-assisted valves must have a means to indicate to the flight crew when the valve is in the fully open or closed position, or is moving between these positions."
    
2. What aspect of the latest standards would not be met by the proposed change?
    
   "The proposed APU fuel valve position indication system does not provide the flight crew with fuel valve position or transition indication, and therefore does not comply with the standards of FAR 25.1141(f)."
    
3. Provide evidence that the proposed certification basis for the changed product, together with the applicable service experience and other compliance measures, will provide an acceptable level of safety.
    
   "The APU fuel shut off valve and actuator are unchanged from those used on the current family of aeroplanes, and has been found to comply with the earlier Amendment 25-11 of FAR 25.1141(f). The existing fleet has achieved approximately xx flights during which service experience of the existing design has been found to be acceptable. If one assumes a complete APU cycle, i.e. start up and shutdown for each flight, the number of APU fuel shut off valve operations would be over 108 cycles, which demonstrates that the valve successfully meets its intended function and complies with the intent of the requirement. In addition, the system design for the changed product incorporates features, which increase the level of functionality and safety."
    
4. Describe the design feature and its intended function.
    
   "The fuel shut off valve, actuator design, and operation is essentially unchanged; with the system design ensuring that the valve is monitored for proper cycling from closed to open at start initiation. If the valve is not in the appropriate position (i.e. closed) then the APU start is terminated, an indication is displayed on the flight deck and any further APU starts are prevented. Design improvements using the capability of the APU Electronic Control Unit (ECU) have been incorporated in this proposed product change. These design changes ensure that the fuel valve indication system will indicate failure of proper valve operation to the flight crew, albeit the system does not indicate valve position as required by FAR 25.1141(f)."
    
5. Provide data for the product pertinent to the requirement.
    
   "An issue paper was co-ordinated, which included data or referenced reports, documenting relevant service experience that has been compiled from incident reports, fleet flight hour/cycle data, and maintenance records. The issue paper also discussed existing and proposed design details, failure modes, and analyses showing to what extent the proposed aeroplane complies with the latest amendment of FAR 25.1141. Information is presented to support the applicant's argument that compliance with the latest amendment would not materially increase the level of safety. Comparative data pertaining to aircraft of similar design and construction are also presented."
    
6. Establish a conclusion that draws together the data and rationale.
    
   "The additional features incorporated in the APU fuel shut off valve will provide a significant increase in safety to an existing design with satisfactory service experience. The applicant proposes that compliance with the latest standard would not materially increase the level of safety, and that compliance with FAR 25.1141 at Amendment 25-11 would provide an acceptable level of safety for the proposed product change."