2.3 Phase II - Curriculum Development

From: Transport Canada

Commercial & Business Aviation

2.3.1 Overview

Phase II is the development phase of the pilot training curriculum described in the Phase I, Application/Administration Document. There are five general stages in this developmental process of this phase, with associated documents:

  1. Job Task Analysis (JTA);
     
  2. Qualification Standards; 
     
  3. Instructional Systems Development (ISD) Methodology; 
     
  4. Curriculum Outlines; and 
     
  5. Implementation and Operations (I&O) Plan.

These steps are all inter-related. Each step builds on the previous step. The JTA supports the development of the Qualification Standards. ISD Methodology clarifies how JTA and Qualifications Standards will be used to support the development of the Curriculum Outlines. The end product is presented through the Implementation and Operations Plan which is used in Phases III and IV.

The applicant must establish, demonstrate and maintain a clear linkage between each of these steps. This linkage is provided by a systematic approach to the development of a complete instructional system. This section recommends a systematic approach and a methodology that is acceptable to Transport Canada, but innovation and practical application may result in equally acceptable variations.

NOTE: Instructors and evaluators shall be the first group to be transitioned from a traditional training program to an AQP.

  1. Rationale
     
    Under a traditional training program, Transport Canada evaluates the finished training curricula by comparing their contents, as described in each curriculum outline, to the requirements listed within the applicable CARs and the Air Carrier Inspector Manual (ACI). Under an AQP, Transport Canada evaluates a curriculum by monitoring its development, and by approving a series of required documents. These documents include the Job Task Analysis, the Qualification Standards and the ISD Methodology documents. This process affords Transport Canada greater insight into the rationale used by the applicant to develop each component of each curriculum.
     
  2. Analysis Based Approach
     
    Under an AQP, Transport Canada monitors both the process and the product. Instead of basing curricula on Transport Canada prescribed manoeuvres, procedures and knowledge items, AQP curricula are based on a detailed analysis of the requirements of each duty position at each organization. To receive Transport Canada authorization, AQP curricula must be accepted to be safer than, or at least as safe as, traditional training programs.

    The analysis-based approach allows each applicant the opportunity to develop air operator specific training programs. Consequently, AQP curricula will be more sensitive to differences in aircraft, operating conditions, emergency and abnormal contingencies, student skill levels and other operational variables, than are traditional programs.

2.3.2 Job Task Analysis (JTA)

In order to understand task analysis as it applies to AQP it is necessary to build upon a few definitions:

  1. Job
     
    A job is the summation of functions performed by an individual at work.
     
  2. Function
     
    One of the major subdivisions of work activities performed by one individual. One or more functions constitute a job. Examples of functions applicable to AQP would be a phases of flight such as “Take-off”, “Climb”, “Cruise”, etc.
     
  3. Task
     
    A task is a unit of work within a function having identifiable beginning and ending points which results in a measurable product. An example of a task applicable to AQP would be to “perform a normal take-off”.
     
  4. Sub-task
     
    Specific separate step or activity required in the accomplishment of a task. An example of a sub-task applicable to AQP would be to “perform rotation and lift-off”.
      
  5. Element 
     
    A further component of training analysis necessary in the accomplishment of a sub-task. An example of an element would be to “rotate the aircraft at VR to 12 degrees pitch”.

JTA is the method or procedure used to reduce a unit of work to its base components. The JTA document consists of a detailed, sequential listing of tasks, sub-tasks, and elements (if required) with the knowledge and skills (technical and CRM) that clearly define and completely describe the job. This includes the Knowledge, Skill, and Attitudes (KSA) characteristics that clearly define and completely describe the job. A JTA provides consideration for conditions surrounding the job both in the environment and in the equipment used. It establishes standards (parameters and tolerances) that provide safe and effective job accomplishment. It also identifies characteristics such as consequence of error, relative difficulty, frequency of occurrence in specific operations, and the time needed to accomplish the task. As a document the JTA has three parts: a Job Task List, a Task Factors Analysis, and a Learning Analysis.

2.3.3 Job Task List

The task analysis process begins with the development of a job task list that includes all of the major tasks performed by those who hold a particular duty position. For example, a pilot-in-command (PIC) job task list includes all major activities involved in operating an aircraft. These would include conducting ground operations, performing take-offs, etc. An instructor job task list includes all the major activities involved in teaching students. These would include preparing training materials, managing the learning environment, operating training equipment, etc.

  1. Structure 
     
    For complex jobs, it may be best to divide a job into several functions, which can be then divided into tasks. Each task is subsequently divided into sub-tasks. Finally, sub-tasks are then divided into elements. Each of these divisions is identified with a corresponding number code. Figure 2-2 shows an example of this relationship.

    For our discussion, we will consider the job of a PIC. We will divide this job into functions as defined by the various phases of flight. Each function (phase of flight) will be identified with a number code. For example, “2.0” will represent the second phase of flight, “Take-off”.

    Each function (phase of flight) can then divided into job tasks. For example, “2.0 Take-off”, can be subdivided into several tasks, each with a corresponding number code. Since all of the tasks in our example are derived from the same function, “2.0 Take-off”, they will all have number codes that begin with “2”. The first digit identifies the function and the second digit identifies the individual task. These will include:  “2.1 Perform Normal Take-off”, “2.2 Perform Instrument Take-off”, “2.3 Perform Engine Failure After V1 Take-off” and “2.4 Perform Rejected Take-off”.

    Each of these job tasks can then be further divided into sub-tasks. For example the task,“2.1 Perform Normal Take-off”, can be subdivided into numerous sub-tasks. For “2.1 Perform Normal Take-off”, the sub-tasks will include:  “2.1.1 Assess Performance and Environmental Factors”, “2.1.2 Perform Take-off Roll”, “2.1.3 Perform Rotation and Lift-off”, and so on. Here, the first two digits represent the task and the third digit defines the individual sub-task.

    Where necessary, these sub-tasks can be further subdivided into elements. For example, “2.1.3 Perform Rotation and Lift-off”, can be subdivided into “2.1.3.1 Rotate aircraft at VR to target pitch angle”, “2.1.3.2 Observe barometric/ADC altimeter increase [PF]”, “2.1.3.3 Call out positive rate [PNF]”, etc. In this case, the first three digits represent the sub-task, and the final digit defines the individual element.
     
  2. Hierarchy 
     
    By dividing tasks into sub-tasks and elements, each Job Task Listing produces a numbered hierarchy of job requirements for each duty position. These job requirements are essentially the graduation requirements for the courses that are developed from them. Through a series of additional analyses, these job requirements are translated into the training requirements of the various AQP curricula: Qualification and Continuing Qualification.

    The tasks are translated into Terminal Proficiency Objectives (TPOs) and the sub-tasks into Supporting Proficiency Objectives (SPOs). Elements are translated into Enabling Performance Objectives (EPOs). The hierarchical numbering system is retained as the basis of the audit trail that connects job requirements and performance with curriculum requirements and performance. Figure 2-2 illustrates this hierarchy and serves as an example of building a task listing.
     
  3. Rationale  
     
    Proficiency-based training systems always begin with the development of a detailed task listing. This means that required job proficiency is the basis for designing, developing, operating and maintaining the training system. Task 1.1.1 will be taught in lesson 1.1.1, topic 1.1.1, assessed using test item 1.1.1, and those test results used to validate that the individual/crew can perform task 1.1.1.  This audit trail links job performance to training performance throughout every component of the training curriculum.

Fig 2–2:  Sample Pilot Job Task Listing

Function Job Tasks
(TPO) 		Sub Tasks
(SPO) 		Elements
(EO) 		References
Source/Page
1.0: Ground
Operations 		      COM /2-12
2.0: Take-off       COM /3-1
  2.1: Perform Normal
Take-off 		     

COM /3-2, 3-3

    2.1.1: Assess Performance and
Environmental Factors 		  Airport Analysis
Chart/Specific City
    2.1.2: Perform Take-off Roll   COM/ 3-2, 3-3
    2.1.3 Perform Rotation and Lift-off 2.1.3.1: Call out V speeds [PNF] COM/3-3
      2.1.3.2: Rotate aircraft at VR to target pitch angle [PF] COM 3-3
      2.1.3.3: Observe barometric / ADC altimeter increase [PNF] COM 3-3
      2.1.3.4: Call out positive rate [PNF] COM 3-3
  2.2: Perform Instrument
Take-off 		     
  2.3: Perform Engine
Failure After V1
Take-off 		     
  2.4: Perform Rejected
Take-off 		     
3.0: Climb        
4.0: Cruise        
5.0: Descent        
6.0: Approach        
7.0: Landing        
8.0: After Landing        
9.0: Abnormal
Procedures 		       
10.0: Supplementary
Procedures 		       

2.3.4 Learning Analysis

The second part of the JTA is sometimes called competency analysis, skill analysis, Knowledge Skills and Attitudes (KSA) analysis, or hierarchical analysis. Here, those tasks, sub-tasks or elements that were selected for training as part of the task factors analysis are further analyzed into their more basic knowledge and skill level (attitudes are optional). This learning analysis will determine to a finer level of detail, exactly what should be learned, and the best approaches for teaching and testing what is to be learned. While the task factors analysis adds greater specificity to the performance and training requirements of the tasks, the learning analysis defines in greater detail exactly what should be taught and tested, and how it should be taught and tested, to assure that the students acquire those job performance requirements.

2.3.5 Job Task Analysis Document

The JTA document is the second of the six documents unique to AQP that must be maintained in a current status throughout the life of the program and must have an acceptable revision control methodology. Not all of the results of the JTA need to be reported to Transport Canada in the JTA document itself. While Transport Canada requires that the results of the learning analysis to be reported in the JTA document, it may be more convenient to report the results of some of the Task Factors Analysis in the Qualification Standards document.

2.3.6 Qualification Standards

A Qualification Standard is a job task proficiency objective (TPO or SPO) linked to an evaluation strategy. Qualification Standards define the requirements of mastery for the duty position. Demonstration that an individual has met the required standards will lead to certification. As a document the Qualification Standards is the single most important part of any AQP. It provides the complete proficiency baseline for all duty positions and serves as the basis of development for the Qualification Curriculum and Continuing Qualification Curriculum. The first step in the development of Qualification Standards is the development of proficiency objectives.

2.3.7 Proficiency Objectives

A Proficiency Objective is the result of applying a performance statement, conditions(s), and proficiency standard(s) to a task or sub-task. It is a statement describing the behaviour that the candidate must be able to demonstrate to successfully perform a task. For each duty position, there are two types of Proficiency Objectives, both of which are developed from the Job Task Analysis (JTA). These are: Terminal Proficiency Objectives (TPOs), which are developed from tasks; and Supporting Proficiency Objectives (SPOs), which are developed from sub-tasks.

Each proficiency objective statement has three parts:

  1. A performance statement that specifies precisely what behaviour must be exhibited.
     
  2. A condition statement identifying the operational and equipment contingencies as well as the environmental conditions under which the behaviour will be accomplished.
     
  3. A standard or criterion statement establishing the parameters and tolerances that define satisfactory performance.

Note: All document references used in defining the performance, conditions, and standards for each proficiency objective must be listed by title and chapter in the documentation of the proficiency objectives in the Qualification Standards Document.

  1. Terminal Proficiency Objectives (TPOs)
     
    TPOs are statements of performance, conditions and standards established at the task level. A complete set of TPOs will fully describe a particular job in the applicant's flight operation.  TPOs are classified by the air operator as either critical or currency items based on an operational assessment in the Task Analysis process. This classification will determine the frequency with which these tasks are evaluated during the Continuing Qualification Cycle. TPOs include the range of flight training equipment and the abnormal and emergency contingencies to be considered for training and evaluation.
     
  2. Supporting Proficiency Objectives (SPOs) 
     
    SPOs are statements of performance, conditions, and standards established at the sub-task level. SPOs are used to develop training and evaluation curriculum lessons, modules, and segments. SPOs include a list of flight training equipment and the abnormal and emergency contingencies to be considered for training and evaluation.
     
  3. Enabling Objectives (EOs)
     
    EOs are used to prepare individuals and crews for subsequent training in an operational cockpit environment. An applicant may identify a certain knowledge factor, cognitive skill, motor skill, or CRM factor as an Enabling Objective. These are normally not carried forward in the supporting performance objective statement and, therefore, are not normally found in the Qualification Standards Document. However, performance of a SPO would depend on a student acquiring the particular knowledge, skill, attitude or CRM factor.

Note: A learning objective (usually an EO), which doesn't have condition, can be demonstrated in a classroom or academic type setting. A performance objective (usually a terminal or supporting proficiency objective) must be demonstrated in an environment equivalent to the operational environment.

2.3.8 Task Factors Analysis:

This process rates TPOs and SPOs using the following factors: criticality, currency, need for training, applicable conditions, and applicable standards. The determination of criticality and currency determines ‘when’ and ‘how’ the objective is trained, validated or evaluated (see figure 2-3). To make this determination, the analyst answers a series of questions about each TPO and SPO to describe its performance requirements, both on the line and in the training setting.

Criticality is a determination of the relative impact of substandard task performance on overall safety.  It indicates an increased need for awareness, care, exactness, accuracy, or correctness during task performance. Critical tasks are proficiency objectives that are trained, validated or evaluated more frequently during an AQP evaluation period in a simulator or FTD.

A currency task is a proficiency objective for which individuals and/or crews maintain proficiency by repeated performance of the item in normal line, duty or work operations.  Most currency items are validated during online evaluations (OE) and may be sampled in the Continuing Qualification cycle.

Tasks that are determined to be critical and not current are trained, validated or evaluated each evaluation period. Tasks that are determined to be neither critical nor current are trained, validated or evaluated each Continuing Qualification Cycle.

Note:  Many flight crew job task SPOs do not fit the classic definition of a sub-task, a specific separate step or activity required in the accomplishment of a task. In recognition, non critical/non current SPOs under a common TPO that differ only in knowledge requirements may be trained, validated or evaluated in a simulator/FTD, during OEs, using oral, written or electronic exams, class room briefings or distributed material. However, it is recommended that these SPOs, as appropriate, be demonstrated in a simulator/FTD on a recurring cycle authorized by Transport Canada.

Transport Canada recommends that the applicant examine each task, sub-task, and element, as appropriate, for the following factors:

Primary factors to be considered:

  1. Statement of performance; 
     
  2. Environmental conditions affecting difficulty/success; 
     
  3. Performance standards (parameters with tolerances); 
     
  4. Abnormal and emergency procedure contingencies; 
     
  5. Student entry-level performance evaluated against proficiency objective; 
     
  6. Document references (title and section) governing or specifying the operation; 
     
  7. Consequence of error to safety; and 
     
  8. Relative difficulty.

Additional Factors:

  1. Equipment and system operation dependencies (if used for establishing learning sequences for curriculum development); 
     
  2. Criterion for success upon which performance standards are based. If new performance standards are created, this criterion should be established for each task and sub-tasks (e.g., the tracking standards for VHF omni-directional radio (VOR) approaches are based on navigation requirements). The navigation requirements are the criteria for success. Success criteria are developed in those cases where current standards are missing or thought to be inadequate.

Fig 2-3: AQP Continuing Qualification Critical/Currency Chart

Training
Priority 		Critical
(Y/N) 		Currency
(Y/N) 		Terminal Proficiency Objective/Supporting Proficiency Objective
1 YES NO Train, validate and evaluate each Evaluation Period. (e.g., Engine failure after take-off, CAT I and CAT II Approaches, Wind shear, Engine-out precision approach)
2 NO NO Train, validate and evaluate each Continuing Qualification cycle. (e.g., pilot operation of passenger doors, unpressurized flight procedures, alternate landing gear procedures)
3 YES YES

Sample at First-Look (FL)/Manoeuvres Validation (MV)/Line Operational Evaluation (LOE) and/or Online Evaluation (OE) for each Evaluation Period. (e.g., Perform high altitude airport operations, perform adverse weather (icing) procedures)
NOTE: For AQP, the sample size has to be large enough to provide reasonable assurance that the population is remaining proficient.

4 NO YES Sample at First-Look/Manoeuvres Validation (MV)/Line Operational Evaluation (LOE) and/or Online Evaluation (OE) each Continuing Qualification cycle (e.g., Perform Normal Landing, Perform Cruise Operations)
NOTE: For AQP, the sample size has to be large enough to provide reasonable assurance that the population is remaining proficient.

 

Training
Priority 		Critical Current Month Sample
1 Yes No 12 No
2 No No 24 No
3 Yes Yes 12 Yes
4 No Yes 24 Yes

 

NOTE:  The sample size must be large enough to provide reasonable assurance that the population is remaining proficient.

NOTE:  The Qualification Standards Document for instructors/evaluators does not need to include conditions or a criticality/currency analysis.

2.3.9 Evaluation Strategy

The Qualification Standards document will identify the curriculum (Qualification and/or Continuing Qualification) in which specific proficiency objectives will be met. The applicant should consider student entry level in determining this allocation. All TPOs must be included in a Qualification Curriculum (Qualification Course) regardless of entry-level analysis. For SPOs, an entry-level analysis determines what objectives will be taught under each curriculum. All objectives should also be covered in Continuing Qualification Curriculum test and evaluation strategies.

2.3.10 Consolidation of Objectives 

In the Qualification Standards document, qualification standards are developed at the task and sub-task level only and at no lower level. Tasks become TPOs and sub-tasks become SPOs by combining performance statements, conditions and standards. TPOs and SPOs having common knowledge, skill, attitude, and/or CRM factors may be consolidated to avoid duplication. The consolidated tasks are translated into TPOs, and a terminal level qualification standard is developed for each one. The consolidated sub-tasks are translated into SPO, and a supporting level qualification standard is developed for each one of those as well. An example of consolidation would be non-precision approaches. VOR and Non-Directional Beacon (NDB) approaches may be consolidated as a single proficiency objective if the performance statement and standards are the same.

2.3.11 Conditions, Contingencies, and Media 

The Qualification Standards will include a listing of relevant operational and environmental factors along with equipment contingencies to be addressed in training. They will also identify the media that will be employed to test, validate, or evaluate the specific training objective. There is a difference between the Qualification Standard for a Qualification Curriculum and one for a Continuing Qualification Curriculum. The TPOs and SPOs in a Qualification Standard for a Qualification Curriculum must identify the specific set of conditions and contingencies to be employed in training and evaluating a task (usually with an asterisk). In addition, the media description will describe the specific media in which the task will receive final evaluation. In contrast, in a Continuing Qualification curriculum, the TPOs and SPOs may identify a selectable menu of conditions and contingencies to be employed in training or evaluation, and a media range that will specify the minimum media level in which the task may be trained, validated, or evaluated.

2.3.12 Qualification and Continuing Qualification Curricula 

The Qualification Standards for both Qualification and Continuing Qualification Curricula can be combined into one document as long as the differences in conditions and media are addressed and the tasks are annotated for applicable curricula. However, if applicants find it more convenient to submit separate Qualification Standards documents for these curricula (e.g., because of differences in conditions, contingencies, and media between the Qualification and Continuing Qualification curricula), they may do so. In addition, some Qualification Standards may be “aircraft generic” in that they may apply to more than one aircraft type. For ease of use and definition of “fleet common” curricula, applicants may also find it convenient to specify these Qualification Standards separately from those Qualification Standards that apply to only one aircraft fleet type.

2.3.13 Qualification Standards Document 

The Qualification Standards document is the third of the six documents unique to AQP that is maintained in a current status throughout the life of the program and must have a revision control methodology. It is the central AQP document because it provides the regulatory basis for all deviations from current regulations, and identifies the basic training and evaluation requirements. The Qualification Standards document has four parts: the Prologue, a Regulatory Comparison, the Evaluation/Remediation Methodology, and the specific TPO or SPO Qualification Standard information.

  1. Prologue
     
    The Qualification Standards document requires an introductory section that explains the methodology, format, and terminology of the standards to the broad range of company and Transport Canada personnel who will need an in-depth understanding of the document to perform their job functions.
     
  2. Regulatory Requirements Comparison
     
    The Qualification Standards document must also include regulatory comparison information. The comparison must indicate specifically the requirements of Parts IV, VI and VII of the CARs as applicable that would be replaced by an AQP curriculum. Once approved, the Qualification Standards document becomes the regulatory foundation for the operator’s AQP.

    Note:  The purpose of this regulatory comparison is not to justify differences from current traditional practices, but merely to document them once they have been justified. It is the quality control processes of AQP that form the basis for establishing an equivalent level of safety.
     
  3. Testing/Validation/Evaluation & Remediation Methodology 
     
    Before implementing an AQP curriculum, the applicant must decide how, when, where, and who will assess a student’s proficiency on each terminal and supporting objective. The guidelines in figure 2-3 may be used to translate TPOs’ and SPOs’ criticality and currency ratings into testing strategies for the Continuing Qualification Cycle. This testing strategy may include a discussion describing how similar SPOs will be addressed. For example, the TPO non-precision approach may have many SPOs, such as VOR, Non Directional Beacon (NDB), localizer (LOC), etc. Depending on the criticality/currency rating, all of these approaches do not have to be evaluated each evaluation period. This section of the document is where the applicant can describe how these approaches will be alternated/sampled over multiple evaluation periods or Continuing Qualification Cycles.

    The testing/validation/evaluation methodology also addresses the applicant’s approach to documenting the different qualifications of the instructor and evaluator, in terms of who will conduct the testing, validation, LOE, and OEs. In this section, the applicant can describe the level of device appropriate for conducting the evaluation. It also identifies the point in the curriculum when the testing/validation/evaluation methodology will be applied, and identifies what constitutes a failure and/or unsatisfactory performance. In addition, the applicant must specify the strategy for remedying unsatisfactory performance.  
     
    1. General Strategies:  For each terminal and supporting proficiency objective, the applicant will designate the testing/validation/evaluation strategy. These strategies could include, and are not limited to: 
       
      1. Train to Proficiency; 
         
      2. Systems/Knowledge Validation (SKV); 
         
      3. Procedures Validation (PV); 
         
      4. Manœuvres Validation (MV); 
         
      5. Line Operational Evaluation (LOE); 
         
      6. Operating Experience; 
         
      7. Online Evaluation (OE).
         
    2. Training Media:  The applicant also will designate the level of training devices, simulators, or aircraft to be used to evaluate the proficiency objective.
       

    3. Rating Scale/Scoring:  The applicant will identify the rating methodology that will be used to grade the performance of the proficiency objectives against the Qualification Standards. Typically, the measurement codes associated with performance events are ratings, repeat counts, and reason codes or skill categories.

      Ratings are used to define different quality levels of performance. Rating codes usually are air operator specific and Transport Canada requires that something more sensitive to performance differences than a binary code is used (i.e. some rating method that provides more performance differentiation than pass/fail for individual items being evaluated.

      Note:  For Manoeuvres Validations (MV), Line Operational Evaluation (LOE) and Online Evaluation (OE) a minimum four point grading scale shall be used.

      Each carrier should ensure that the grades established on the rating scale are clearly defined, meaningful to the instructor and evaluator, and easily used for performance assessment. Although consistency among fleets and across different types of evaluations - Online Evaluation (OE), Manoeuvres Validation (MV), and Line Operational Evaluation (LOE) - is important and generally desirable, rating scales may be slightly different when used for different purposes such as training vs. evaluation. Figure 2-4 provides an example of a rating scale that discriminates among performance levels.

      Fig 2-4:  RATING SCALE EXAMPLE - First-Look, Manoeuvres Validation, Line Operational Evaluation, or Online Evaluation

      GRADE CRITERIA
      1 Unsatisfactory

      Major deviations from the prescribed qualification standards occur that are not recognized or corrected in a timely manner commensurate with safety. Individual or crew performance could result in hull loss or loss of life. CRM skills are not effective.

      2 Satisfactory

      Deviations from the prescribed qualification standards occur and are corrected in a timely manner. Individual or crew performance is safe but would be unsatisfactory if diminished by any amount. CRM skills are not completely effective.

      3 Standard

      No deviations occur from the prescribed qualification standards. Individual or crew performance meets expectations. CRM skills are clearly effective.

      4 Excellent

      Performance remains well within the prescribed qualification standards. Individual or crew performance, management and CRM skills are exemplary.

      NOTE:  This example should not be taken as limiting possible intervals to a four-point scale. With appropriate scale construction and instructor and evaluator training, carriers may elect to define other scales that maximize the quality (sensitivity, reliability, validity) of the collected data.

    4. Remediation Strategy:  This section of the Qualification Standards document should describe the methodology that will be used to re-mediate unsuccessful testing, validation, or evaluation sessions. This remediation strategy must detail when and what may be repeated and whether or not additional training is warranted. Remediation strategy must also specify when no more training will be offered to the individual and the resulting actions such as “Referred to Committee,” returned to previous position, etc. The AQP Evaluator Manual includes expanded discussions on validation, evaluation, and remediation in the Qualification and Continuing Qualification Curricula. This strategy may be presented in narrative text or flowchart format.
       
    5. Special Tracking:  This is a program for monitoring the proficiency of an individual at scheduled intervals. It is applied to individuals who have failed to demonstrate proficiency during an evaluation event (e.g., LOE). There may be other criteria that the air operator may use to place an individual on special tracking. These could include continuing difficulty in completing the Manoeuvres Validation (MV) or a failure of an OE. This section should discuss the following: 
       
      1. The situation(s) that requires an individual to be placed in special tracking 
         
      2. The strategy to be used. 
         
      3. When special tracking is no longer required. 
         
  4. Qualification Standard Information:
     
    The information contained in the specific qualification standard is the basis for determining proficiency and evaluation criteria.  The qualification standard is constructed by applying a performance statement, conditions, and standards to a task or sub-task, thereby creating a TPO or an SPO. 

    Although each air operator will determine the format and content of its Qualification Standards, component fields have developed out of practice and are illustrated in Figure 2-2.  In this example, the phase of operations is 2.0: Takeoff.  The TPO is 2.1: Perform Normal Takeoff.  The SPOs are 2.1.1: Assess Performance and Environmental Factors, 2.1.2: Perform Takeoff Roll and 2.1.3: Perform Rotation and Lift-off.  EPOs are 2.1.3.1: Call out V Speeds, 2.1.3.2: Rotate Aircraft at VR to Target Pitch Angle, 2.1.3.3: Observe Barometric/ADC Altimeter Increase and 2.1.3.4: Call Out Positive Rate.  

    Variation in the format of a given air operator’s Qualification Standards is permissible if all of the categories of information in the example are addressed.
     
    1. A header identifies the airline and the document. 
       
    2. Page revision control dates and revision numbers. 
       
    3. Consecutive page numbers. 
       
    4. Phase of Operations.  Number and title from task listing. 
       
    5. Qualification Standard Title.  Either TPO(s) or SPO(s). 
       
    6. Hierarchical Task or Sub-task.  Identifier and title from task listing. 
       
    7. Crew Duty Position(s).  This identifies which crew member(s) will be evaluated performing the task. 
       
      1. Pilot-in-Command = PIC 
         
      2. Second-in-Command = SIC 
         
      3. Flight Engineer = FE 
         
      4. Captain, First Officer, Second Officer 
         
    8. Criticality/Currency Rating.  From the task factors analysis of the job task listing.  This may be the first place that the task factors analysis is tied to the tasks.

      NOTE:  The Qualification Standards document for instructors and evaluators does not need to include conditions or a criticality/currency analysis. 
        
    9. Curriculum.  This field identifies the curriculum(s) in which the task will be trained and evaluated. 
       
    10. Evaluation Strategy.  The evaluation point for a particular qualification standard (e.g.,  (1) train to proficiency;  (2) systems validation;  (3) Procedures Validation (PV);  (4) Manoeuvres Validation (MV); (5) LOE; or  (6) OE). 
       
    11. Media.  The specific media in which training and/or evaluation will be conducted. For qualification, the media is the lowest media used for final evaluation.  For continuing qualification, the media includes the range of media used for training, validation, and evaluation. 
       
    12. Performance Statement.  An expanded statement of expected behaviour that, when executed, will complete the work required for a specific portion of a job.  A performance statement specifies precisely what behaviour must be exhibited, and may include the knowledge and skill issues that comprise the EO supporting that performance. 
       
    13. Operational and Environmental Conditions. Conditions describe the range of circumstances under which student performance will be measured and evaluated. Conditions include the operational environment (unserviceable navigational aid (NAVAID), different aircraft weight, passengers not seated, aircraft configuration, etc.) and natural environment (ceiling, visibility, wind, turbulence, etc.). The qualification standard should: (1) indicate those specific conditions to be trained and tested as part of the Qualification Curriculum; and (2) provide a more exhaustive listing of conditions over which crew members will be trained and tested during the course of successive Continuing Qualification Cycles. 
       
    14. Contingencies.  Contingencies include abnormal situations, MEL/CDL, and emergencies. The qualification standard should (1) indicate those specific contingencies to be trained and tested as part of the Qualification Curriculum; and (2) provide a more exhaustive listing of contingencies over which flight crewmembers will be trained and tested during the course of successive Continuing Qualification Cycles. 
       
    15. Manœuvre Standards.  Observable, measurable parameters of performance with tolerances (e.g., course deviation degrees (+ or -)). Standards include manoeuvres, procedures, and CRM considerations.
        
    16. References.  Identify the primary references from which performance statements and associated standards were derived. Cite documents by title and where applicable, chapter or section.  Page numbers are not required.

2.3.14  Instructional Systems Development (ISD) Methodology Document

This is the fourth of the six documents unique to AQP and is maintained throughout the life of the program.  It must have an acceptable revision control methodology. Applicants with established curriculum development guidelines may submit these for consideration. Others should describe a systematic approach for developing a proficiency-based training system that is organized around the teaching and testing of terminal, supporting and enabling proficiency objectives. The methodology identifies the rationale, justification, and subsequent documentation to be used in the applicant’s curriculum development process. The instructional systems development methodology document describes the approach to be used by applicant airlines to develop and maintain all AQP curricula. 

ISD Methodology document should be finalized before constructing curricula for each duty position. It applies to pilot, instructor, and evaluator programs. This document is divided into two sections. The first section, Curriculum Development Procedures, describes the applicant’s approach for using the JTA and Qualification Standards as baseline documents to construct their general training curricula across all AQP courses. The second section, Line Operational Simulation (LOS) Methodology, describes the approach for developing LOS scenarios.

  1. Curricula Development Process/Methodology  
     
    Applicants should describe the process they will use to build their curricula based on the JTA, Qualification Standards and proficiency objectives they develop for each duty position.  This document should discuss how: 
     
    1. TPOs and SPOs are allocated to curricula. 
       
    2. Learning and evaluation activities are developed to support these objectives.
    3. Instructional media and methods are assigned to objectives. 
       
    4. Objectives are clustered and sequenced into lessons, modules, segments, and curricula (see figure 2-5 and figure 2-6 for examples). 
       
    5. An audit trail (hierarchical numbering or a matrix) will be maintained to link job tasks, proficiency objectives, lesson activities/content, and test items.
       

    The resulting curricula are translated into a course footprint and are documented in each curriculum outline.  These curricula are expanded in more detail in the student and instructor syllabi and in individual lessons and tests. 

    Fig 2-5:  Curriculum Development

     


     
  2. Develop Line Operational Simulation (LOS) Development Methodology
     
    Transport Canada will approve the methodology by which LOS scenarios (i.e., SPOT, LOFT and LOE) are generated and will review and accept individual LOS scenarios. The LOS approach divides the typical LOFT scenario into a series of relatively independent segments, called event sets. A typical scenario might have for example eight event sets, relating or not to phases of flight (e.g., pre-departure, take-off, climb, cruise, descent, approach, landing, and taxi-in). Each event set consists of a series of training or evaluation events (graded events/tasks), which include both technical and CRM activities. An LOE shall contain a minimum of eight to a maximum of eleven event sets.

    The above technique enables scenarios to be constructed in a building block approach, assuring that each event set is carefully scripted, sequenced and considered in relation to the other event sets in any given scenario.

    The criteria used by Transport Canada when validating LOS scripts are located in Chapter 3 – AQP Authorization Process and Documentation.

    Applicants with established curriculum development guidelines may submit these for consideration. Others should describe a systematic approach for developing a proficiency-based training system that is organized around the teaching and testing of TPOs, SPOs and EOs.

    This document should discuss the following: 
     
    1. Proficiency objectives are developed and organized into curricula; 
       
    2. Learning and evaluation activities are developed to support these objectives; and 
       
    3. An audit trail will be maintained to link objectives, lesson activities/content, and test items.

2.3.15  Curriculum Outline Document

This is the fifth of the six documents unique to AQP that must be maintained in a current status throughout the life of the program. It must have an acceptable revision control methodology. The curriculum outline provides the footprint, which is a high level description of the training and evaluation activities and planned time allotment for each day in the curriculum. 

The Curriculum Outline Document contains a listing of course material divided into segments. Typical segments would be Ground School or Flight Training. These segments are then divided into modules. For example, within the Ground School segment, there could be several modules including: Aircraft Systems, SOPs, and Long Range Navigation. The Flight Training Segment would typically include FTD, Fixed Base Simulator (FBS) and Full Flight Simulator (FFS) modules.  These modules are further divided into lessons. The first lesson in the FTD module might focus on Pre-Flight Operations and Normal Checklists. Finally, lessons are divided into elements or topics.  The FTD module on Pre-Flight Operations and Normal Checklists would cover elements or topics such as Flight Compartment Inspection, Flows, Before Starting Engines Checklist, etc. Figure 2-6 provides an example of a curriculum outline showing portions of ground training and flight training segments which have been divided into modules, lessons and elements (topics).

Curriculum outlines are developed and submitted with the understanding that application of the course material may require some flexibility regarding the actual day on which each activity is accomplished. Each part of the curriculum outline must clearly indicate the subject matter to be taught and correspond directly to the hierarchical system of the task analysis. While the curriculum outline document need only go down to the level of the element under each lesson title, the applicant will be required to show the terminal, supporting and EOs associated with each lesson. Refer to Figure 2-5 for an illustration of this association. This is part of the necessary audit trail that links the job requirements (contained in the JTA) to the training requirements (contained in the Qualification Standards) to the training activities (contained in the Curriculum Outline). 

A curriculum outline provides the basis for the curriculum footprint, which is a high level graphical overview of the curriculum content depicting the training and evaluation activities and the planned duration of each day of the curriculum (see figure 2-5). The curriculum outline document should reference the results of the student entry analysis, if one was conducted, and will include a curriculum footprint. 

Fig 2-6:  Sample AQP Curriculum Outline

B 737 Qualification Curriculum Outline 

Segment: Ground Training

Module:  Introduction to Aircraft

Lesson #: Aircraft Overview 9.1.4
Topic:  Fuselage 9.1.4.1
Topic:  Wings 9.1.4.2
Topic:  Flight Controls 9.1.4.4, 9.1.4.5, 9.1.4.6
Topic:  Landing Gear 9.1.4.7
Topic:  Powerplant 9.1.4.3
Topic:  Fuel System 9.1.4.8
Topic:  Hydraulic System 9.1.4.9
Topic:  Electrical System 9.1.4.10, 9.1.4.10.1, 9.1.4.10.2, 9.1.4.10.3

Lesson #:  Aircraft Lighting 9.1.9
Topic:  Exterior Lighting 9.1.9.1
Topic:  Cockpit Lighting 9.1.9.2
Topic:  Cabin Signs and Lights 9.1.9.9, 9.1.9.4, 9.1.9.6
Topic:  Lighting Power Sources 9.1.9.5
Topic:  Emergency Lighting 9.1.9.5, 9.1.9.5.1, 9.1.4.10.2
 

B 737 Qualification Curriculum Outline

Segment: Flight Training (FT)

Module: Flight Training Device (FTD)

Lesson: FTD  #1 Pre-Flight Normal Checklists
Topic: Flight Compartment Safety Inspection 1.2.5.1
Topic: Flows 1.2.7.1
Topic: Acceptance Checklist 1.2.8.1
Topic: Before Starting Engines Checklist 1.2.11.1
Topic: Normal APU First Engine Start 1.9.1.1
Topic: Normal APU Second Engine Start 1.9.1.4
Topic: Pushback 1.9.6.1
Topic: Before Taxi Checklist 1.4.1.1
Topic: Normal Taxi 1.4.2.1
Topic: Before Take-off Checklist 1.4.9.1
Topic: Line-Up Checklist 1.4.4.1

  1. Entry Level Analysis 
     
    As an option, the applicant may develop and document a student entry-level performance analysis for TPO and SPO. This analysis compares the Knowledge, Skills and Attitudes (KSAs) of the student population against the TPOs and SPOs in the JTA in order to tailor the instruction to the student. A four-point performance difference rating scale is suggested (Figure 2-7). Highly skilled instructors who are familiar with the experience and background of the student population and knowledgeable of the TPOs and SPOs should make the rating. This analysis provides guidance to determine efficient teaching strategies for the Qualification Curriculum.  This analysis can also identify where training is not needed, where basic “enabling” skills must be taught, and what number of trials is expected for an applicant to reach TPO standards. More than one population group may be used in conducting the student entry analysis for a single duty position. The results of such an analysis may be used, for example, to justify alternative curriculum tracks or modules targeted at expected differences in entry-level background, and in order to achieve the most efficient use of training resources.

    Fig 2-7:  EXAMPLE - Performance Difference Rating Scale
     
    Performance Discrimination Scale Scale Description
    4 Meets or exceeds the required performance.
    3 Can accomplish tasks with minor errors or omissions. May take longer than expected or allowed.
    2 Cannot accomplish tasks. Does not demonstrate basic background skills and knowledge.
    1 Does not demonstrate basic background experience, skills or knowledge. Unfamiliar with simplest elements of a task.
     
  2. Trainee and Instructor/Evaluator’s Curricula
     
    The instructor and evaluator curricula must be developed in the same systematic manner as those developed for the duty positions with which they are associated. This requires the development of separate JTA, Qualification Standards, Curriculum Outlines and other documents for these instructor and evaluator positions. These instructor and evaluator curricula may share common modules or lessons.  Instructors and evaluators also require separate Qualification and Continuing Qualification curricula. 
     
  3. Link Qualification Standards to Curricula
     
    Both the Qualification and Continuing Qualification curricula for a given duty position are derived from the same set of Qualification Standards. The link between Qualification Standards and a curriculum is the Proficiency Objective (TPO or SPO). As a rule of thumb, Qualification Standards developed from TPOs focus more on the higher-level evaluation activities than Qualification Standards developed from SPOs. Qualification Standards developed from SPOs focus more on the lower level learning activities that may be validated and represent the components of those higher level standards. Therefore, a Qualification Curriculum will focus equally on Qualification Standards developed from TPOs and SPOs, while the Continuing Qualification Curriculum will focus more heavily on the Qualification Standards developed from TPOs. 

2.3.16  Implementation and Operations Plan (I&O) Document

This document, like the other five unique AQP documents, must be maintained in a current status throughout the life of the program.  It must have an acceptable revision control methodology and must be updated as necessary to accurately reflect the status of the AQP applicant’s plan for implementing and operating each of the AQP curricula. This document is a milestone schedule detailing the transition to an AQP for flight crewmembers, instructors and evaluators and a blueprint describing provisions for maintenance, administration, data management, and continuing quality control of curricula. The Implementation and Operations Plan Document can be sectioned into two major parts. 

The first part of the Implementation and Operations Plan spells out how the operator proposes to implement the AQP.  Included in this proposal is the schedule for Phase III, training and evaluation to include instructor and evaluator training and small group try-outs. It should also include provisions for evaluating the effectiveness of performance measurement tools, and provisions for evaluating facilities, courseware, and equipment before starting the plans for the small group try-outs. 

The second part of the Implementation and Operations Plan contains an explanation of how the Operator intends to operate the AQP in phases IV and V. Included in this section are strategies for maintaining the program, crew pairing policy, First-Look administration, instructor and evaluator requirements. The operations plan should also include the data management plan, a statement of understanding addressing the collection and analysis of performance/proficiency data, a description of the PPDB, the data management collection process, and the Transport Canada data submission, analysis, and reporting requirements. 

  1. Implementation
     
    This section provides the schedule for evaluating curricula in the small group try-out, as well as instructor and evaluator training, and the strategies for evaluating facilities, courseware, and equipment.  It also includes the proposal for evaluating instructors, evaluators, and performance measurement tools such as the rating scale and grade sheets. The schedule for initiating phase III should correlate with the MATS as submitted in the application. This part of the I&O Plan is updated each time a new curriculum is added to the AQP and a small group try-out is planned. 

    NOTE:  If the air operator is requesting no-jeopardy credit for the students in the small group try-out, it must be indicated in the I&O Plan and requested in writing to the POI. 
     
  2. Operations  

    This section of the I&O Plan describes the guidance and policies that will be used to provide for AQP maintenance, First-Look administration, crew pairing policy, instructor and evaluator requirements and data management. Once established, the operations section remains reasonably stable and does not necessarily change with the addition of a new curriculum. 
     
    1. AQP Maintenance Strategy for Phases IV and V.  This section describes quality control procedures (i.e., plans for acquiring and measuring data for tracking curricula, students, instructors, and evaluator performance).  It also includes the strategy employed for curriculum maintenance and update. Maintenance includes the methodology for maintaining control of the AQP. It includes approval documents, maintaining curriculum currency, upgrading equipment, monitoring and responding to demographic changes, and for using training/evaluation feedback from the PPDB and other forms of surveillance to maintain and improve the AQP.  
       
    2. First-Look Activity Administration.  First-Look performance items are graded procedures/manoeuvres performed for the first time since the previous training cycle. The First-Look grades are analyzed to determine trends of degraded proficiency due to numerous factors, including the length of the training interval. To maintain the validity of the performance proficiency data obtained from the performance of the First-Look activities, this section must describe the strategy employed for those activities. This strategy must state that information or techniques that would unduly interfere with the First-Look performance validity will not be briefed before the first execution of these items.  In addition, this strategy should discuss how First-Look manoeuvres are selected and how they will be administered. 
        
      Note:  Phase V of AQP may allow for modified training and evaluation cycles.  First-Look requirements are an integral part of an AQP’s development process and are useful to air operators who wish to modify their training and evaluation cycles from current regulatory requirements.  In such cases, the air operator must have previously implemented First-Look Manoeuvres and collected sufficient data through one full Continuing Qualification Cycle in order to establish a base line by which to measure the effect of modified intervals. 
       
    3. Identify Line Operational Simulation (LOS) Crew Scheduling and Pairing Strategy.  A basic requirement of AQP is to train and evaluate flight crewmembers in a crew configuration identical to line operations. Therefore, LOS should take place in a line operational environment with a complete crew. A complete crew concept allows crewmembers to use their full resources and creativity to create a complete learning experience. A complete crew consists of flight crewmembers for each seat position who are line-qualified or in qualification training for their respective seat positions.

      In AQP, line crewmembers must be scheduled and paired together, as much as practical, in a standard crew configuration (e.g., line Captain with line First Officer). Transport Canada recognizes that circumstances will occur where the initial composition of the schedule cannot be maintained. Hiring requirements, illness, high First Officer to Captain ratios, or failure of a crewmember to progress, are all situations that would necessitate providing a seat substitute to complete the training. This section must address the decision rules that will apply to the selection of seat substitutes for those circumstances. In all cases, the seat substitute must be task-familiar with the duty position. Task familiar describes a flight crewmember who is familiar with and can satisfactorily accomplish the duties of a particular crew duty position, though not qualified for that duty position. For example, an SIC candidate who performs the duties of the PIC during simulator training.

      The following paragraphs provide examples of substitution rules that could be applied when addressing selection of seat substitutes for LOFT, SPOT and LOE. Transport Canada strongly encourages air operators to develop such decision rules for all types of AQP training and validation/evaluation events.

      Recurrent LOFT stresses scheduling of a complete crew who should be line-qualified. The use of substitutes is discouraged, and substitution should be rare. When the composition of the scheduled line-qualified crew cannot be maintained, the operator may use substitutions based on the guidelines in the table below. However, the air operator will attempt, first, to substitute with another line-qualified flight crewmember. This table should be used only as a last resort to prevent interruption of scheduled training.

      Recurrent LOFT Substitution Table

        Pilot-in-Command Position Second-in-Command Position

      Flight Engineer Position

      1. PIC1 SIC1 FE1
      2. SIC2 PIC1 FE Instructor3
      3. Instructor3 Instructor3  

      1 Includes those who are either line-qualified, or in training, and are line and task familiar with the position in which they are substituting.
      2 An SIC may be substituted for this position if the pilot has received a type rating in the aircraft the simulator replicates.
      3 An instructor (aircraft or simulator) as provided for under part VII of the CARs. The instructor should not have previous knowledge of the scenario; however, when this is unavoidable, the instructor should not use that knowledge to influence or direct the scenario.

      NOTE: The instructor conducting the LOS session will not act as a substitute crewmember.

      Qualification LOFT requires a complete crew complement. It is preferable to schedule a flight crewmember who is qualifying with other flight crewmembers who are fully line-qualified. As a minimum, LOFT flight crewmembers will be task familiar with their assigned duty position. The use of substitutes is highly discouraged and substitution should be implemented rarely. When the composition of the scheduled crew cannot be maintained, the operator may substitute flight crewmembers using the table below.

      Qualification LOFT Substitution Table

        Pilot-in-Command Position Second-in-Command Position

      Flight Engineer Position

      1. PIC1 SIC1 FE1
      2. SIC1 PIC1 FE Instructor2
      3. Instructor2 Instructor2  

       
      1 Includes those who are either line-qualified, or in training, and are line and task familiar
      with the position in which they are substituting.
      2 An instructor (aircraft or simulator) as provided for under Part VII of the CARs. The instructor should not have previous knowledge of the scenario; however, when this is unavoidable, the instructor should not use that knowledge to influence or direct the scenario.

      NOTE:  The instructor conducting the LOS session will not act as a substitute crewmember.

      Crew composition for SPOT may include the use of a complete or partial crew, depending upon the training objectives.  The use of crew substitutes in SPOT depends upon the type of training being provided.

      With respect to an LOE, a complete crew complement should be scheduled and maintained. Flight crewmember substitution is highly discouraged. If crew substitutions are necessary, the substitute flight crewmember will be either another line-qualified flight crewmember or a task familiar flight crewmember in a training status comparable to the person being evaluated. Evaluators conducting the LOE may not serve as a substitute flight crewmember. The LOE substitution table/matrix will be part of the air operator’s approved AQP documentation.

    4. Instructor and evaluator Requirements.  This part will address the specific job functions, training, validation, or evaluation that instructors and evaluators are authorized to perform.  It will identify the title of each position and describe the training that each receives in order to perform the job function associated with that event. 
       
    5. Data Plan.  Before an applicant can proceed with data collection and analysis, it must establish the intended purpose and method for the collection, entry, reporting, and analysis of AQP training/evaluation data for each curriculum. The plan must be thorough and accurately reflect the airline’s PPDB system. The AQP applicant must also acknowledge its responsibility to collect and analyze more data than required for submission to Transport Canada in order to adequately identify performance trends and requisite changes to factors that impact the performance. 
        1. Data Collection.  This part of the Data Plan should address the methods used to collect performance/proficiency data for all curricula. These methods will include the rationale for employing the method as well as providing the data input medium (e.g., grade sheets, computer-input screens, etc.) as examples that exemplify the data acquisition rationale.  In addition, the data collection method should explain data input quality control, security, and usability. 
           
        2. Data Base and User Interface Management.  This part of the Data Plan should explain the means and strategy the air operator intends to employ to enter, access and utilize the AQP performance/proficiency data that is collected. Included in this explanation should be: 
           
          • The type of software data management system employed (e.g., relational database, spreadsheet, etc.); 
             
          • The organization of the information in the electronic medium (e.g., database definition, database table relationships, spreadsheet description, etc.); 
             
          • A description of the user interface to this data management system.
        3. Data Analysis. This part of the Data Plan should discuss the type of analysis that will be employed to facilitate the AQP performance information needs of the air operator and Transport Canada. This discussion of the data analysis must address how each type of AQP data will be analyzed, including training and evaluation feedback as part of determining the effectiveness of the program. This discussion should be used as a preamble to the annual AQP report. 
           
        4. Data Reporting.  This part of the Data Plan must discuss the AQP data reporting requirements that it must meet for Transport Canada inspection and audit purposes to include format and frequency. In addition, it should discuss the type of data reporting it will employ, to include report example types (e.g., tabular reports, graphs), frequency, and the air operator personnel for whom the reports are intended.

2.3.17 Approval

Transport Canada’s authorization to proceed with the implementation phase, following the approval of all documents required under Phase II, marks the end of Phase II and the beginning of Phase III.

Report a problem with this page

Date modified: