Non-Technical Skills in Proficiency Checks: How to Assess What Cannot Be Timed

Salle de débriefing avec deux chaises face à face, horloge murale simple et feuille vierge sur la table

Two examiners witness the same go-around in a simulator. One rates the workload management at a higher level, the other finds it barely acceptable. Neither is wrong: they interpret the grid differently. As long as the assessment measured tolerances — an axis held, an altitude, a time — the gap between raters remained marginal. Now that the score assesses communication, decision-making, or leadership, the grading method has become a standalone safety issue: a poorly calibrated grid produces indefensible scores and control decisions that won’t hold up to scrutiny.

Grade behavior, never personality

Early generations of crew competence evaluations rated impressions: “good team spirit,” “natural leadership.” A double dead end. Impressions aren’t contestable — you don’t debrief a personality trait — and they’re not actionable: they don’t say what to train. Current frameworks, derived from European work on non-technical skills assessment and later standardized by ICAO’s competence framework, represent a complete reversal: you don’t score a quality, you score observable behaviors, described by action verbs. “Announces the deviation as soon as it’s detected.” “Distributes tasks before the high-workload point.” “Verbalizes intent before acting on automation.”

A well-written indicator has three properties. It’s observable: no mental states, no assumed intentions. It’s factual: described without adjectives, so it can be recorded verbatim in a session report. It’s datable: tied to a specific moment, allowing it to be retrieved — on video, in simulator data, or in the other observer’s notes. The entire follow-up mechanism — scoring, calibration, disputes — relies on this unit of measurement and collapses if it’s vague.

Inference chain behind the gradeFour steps linked by arrows labelled sorting, interpretation and scale: time-stamped facts, observed indicator, assessed competency, grade on the scale.Each arrow adds an interpretationTime-stampedfactsObservedindicatorAssessedcompetencyGradeon the scaletriinterpretationscale
The grade is the end of a chain of inferences, not a direct measurement. Between what happened and the figure entered on the grid, the examiner sorts, interprets and then converts: each step must remain traceable back to the facts.

Une note de compétences repose sur une chaîne d’interprétations. D’abord, on observe des faits précis (comme “annonce une déviation”). Ensuite, on les relie à une compétence (“gestion de la charge de travail”). Enfin, on donne une note. Si un maillon manque, la note n’est pas fiable.

Un contrôleur annonce un conflit de trafic à 15h30:22, puis le résout à 15h32:10. L’examinateur note “anticipe les conflits” (fait) → “gestion proactive” (compétence) → niveau 3 (note). Sans ces horodatages, la note serait subjective.

The figure above sums up the core issue: the grade is the end of a chain of inferences, not an instrument reading. The examiner filters what they retain, interprets it as indicators, aggregates indicators into competence, and converts competence into a level. A defensible grade is one that can be unwound backward, from the number to the timestamped facts. An indefensible grade is one missing a link — and it’s almost always the first: the facts.

What a grading grid actually contains

An active grading grid has three layers: competencies — a handful, remarkably stable across operators — the behavioral indicators that break them down, and a scale with a few levels, each described by a performance profile. The scale hides the heaviest doctrinal choice, as two logics compete for dominance.

Two scale logics for the same grid
Lecture Criterion-referenced scale Échelle normative
Question asked Are the expected behaviours present, and how often? Where does the performance stand relative to what is expected at this level?
Point fort Clear floor: acceptable is defined by facts Discriminates the top of the spectrum, useful for guiding training
Point faible Compresses the top of the scale: every “satisfactory” looks alike Depends on each examiner’s own reference population
Typical drift Disguised pass/fail grading Leniency or severity depending on the grader’s experience
Usage dominant Pass threshold, check decision Upper levels of the grid, guiding training

Most active grading systems blend both: a criterial baseline — the acceptable is defined by present or absent behaviors, not comparison — and higher, appreciative levels, which serve training orientation more than control decisions. Knowing which logic applies at a given moment is the rater’s first skill: much of grading disagreement stems from scale logic mismatch.

The baseline itself remains. In line with historical non-technical skills assessment methods, an unfavorable grade requires more than an isolated fact: repeated behavior and its observable impact on flight conduct. This double safeguard protects both parties — the candidate from anecdotal overreach, the examiner from arbitrariness claims.

Repetition and consequence matrixMatrix crossing the repetition of the behaviour with its consequence on the conduct of the flight: only the repeated quadrant with an observable consequence carries a defensible unfavourable grade, the other three calling for recording or debriefing.aucuneobservableconsequence on the conduct of the flightisolatedrepeatedrepetitionisolated fact:record, do not gradefait marquant :documentsignal faible :debriefunfavourable gradedefensible
A defensible unfavourable grade requires repetition and an observable consequence. The other three quadrants are not wasted: they feed the debriefing and the session documentation, without carrying the figure.

Calibration: creating examiner consensus

A good grid isn’t enough. Show the same recorded session to ten qualified examiners without prep: the score dispersion always surprises the first time — not because some observe poorly, but each weights indicators with their own line and training experience. Examiner consensus isn’t a qualification byproduct: it’s a maintenance product.

Three men in blue shirts are seated at a desk in a control room, looking at a large monitor displaying an airfield simulation with an airplane, while taking notes.
Soft skills training requires observation and documentation to evaluate performance.
Examiner concordance loopFour-step cycle: a reference session is graded independently by each examiner, the differences are measured and archived, the common standard is adjusted, then a new reference session restarts the loop.reference sessionindependent gradingmeasuring the differencesadjusted standardconcordance loop
Concordance cannot be decreed; it is maintained in a loop. Each round — independent grading of the same session, measurement of differences, adjustment of the standard — narrows the spread between examiners without ever eliminating it.

The loop is simple and demanding. A reference session, scored independently by each; a gap measurement, indicator by indicator, archived; a discussion seeking not soft consensus but the root of each gap — differing indicator interpretation, differing fact weighting, or a fact seen by one but not the other; finally, a common reference adjusted, to guide the next session. The goal isn’t identical scores: it’s that diverging examiners can explain their divergence via the grid, not habit.

Between calibration sessions, monitoring becomes statistical. Each examiner has a scoring profile — average, spread, competence correlation — and this profile is a signature. A sustained average drift from peers signals entrenched leniency or severity; a collapsing spread signals a non-decision maker; over-correlated columns signal halo effect. None of these signals condemn: each opens a conversation, ideally before candidates are affected.

Two examiner grading profilesTwo grade distributions on the same scale: examiner A, solid line, concentrates almost all grades on the central level; examiner B, dashed, uses the full range of levels. Identical sessions, opposite statistical signatures.A: central tendency (solid)B: full range (dashed)12345level entered on the gridfrequency
For identical sessions, the grading profile is a signature. A distribution compressed onto the central level says nothing about the candidates: it points to a grader who no longer decides — exactly what the periodic review of profiles seeks to detect.

Classic drifts and their signatures

No grading drift is original. All are documented, measurable, and reappear as soon as monitoring stops — because none stems from bad faith: they’re normal cognitive shortcuts, worsened by simulator fatigue.

Five grading drifts, their signatures and their countermeasures
Drift Signature in the grades Contre-mesure
Halo effect Highly correlated competencies: one strength pulls all the columns up Grade competency by competency, from distinct facts
Tendance centrale The middle level absorbs almost all grades Require a verbatim record for the middle level as for the extremes
Leniency or severity Average persistently away from that of peers Review of grading profiles, concordance sessions
Recency The grade reflects the last quarter of the session Time-stamped facts during the session, deferred grading
Contamination technique A successful approach “buys” the non-technical grades Separate the record of parameters from the record of behaviours

The most costly in practice isn’t halo or leniency: it’s recency, because it attacks the facts link. A checkride lasts hours; the examiner’s memory doesn’t retain a uniform image but a U-curve — the start stands out, the end crushes, the middle fades. Yet the middle is precisely where routine communication, task sharing, and workload management outside peaks play out — the competencies the grid aims to capture.

Primacy and recency effects on gradingU-shaped curve of the weight of events in the examiner’s memory: high at the start through primacy, low in the middle, maximum at the end through recency, while a dashed straight line shows the real, uniform weight of the facts.primacyrecencymiddle of the session: the dipreal weight of facts (dashed)startfinsession timeweight in the grade
Without time-stamped note-taking, memory reproduces the session as a U. The start, and especially the end, weigh more than their real weight; the middle of the session — often the richest in workload management — fades from the grading.

The trap: scoring from memory during debrief. A grid filled afterward isn’t filled by the session: it’s filled by what memory retained — the start, the end, and the most spectacular event. The examiner thinks they’re replaying the session; they’re replaying their U-curve. And if the candidate contests, there’s nothing to counter: no facts, no times, no indicators — just an impression, exactly what the grid was meant to eliminate.

The grade as raw material for training

In an evidence-based training program, the grade doesn’t end up in a file: aggregated and anonymized, it guides subsequent modules — competencies with fleet-wide score drops become program priorities. This loop changes the cost of lenient grading. It no longer just falsifies an individual file: it deprives the system of its signal and trains an entire fleet on the wrong weaknesses.

The debrief is the other half of grading. A grade is accepted when explained, and explained when the examiner can present timestamped facts the candidate recognizes — exactly what indicator-based note-taking produces. Conversely, a debrief starting with the number has already lost: everything that follows looks like justification.

The right reflex: separate data collection from grading. During the session, don’t grade — record timestamped facts in indicator language: “doesn’t announce speed deviation, two occurrences during climb.” After the session, grade competence by competence, starting with those least marked in memory. If facts are missing to grade, it’s the note-taking that needs fixing at the next session — never the grade that needs inventing.

What to check in your own system

Four questions suffice to assess a grading system’s maturity, and they apply equally to the training manager and the candidate who wants to understand how they’ll be evaluated:

  • Is each grid indicator written with an observable action verb — or are there still disguised adjectives?
  • Does the scale explain what separates adjacent levels, or only what the extremes look like?
  • Do calibration sessions produce measured and archived gaps, or just discussion?
  • Who reviews examiner statistical profiles, how often, and with what authority to act?

A gray area current grids handle poorly remains: silent prevention. A crew that anticipates so well nothing exceeds shows few remarkable behaviors — no announced deviation, since none occurred; no redistributed workload, since never saturated. Recent references attempt to grade anticipation itself — creating margins, verbalizing the plan before the load — but the temptation persists to better grade a well-recovered difficulty than a never-occurred one. For those preparing for examiner or instructor qualification, the practical consequence is clear: train fact-taking before grading — it’s what’s missing in early sessions, and what qualification preparation pathways address most usefully.

Finally, these mechanisms vary by authorities, operators, and approved programs: the applicable regulation and the operator’s approved documentation — operations manual, active grids and scales — prevail in all circumstances over this article’s content.

Commentaires

Leave a Reply

Découvrez l’écosystème FL330

FL330, c’est toute l’aviation professionnelle — explorez nos autres espaces.