Two examiners witness the same go-around in a simulator. One rates the workload management at a higher level, the other finds it barely acceptable. Neither is wrong: they interpret the grid differently. As long as the assessment measured tolerances — an axis held, an altitude, a time — the gap between raters remained marginal. Now that the score assesses communication, decision-making, or leadership, the grading method has become a standalone safety issue: a poorly calibrated grid produces indefensible scores and control decisions that won’t hold up to scrutiny.
Grade behavior, never personality
Early generations of crew competence evaluations rated impressions: “good team spirit,” “natural leadership.” A double dead end. Impressions aren’t contestable — you don’t debrief a personality trait — and they’re not actionable: they don’t say what to train. Current frameworks, derived from European work on non-technical skills assessment and later standardized by ICAO’s competence framework, represent a complete reversal: you don’t score a quality, you score observable behaviors, described by action verbs. “Announces the deviation as soon as it’s detected.” “Distributes tasks before the high-workload point.” “Verbalizes intent before acting on automation.”
A well-written indicator has three properties. It’s observable: no mental states, no assumed intentions. It’s factual: described without adjectives, so it can be recorded verbatim in a session report. It’s datable: tied to a specific moment, allowing it to be retrieved — on video, in simulator data, or in the other observer’s notes. The entire follow-up mechanism — scoring, calibration, disputes — relies on this unit of measurement and collapses if it’s vague.
Une note de compétences repose sur une chaîne d’interprétations. D’abord, on observe des faits précis (comme “annonce une déviation”). Ensuite, on les relie à une compétence (“gestion de la charge de travail”). Enfin, on donne une note. Si un maillon manque, la note n’est pas fiable.
Un contrôleur annonce un conflit de trafic à 15h30:22, puis le résout à 15h32:10. L’examinateur note “anticipe les conflits” (fait) → “gestion proactive” (compétence) → niveau 3 (note). Sans ces horodatages, la note serait subjective.
The figure above sums up the core issue: the grade is the end of a chain of inferences, not an instrument reading. The examiner filters what they retain, interprets it as indicators, aggregates indicators into competence, and converts competence into a level. A defensible grade is one that can be unwound backward, from the number to the timestamped facts. An indefensible grade is one missing a link — and it’s almost always the first: the facts.
What a grading grid actually contains
An active grading grid has three layers: competencies — a handful, remarkably stable across operators — the behavioral indicators that break them down, and a scale with a few levels, each described by a performance profile. The scale hides the heaviest doctrinal choice, as two logics compete for dominance.
| Lecture | Criterion-referenced scale | Échelle normative |
|---|---|---|
| Question asked | Are the expected behaviours present, and how often? | Where does the performance stand relative to what is expected at this level? |
| Point fort | Clear floor: acceptable is defined by facts | Discriminates the top of the spectrum, useful for guiding training |
| Point faible | Compresses the top of the scale: every “satisfactory” looks alike | Depends on each examiner’s own reference population |
| Typical drift | Disguised pass/fail grading | Leniency or severity depending on the grader’s experience |
| Usage dominant | Pass threshold, check decision | Upper levels of the grid, guiding training |
Most active grading systems blend both: a criterial baseline — the acceptable is defined by present or absent behaviors, not comparison — and higher, appreciative levels, which serve training orientation more than control decisions. Knowing which logic applies at a given moment is the rater’s first skill: much of grading disagreement stems from scale logic mismatch.
The baseline itself remains. In line with historical non-technical skills assessment methods, an unfavorable grade requires more than an isolated fact: repeated behavior and its observable impact on flight conduct. This double safeguard protects both parties — the candidate from anecdotal overreach, the examiner from arbitrariness claims.
Calibration: creating examiner consensus
A good grid isn’t enough. Show the same recorded session to ten qualified examiners without prep: the score dispersion always surprises the first time — not because some observe poorly, but each weights indicators with their own line and training experience. Examiner consensus isn’t a qualification byproduct: it’s a maintenance product.

The loop is simple and demanding. A reference session, scored independently by each; a gap measurement, indicator by indicator, archived; a discussion seeking not soft consensus but the root of each gap — differing indicator interpretation, differing fact weighting, or a fact seen by one but not the other; finally, a common reference adjusted, to guide the next session. The goal isn’t identical scores: it’s that diverging examiners can explain their divergence via the grid, not habit.
Between calibration sessions, monitoring becomes statistical. Each examiner has a scoring profile — average, spread, competence correlation — and this profile is a signature. A sustained average drift from peers signals entrenched leniency or severity; a collapsing spread signals a non-decision maker; over-correlated columns signal halo effect. None of these signals condemn: each opens a conversation, ideally before candidates are affected.
Classic drifts and their signatures
No grading drift is original. All are documented, measurable, and reappear as soon as monitoring stops — because none stems from bad faith: they’re normal cognitive shortcuts, worsened by simulator fatigue.
| Drift | Signature in the grades | Contre-mesure |
|---|---|---|
| Halo effect | Highly correlated competencies: one strength pulls all the columns up | Grade competency by competency, from distinct facts |
| Tendance centrale | The middle level absorbs almost all grades | Require a verbatim record for the middle level as for the extremes |
| Leniency or severity | Average persistently away from that of peers | Review of grading profiles, concordance sessions |
| Recency | The grade reflects the last quarter of the session | Time-stamped facts during the session, deferred grading |
| Contamination technique | A successful approach “buys” the non-technical grades | Separate the record of parameters from the record of behaviours |
The most costly in practice isn’t halo or leniency: it’s recency, because it attacks the facts link. A checkride lasts hours; the examiner’s memory doesn’t retain a uniform image but a U-curve — the start stands out, the end crushes, the middle fades. Yet the middle is precisely where routine communication, task sharing, and workload management outside peaks play out — the competencies the grid aims to capture.
The trap: scoring from memory during debrief. A grid filled afterward isn’t filled by the session: it’s filled by what memory retained — the start, the end, and the most spectacular event. The examiner thinks they’re replaying the session; they’re replaying their U-curve. And if the candidate contests, there’s nothing to counter: no facts, no times, no indicators — just an impression, exactly what the grid was meant to eliminate.
The grade as raw material for training
In an evidence-based training program, the grade doesn’t end up in a file: aggregated and anonymized, it guides subsequent modules — competencies with fleet-wide score drops become program priorities. This loop changes the cost of lenient grading. It no longer just falsifies an individual file: it deprives the system of its signal and trains an entire fleet on the wrong weaknesses.
The debrief is the other half of grading. A grade is accepted when explained, and explained when the examiner can present timestamped facts the candidate recognizes — exactly what indicator-based note-taking produces. Conversely, a debrief starting with the number has already lost: everything that follows looks like justification.
The right reflex: separate data collection from grading. During the session, don’t grade — record timestamped facts in indicator language: “doesn’t announce speed deviation, two occurrences during climb.” After the session, grade competence by competence, starting with those least marked in memory. If facts are missing to grade, it’s the note-taking that needs fixing at the next session — never the grade that needs inventing.
What to check in your own system
Four questions suffice to assess a grading system’s maturity, and they apply equally to the training manager and the candidate who wants to understand how they’ll be evaluated:
- Is each grid indicator written with an observable action verb — or are there still disguised adjectives?
- Does the scale explain what separates adjacent levels, or only what the extremes look like?
- Do calibration sessions produce measured and archived gaps, or just discussion?
- Who reviews examiner statistical profiles, how often, and with what authority to act?
A gray area current grids handle poorly remains: silent prevention. A crew that anticipates so well nothing exceeds shows few remarkable behaviors — no announced deviation, since none occurred; no redistributed workload, since never saturated. Recent references attempt to grade anticipation itself — creating margins, verbalizing the plan before the load — but the temptation persists to better grade a well-recovered difficulty than a never-occurred one. For those preparing for examiner or instructor qualification, the practical consequence is clear: train fact-taking before grading — it’s what’s missing in early sessions, and what qualification preparation pathways address most usefully.
Finally, these mechanisms vary by authorities, operators, and approved programs: the applicable regulation and the operator’s approved documentation — operations manual, active grids and scales — prevail in all circumstances over this article’s content.

Leave a Reply
You must be logged in to post a comment.