Rating grids for crew selection: making the decision defensible after the fact

A panel of aviation evaluators in a control room reviewing a candidate's assessment grid, highlighting observable behaviors under dim lighting.

Six months after the campaign, a rejected candidate asks for the reasons behind the refusal. The panel remembers it roughly: he wasn’t up to standard on priority management. Now it has to be shown. If the grid produced nothing but a global score and three adjectives, there is nothing to produce, and the file leaves technical ground for probative ground — the ground on which it generally falls to the operator to show that its decision rested on objective elements.

A grid is not a measuring instrument, it is an evidentiary device

A grid is expected to make evaluation objective. That asks too much of it, and not enough: it does not make a human judgement objective, it makes it reconstructible. Three functions coexist in it — standardising what evaluators look at, comparing candidates who did not sit the same day, and leaving a trace that a third party will be able to re-read without having attended the exercise.

Only the third withstands challenge. A score is an irreversible summary: from three out of five, nothing can be traced back. What can be defended is the whole chain — the wording of the criterion as it was written before the exercise, the scale and its anchors, the fact observed, each evaluator’s rating before any exchange, the aggregation rule fixed in advance, the decision and its grounds.

The robustness test fits in one sentence: hand the file to someone who was not in the room and ask them to reconstruct the decision. If they manage it, the grid has done its job. If they have to phone you to understand, what decided things was not in the grid.

A good grid doesn’t just give a score—it lets someone who wasn’t there understand exactly why a candidate was accepted or rejected. It’s like a recipe: if you follow the steps, you should get the same result every time.

From inferred judgement to observable behaviour

Leadership, maturity, motivation: these words denote traits, and a trait is not observed, it is inferred. Inference is exactly what a third party can dispute, because it rests on reasoning the evaluator never wrote down. A behaviour, by contrast, either occurred or did not: he announced his intention before acting on the autopilot, he had an ambiguous clearance read back, he deferred a checklist reading to fly the aircraft first.

Hence a simple drafting rule. A criterion is worded with an action verb, an object, and the condition under which it is observed. It carries no degree: degree belongs to the scale. “Communicates effectively” mixes the two and lets each evaluator set their own bar without saying so.

The competency frameworks published for aircrew already supply this vocabulary. Rewriting them from scratch loses comparability with training and checking, where the same competencies are scored; instead they need to be instantiated to the exercise actually organised.

There remains the criterion that presupposes experience. “With the authority expected of a commander” cannot be reached by a cadet: selection becomes an undeclared experience filter. If experience is a prerequisite, it belongs in the access conditions.

The same criterion, two wordings: what gets scored changes with the phrasing
Competency targeted Inferential wording Observable wording What gets recorded
Situational awareness Good situational awareness Announces a trajectory or parameter deviation before it is flagged by the other seat or by a system The deviation, the moment, who announced it first
Communication Communicates effectively States their intention before acting on a control or a mode The actions announced and those that were not
Workload management Copes well with stress Defers or delegates a non-priority task when two demands arise at once The deferred task and the moment
Teamwork Good team spirit Asks for the other seat’s opinion before settling on an option, then restates the answer The request and the restatement

During a simulator exercise, a candidate flying an Airbus A320 announces, “I’m going to engage the autopilot in NAV mode” before doing so. This matches the observable criterion “States their intention before acting on a control or a mode.” If they skip the announcement, it’s a clear failure on that competency.

The scale: notches, anchors, and the biases it channels

The number of notches is the first arbitration. An odd-numbered scale offers a median refuge, where the hurried evaluator takes shelter; an even-numbered scale forces a choice but erases the nuance of “conforming without more.” The debate matters less than one might think: the quality of a scale lies in its anchoring.

Anchoring means describing each notch with a behaviour, not an adverb. “Three: satisfactory” says nothing to anyone; “three: detected the deviation and announced it without being prompted” says something verifiable. Writing these anchors is the real design work, and it is the part a third party will reread the day the decision is questioned.

Anchoring does not remove biases, it makes them visible. The halo effect lets a salient performance bleed into other criteria; leniency and severity are personal constants, not requirements; central tendency pulls everything toward the middle; order effects penalise whoever follows a brilliant candidate. None of them yield to good will: they yield to measurement.

Scores given by two evaluators to the same cohortTwo frequency curves on a one-to-five scale: the solid curve peaks at notch three and spreads out, the dashed curve peaks at notch four and stays tight.12345score givencandidatessystematic offsetevaluator Aevaluator B
Two evaluators can agree on every candidate and still be incomparable. On the same cohort, one spreads their scores around the middle of the scale, the other tightens them higher up: the ranking may coincide, but the scores do not transfer from one panel to another.

The measurement is within easy reach: compare the distribution of scores from each evaluator over an entire campaign. Whoever’s average deviates persistently from that of their peers is not more demanding, they are scoring on a different scale — and two candidates who went before different panels stop being comparable. The offset is dealt with through calibration, never through a correction applied after the fact: recalculating scores destroys the trail.

Measuring the gap between evaluators before resolving it

Double rating is only worthwhile if it is independent, and the word that matters is beforehand. If the first announces their score, the second aligns without even noticing: the agreement observed becomes an artefact of the conversation, and the second rating no longer proves anything.

The two values therefore need to be recorded, and the first one must never be overwritten. A file that shows a disagreement and how it was handled is stronger than a smooth-looking one: an entire campaign without the slightest gap should first raise doubts about the real independence of the ratings.

The trigger threshold is set and written down before the campaign — for example, a two-notch gap requires reconciliation. This records the fact that changed the assessment, not an arithmetic compromise. An average between two and four means nothing: either the behaviour occurred or it did not, and one of the two evaluators missed it.

Agreement between two evaluatorsEach candidate is a point: the first evaluator’s score on the x-axis, the second’s on the y-axis. The diagonal band contains one-notch gaps; three points fall outside it.135135evaluator A’s scoreevaluator B’s scoregap ≤ 1one candidateoutside the band:reconciliation
The gap between two ratings is read on a matrix, not in an average. Each point is a candidate rated twice; the band contains one-notch gaps, which are tolerated. Outside the band, the written rule requires the file to be reopened before any decision.

Upstream, calibration is practised on benchmark cases: recordings or written scenarios that all evaluators score separately before comparing results together. The date of the last calibration session and the gaps observed are recorded on the same footing as candidate files.

The right reflex. Score blind, then open the discussion: each person records and confirms their score before knowing the other’s, and the reconciliation is written in a separate field, without erasing the two original values. It is this pair of numbers, not their average, that will make the decision legible six months from now.

From score to decision: compensation, floor, elimination

A grid produces scores; the decision itself comes from an aggregation rule — and that is exactly what people forget to write down. Three regimes coexist, with very different consequences.

Four professionals in business attire review documents under a focused lamp, with one pointing at a rating grid.
Rating grids ensure objective and defensible crew selection decisions.
Three aggregation regimes and what they require you to be able to justify
Regime Rule applied What it allows What must be demonstrable
Compensatory sum Points accumulate; a weak criterion can be made up for elsewhere A highly contrasted profile That the weak competency is not critical to the role targeted
Floor per competency A minimum is required on each competency, in addition to the total Contrast, within an announced limit The origin of the floor and its consistency from the first to the last candidate
Eliminatory criterion An observed behaviour stops the selection, regardless of the rest An immediate decision The precise fact, its date, and the rule written before the exercise

Pure compensation is the most fragile regime in crew selection: a flattering total can conceal a competency below the minimum, and it will have to be explained why that one could be made up for elsewhere. A floor per competency answers the objection, provided it is identical for everyone.

Two competency profiles with the same totalFive competencies on the x-axis, the score on the y-axis, a dashed line for the required minimum. The solid profile stays at or above the minimum everywhere; the dashed profile exceeds it on three competencies and falls below it on two, for an identical total.135scorefloorComm.Proced.DecisionWorkloadTeamcandidate Acandidate B
Two candidates with the same total, only one possible decision. The dashed profile dominates on three competencies and falls below the floor on two others: without a minimum per competency, the sum ranks it ahead.

The same logic applies to weighting: if competencies do not carry the same weight, that is decided and written down before the campaign. Adjusted along the way, it does not correct an injustice, it destroys the evidence.

Lastly, an absolute threshold and a quota do not say the same thing. Taking the best candidates for a number of posts is a ranking; declaring a candidate below the threshold is a judgement of fitness. The rejection letter must state which was applied, otherwise it implies an unfitness that could not actually be demonstrated.

What makes a decision open to challenge

The weak point of a good grid is almost always the free-text field. That is where, in ten seconds and without rereading, the costly phrasings get written: allusions to age, family situation, appearance, accent, or the notorious “he won’t fit into the team.” These sentences become part of the file and can be read years later.

The trap. The free comment is the only part of the grid nobody rereads before sending, and the first thing pulled out in case of a dispute. “Too full of himself,” “atypical profile,” “won’t fit the group” describe no behaviour at all: they describe the panel’s impression, and that impression, on its own, cannot be defended.

The drafting rule follows from this: a free-text field records only an observed fact, dated, tied to a criterion in the grid. Praise outside the criteria weakens the case just as much as a disparaging remark — it shows that the panel was scoring something other than what had been announced.

Under French labour law, two obligations additionally frame the process: the candidate is informed, before they are put in place, of the methods and techniques of assessment used, and the information requested must bear a direct relationship to the role on offer. Other states have equivalent rules, of varying scope: an operator recruiting across several countries checks the local rule rather than duplicating its own. An exercise whose very principle is kept secret remains fragile on this ground whatever the jurisdiction.

On top of that comes the data-protection regime: declared purpose, minimised data, a retention period that is fixed and actually applied, the data subject’s right of access. Writing every line as if the candidate were going to read it — they can ask to — is not a legal precaution, it is a drafting discipline: it mechanically produces factual statements.

What is still worth checking at your own organisation

Even backed by a solid grid, the decision retains grey areas: the informal conversation that takes place alongside the process and carries the final say without leaving a trace, the sole evaluator of small structures, the instructor who trained the candidate and then rates them, the offhand remark made in a meeting and never recorded.

Six checks are enough to gauge the process, and none of them require any tool: attempt a cold reconstruction on a few already-closed files; pull the distribution of scores by evaluator over the last campaign; reread the free-text fields with a third party’s eye; date the last calibration session; verify that the reconciliation threshold exists in writing; check that the announced retention period is the one actually applied.

For those preparing for selection, the consequence is direct: the framework, the criteria and the scale are not trade secrets, and it is legitimate to ask what will be observed before taking the seat. Preparing, then, means training to produce announceable behaviours — and to announce them, since that is what will be scored. The selection preparation modules start from this observation.

The applicable texts, the chosen competency framework and the operator’s approved documentation take precedence over this article, which describes a mechanism, not a procedure: in the event of a discrepancy, the operations manual, the company’s recruitment policy and the advice of its legal counsel prevail.

Questions fréquentes

Should the grid be shared with candidates before the exercise?

Under French labour law, the candidate is informed of the assessment methods and techniques before they are put in place, which covers the nature of the exercises, the competencies observed and the scoring principle. Sharing the framework and the scale does not weaken the selection: a candidate who knows what is being observed produces more legible behaviours, and therefore ones that are easier to score. What is not disclosed are the scenarios themselves and the detailed anchors, to avoid rote learning.

Four or five notches on the scale?

An odd-numbered scale leaves a median notch that captures hesitation and weakens discrimination; an even-numbered scale forces a choice but removes the neutral position, which is nonetheless useful when the performance is conforming and nothing more. The choice matters less than the anchoring: a short scale whose every notch is described by a behaviour generally discriminates better than a long scale whose notches carry only adverbs. Fix the number of notches once and for all and focus the effort on writing the anchors.

What should be done when two evaluators differ by two notches?

The rule is set and written down before the campaign; a common threshold is to open a reconciliation as soon as the gap exceeds one notch. It consists of going back to the observed fact — who saw what, and when — rather than splitting the difference, because an average between two and four corresponds to no behaviour at all. Both initial ratings stay in the file, along with a note of what shifted one of the assessments. If the gap persists, it is settled by a third evaluator or by the planned escalation rule, never by seniority.

Can a candidate be rejected on a single criterion?

Yes, under two cumulative conditions. The eliminatory nature of the criterion must be written before the exercise, not decided after seeing the result, and the fact that triggers it must be an observed behaviour, dated and recorded, not a general impression. An undeclared eliminatory criterion turns a legitimate decision into an arbitrary one, even when the substance is justified.

How long should completed grids be kept?

There is no universal retention period: it is deduced from the declared purpose, fixed before the campaign, and entered in the record of processing activities. The risk runs both ways — a file kept with no defined retention period is a compliance failure, a file destroyed too soon deprives the operator of the means to defend its decision. The trade-off is decided with the data-protection officer when the process is designed, not after a complaint is received.

Can an instructor assess a candidate they trained?

This is the most frequent conflict of interest in small structures, and it weakens the decision in both directions: leniency toward one’s own training work, or compensatory severity. When separation is not possible, it is replaced by recorded safeguards — double rating by an evaluator not involved in the training, the relationship declared in the file, arbitration entrusted to a third party in the event of a gap. Transparency about the relationship is better than omitting it, which always surfaces at the wrong moment.

Commentaires

Leave a Reply

Découvrez l’écosystème FL330

FL330, c’est toute l’aviation professionnelle — explorez nos autres espaces.